Why will no one hire me for a job?

I have been out of work for almost 3 years now. (Of note, I had to go on medical leave due to an illness and had to stay out of work until I was deemed well enough to work). I have applied for job after job after job for the majority of the time I have been deemed well. Other than doing a couple temp jobs, no one has picked me for a more long-term position. Could it be that the references I had to give for all my job applications be the culprit to my unlucky job hunt? How can I find out for sure? Should I try to call each human resource office and ask them why I'm constantly being unconsidered for jobs? I've long since stopped receiving unemployment and am down to my last bit of retirement money, which I have been living on. I don't know what else to do, and I feel like giving up. (And by the way, yes, I have applied to fast food joints, various stores, and supermarkets, and they too have turned me away.)