WENN 1 / 4

Joe Giudice is losing weight and he's apparently losing his mind in prison.

Joe is "miserable" behind bars, a source tells Us Weekly, adding, he "misses his kids badly."

The source added that the former reality TV star "sits alone much of the time and is falling into a depression."

Joe is currently in Fort Dix Federal Corrections Institution in New Jersey, which he reported to in March to serve a 41-month sentence for a variety of fraud charges. His wife, "The Real Housewives of New Jersey" star Teresa Giudice, was in prison for similar charges last year before being released just before Christmas.

The couple shares four daughters: Gia, 15, Gabrielle, 11, Milania, 10, and Audriana, 7.

Us's source says Joe tries to distract himself by playing cards with two other inmates he's befriended, gambling for commissary items and working out.

Last week, Teresa said that Joe has actually lost over 30 pounds since being locked up. She implied that his weight loss is from hitting the gym and eating well, not from depression.

"He looks amazing. I can't wait for him to come home," she told Extra. "I say, 'You're not coming home until you have abs.'"

Teresa said the communication between she and her man is as good as ever, given the circumstances.

"I talk to him every day, sometimes twice a day. He calls me and we email constantly throughout the day, and I see him every week," she said. "We all go through tough times in life. Life is a roller coaster, it's not always peaches and cream. You gotta go through good times and bad times. It's makes you the person that you are."

She also doesn't shy away from her past, as she knows about prison life, having been behind bars for 11 months for similar fraud charges as her husband.

"It's something that happened in my life, part of my journey, and if I can help anyone else out there who's having a hard… I want to inspire anybody else," she said.