Joe Giudice is rarely around to care for his four young daughters, and when he is at home, he often appears preoccupied with the thought that he will soon be stripped of his freedom.

"Joe is a completely unfit parent," a source told OK magazine of the husband of currently jailed "Real Housewife of New Jersey" star Teresa Giudice.

Both Teresa and Joe were convicted on bank fraud-related charges late last year, and at the time of their sentencing, a judge staggered when they would have to report to prison so someone would always be left to care for their daughters.

Currently, the 43-year-old Teresa is now away serving out the remainder of her 15-month sentence. As soon as she is released, Joe is slated to report and begin serving his 41-month bid.

Meantime, the father of 14-year-old Gia, 10-year-old Gabriella, 8-year-old Milania and 5-year-old Audriana is rumored to largely be an absentee dad.

"There was good reason Teresa was so worried about leaving him in charge," a source added.

Reports are Joe typically spends much of his time hanging out at Rails Steakhouse in nearby Towaco. Over the last several months, he's also been romantically linked to several women, though he vehemently denies being unfaithful to his wife.

"Joe knows he's going to jail in a matter of months, so drowning his sorrows is more important to him than his daughters," a source added.

With eldest daughter Gia pressed into the role of "substitute mother," sources lament the girls "have no structure."

"Joe's in-laws aren't watching them, they have no one looking out for their best interests," the source added.

Teresa Giudice is expected to be released from the Danbury Correctional facility sometime around Christmas and has already signed on to rejoin the cast of "Real Housewives."

Reportedly, producers have delayed filming for the new season in hopes of being able to feature Teresa in more of the scenes.