Justice Department and FBI officials have recommended pursuing charges against former CIA head Gen. David Petraeus. However, Attorney General Eric Holder continues to vacillate despite the recommendations. Some senators have also sided with Petraeus and have asked for him to be spared.

On Thursday, the New York Times reported Justice Department officials as well as the FBI recommended Petraeus face felony charges concerning the alleged leak of classified information to his mistress, Paula Broadwell, who was writing at the time a biography on the former CIA head and former general in Iraq and Afghanistan.

Petreaus denies ever providing Broadwell with sensitive information, but a Justice Department inquiry found he gave her access to his CIA email account as well as classified information. Following his resignation from the CIA in 2012, the FBI found classified documents on Broadwell's computer.

He resigned from the post after the affair became public.

His affair with Broadwell began in 2011 soon after he became head of the CIA. Broadwell was writing a biography titled "All In: The Education of General David Petraeus."

Speaking to the New York Times, officials briefed on the investigation asked for anonymity, as they were not allowed to speak publicly on the charges. When asked about the investigation, Justice Department spokesman Marc Raimondi did not comment.

In an interview with ABC's "This Week" on Sunday, Holder said the investigation into Petraeus "will be done in a fair and an appropriate way." Holder's comments come after Republican Sens. John McCain and Lindsey Graham released a statement casting doubt on the investigation's procedure.

"While the facts of the case involving General David Petraeus remain unknown and are not suitable for comment, it is clear that this investigation has been grievously mishandled," they wrote.

"It is outrageous that the highly confidential and law enforcement-sensitive recommendation of prosecutors to bring charges against General Petraeus was leaked to the New York Times. It is a shameful continuation of a pattern in which leaks by unnamed sources have marred this investigation in contravention to fundamental fairness."

On Sunday, Sen. Diane Feinstein expressed a similar sentiment on CNN's "State of the Union" telling host Gloria Borger that the general has "suffered enough."

"This man has suffered enough in my view," she said. "He lost his job as CIA director because of it. How much do we want to punish somebody?"

Nonetheless, investigators have asked for charges, arguing that officials in lower-ranking positions have been charged for lesser infractions, reports the New York Times.

The Los Angeles Times adds that an official said, "It's a top DOJ matter." The Justice Department continues to wait for Holder's decision.