David Letterman is at a point and time in his life where he's rarely ventured before, and it's left him feeling "naked and afraid" he told "CBS Sunday Morning" show host Jane Pauley.

On Wednesday, he will host the "Late Show" for the final time. After that, David Letterman will never again lead the late night show many have come to follow.

"I don't think anybody will ever see me again," Letterman told Pauley of his post-show exit.

But Letterman may actually just be joking. With all the new venues avenues the Internet now offers for someone of his expertise and experience, he may eventually do something else.

Either way, Letterman is making sure to go out with a bang. He recently talked about cleaning his office for the final time with Oprah Winfrey, insisting he's been on the job now for "four or five years."

From there, somehow, the conversation evolved into a discussion on marijuana, with Letterman firing off the question. Letterman asked if Winfrey had smoked marijuana.

After a stunned look, Winfrey insisted she hasn't for at least three decades now. Meanwhile, Letterman's last experience seemed to be a bit more recent.

"It makes you feel dumb, smoking the weed," he said.

That almost seems like nothing compared to what Letterman told Pauley his experiences with alcohol have been like. He recalls starting to drink when he was just 11 and seemingly continuing nonstop until he was 34.

"It was that old thing where my dad used to like scotch and soda," he said. "And, 'Here, here, Dave, you wanna try one?' And I tried one. And I just thought, 'This is fantastic.' ... It was delightful. I -- I just -- I loved everything about it."

Only his fear of destroying what he was building at NBC ultimately got him to put down the bottle.

"I just said to myself, 'You're -- you're a fool, you're a dumb fool," he recalled. "'You can't do this. You -- you know, they just don't give these shows to everybody.'"

Letterman said he's never had another drink. And after NBC bypassed him as Johnny Carson's "Tonight Show" replacement in 1992, he took his talents to CBS. Now, David Letterman says the gig is up; it's time for him to exit stage right and spend more time with his family, particularly 11-year-old son Harry.