No, never had that problem.
I only do it when I'm in the mood, and it happens. Sometimes might take a bit longer, but never had I failed in doing so.
There might have been one or two nights in my life where I came in from a date, or a night out with friends and started to, and may have fallen asleep in the process. hahaha, real good lover to myself, huh.