[Penetrating my boyfriend while he is asleep] (consensual) tips?

Hi! So my boyfriend and I are madly in love and live together. We did have some sex problems since after a few months it wasn't as exciting as the first time but we still enjoy good sex almost daily. I'm mostly a top and he is mostly a bttm, but occasionally we switch roles.Now about a week ago, while having some dirty horny talk he told me he would like to wake up to me fucking him middle of the night and said that would be really hot. I totally agree and want to try things to keep our relationship going strong.fast forward to now... two nights ago I put my finger up his ass while he was asleep and then managed to very slowly put about half of my penis inside him. The first time he woke up and yanked it out, went back to sleep without realizing what happened and then, with a lot of work and stamina (making difficult poses to make it possible with angles and all..) I repeated the same thing and again he woke up and changed positions etc I gave up.The next day I told him what happened and he was very surprised and said he "slept like a prince" and has no memory of waking up, let alone my dick being up his ass.Last night we had great sex but after he went to sleep I tried to do it again, I was excited. I couldn't put my penis inside as much as I had the night before and so I slowly put my finger, all the way in. He woke up and found my hand up there and was completely shocked and got mad that I was waking him up and I guess it wasn't the experience he was hoping for. I think he still enjoyed it as he loves ass play and loved being touched and felt in his ass.My question is what can I do, to make it possible for our fantasies to come true, he is a very sound sleeper but still, the nature of anus is to stay closed. Does anyone have any experience with this? How can I open it up? Anything else we are missing? Ideally we would just stay still or sleep like.