How do you deal with a passive aggressive partner?

I am looking for advice. We are both in our mid-30s and stable in our lives. He often makes me feel like I am a terrible person because of the things I do or because I am direct.For instance, I can't sleep when someone is wrapped around me. I like to cuddle and all, but if I am reading or trying to fall asleep, I just can't with someone wrapped around me. He takes this as me not wanting him at all and then stops talking to me and just goes to his side of the bed in the most extreme sense, basically hanging off the side as far from me as possible. Anyway, this makes me feel terrible and I say that.Another we were out to lunch with my aunt and she said she was going to pay, she told me she would like to pay. Because I did not wrestle her to the ground and pay for the meal I was subjected to the silent treatment for a whole day.I have enough of my own issues that this drains me emotionally and causes me to have panic attacks.I love him and we do enjoy each other, but I feel like I have to walk on eggshells constantly for fear of setting him off.Maybe I am just venting or something, but it wears on me. I don't hold a grudge for the most part and often I don't know why I am getting the silent treatment... Do I have a problem? Am I a bad person?