Breaking Up W/ First Boyf?

So, long story short, I came out like 2 years ago and started hooking up with this guy like a year later. I moved away for 6 months and we were in a relationship, technically non-monogamous, but neither of us slept with anyone else. We've done long-distance for a while, but now that we're back in the same place, I don't even want to see him really.I'm a very masculine guy with very classic interests but he's super flamboyant. I'm a pretty "establishment" guy with high career goals, but he's an "outsider" with no career plans. We're going into our Senior year at the same college, and I've been working my ass off this summer and he's done nothing. I'm constantly annoyed that we always have to do his things, but he's extremely uncomfortable and honestly even like incapable of existing in "my spaces" -- football games, hanging with fratty type guys, engaging effectively with my parents, etc.This issue is, it's both of our first relationships, and he's definitely more invested than I am, and I'm pretty sure he's falling in love with me.He also has some mental health issues, and it's a lot for me to deal with. Honestly, I just don't enjoy spending time with him, but I'm worried about hurting him.The question is, do I just continue this through college and break up with him then, or do I break up with him now? Both of us plan to go our separate ways in May anyway, so I'm unsure.TL:DR -- annoyed with college boyfriend, honeymoon period over, worth it to break up now or hold on longer