I don't know how long I can take my boyfriend mentioning his ex

If we're doing an activity that they used to do a lot, he brings him up. If he's trying to tell me something he doesn't like, he brings him up, and so on. If we see each other for a whole weekend, this happens once, sometimes twice.They're still in touch and considers him a friend, which is fine. They broke up 4 years ago, but he's not even telling me stuff from the present as a friend, he always brings up the past. I've literally talked about my ex 3 times since we got together 6 months ago.I talked to him about this once and he said "well, they're part of our lives and we were together for a long time, so why avoid talking about them?" Nothing changed.Also, on a Friday, we were both at each other's homes and we were supposed to go out dancing. At 4pm he told me he was feeling very lazy and we'd go out the following day. At 8pm he confirmed this and we decided to go out on Saturday. Then, at 1am I get a text from him "I went out dancing with my ex's boyfriend, don't get mad." I mean, why the hell would you go out with your ex's boyfriend to get drunk and dance? I don't even know if the ex was there and he didn't tell me. I just found the whole situation pretty weird.