Bottoms having issues with tops who remove protection during sex?

Whenever I'm bottoming guys keep trying to slip off the condom after we've been having sex for a bit. Is this a problem for any other bottoms? It's such a vulnerable position because they could do it faster than you could react and you might not be able to see. I've fortunately been able to catch it every time because I'm always trying to be conscious of it (also I don't have that much experience) but I might not always be so lucky. Plus guys that do that are more likely to have something.Any way to avoid this in the future?