I get jealous hearing my boyfriend talk about his past

When I met my boyfriend 5 years ago I fell head over heels in love. Fast forward 5 years and details about his past experiences have come up. I was a virgin when I met him, and he was not. For some reason when he talks about how he made out with strangers at parties or bars, and that he had a couple of boyfriends before me just for fun, I get jealous I never experienced that. We've discussed this and he's not unsympathetic but we are monogamous. I really don't want to hurt him, but these feelings make me feel guilty but also like I missed out. He even would kiss other guys behind his boyfriends' backs. He's my first and only and we both love each other very much. However at 25 I feel I'm the only gay man who hasn't "played the field" so to speak.Tl:dr He's never cheated on me, nor I him. Will I be able to get over it or will I only feel worse the older, and less attractive, I get?