My bf has no libido at all. What to do?

HiI have been with my partner for 5 years. I met him when he was 20 and I was 36. We love each other beyond everything we could imagine. He is everything I wanted BUT he has no libido... We had more sex at the begining of our relationship but nothing wild once every two months. He doesnt express his feelings so it makes it harder for me. At the begining he did, sending love notes, etc. I tried to break up as I have libido and this situation makes me feel bad and unwanted. But weeks after weeks the pain of being apart from each other was horrible, I couldnt sleep, I couldnt stop thinking of him, etc. After a maximum of 3 months I agreed to meet as he was asking me. Two days later he asked me to have sex with him and we officially get back together. Since we have once a year sex... To give a bit of a background he had health issues (bipolar), financial issues and now he tries to get a job. Yesterday I told him if he thinks it s fair to me, he said he has no sexual needs so what can he do? He is ok for me to have sex outside our relationship. I told him what s the meaning of being with someone if I have to be unfaithful. He said do what ever you think is good for you and he will follow my decision. This situation is so hard for me as we cant live without each other, we love each other, and with a bit of sex in our relationship it would be the most precfect one!! What can I do??