My boyfriend (32m) came to my apartment (29f) unannounced, and found me with another man (boyfriend's brother)

I (29f) happened to date my boyfriend's (32m) brother (28m) for a hot minute 2 years ago, but it didn't work out. We both moved on with our lives and I met my boyfriend around 10 months ago, while his brother was just a memory of a relationship that never worked out. I started seeing my boyfriend and things were going well, and I was so excited to tell a friend that I had met someone and I was super excited, and my friend was like "cool! but what about his brother?" Like a lot of women pass their boyfriends around to their friends if that makes sense? I said no he dates an insane amount and I don't think he would be interested in you. But the thing is, my ex boyfriend/bf's brother happened to hear me say a lot of guys within my friend circle, and I didn't know this, but he was looking for a booty call, and I was a woman that he had already conquered and he knew I was single. So he reached out to me and I didn't text him back for a while, I liked the idea of having sex with the "bad boy brother" and the excitement of it all was nice. But then my boyfriend and I got more serious and I felt bad talking to his brother so I told the brother I was back in a relationship and he texted back saying "so??". And I don't know, I was down. I guess it was the excitement of it all, and he just came to my apartment, and we had sex once. I texted him afterwards and said it was a mistake and that I couldn't do it anymore. He just said "I get it, if you change your mind let me know" I did, and he came back, and we ended up having sex again, only 2 times in total. But the thing is, he lives 5 minutes away from me, so it was easy for him to come, and my boyfriend and I live 25 minutes away, so it's not easy for him to stop by. I didn't really think about it, until the boyfriend said he was heading out of town for work, and he said he would stop by my place beforehand, but he ended up coming over a day before he said he would, only a few hours after he said he would, and caught my door open, to the brother with his pants down, and he was livid. He told me he was done with me, and he needs space to think about if he wants to work things out or not. And I understand why he would be upset, it was a mistake, and I shouldn't have done it, but it was only 2 times, and I never promised to be faithful, but it's not an excuse, and I feel bad and guilty, and I wish I could turn back time and never do it. But now I'm in this situation where I have to wait for him to decide what he wants to do, and it sucks because I can't do anything to fix it, and I have to wait for him to figure out how he feels about it all. What can I do?