My boyfriend doesn't trust me...(LONG)?

A little history/backstory: he and I met and began dating only about two months ago, but things moved fairly quickly and we both fell for each other very hard. We have so much in common, but at the same time, we are very different. I felt comfortable enough with him to tell him details of my past, and he's said things to me out of anger that make me think that he is judging me for my past, but he'd apologize and things would be okay. Now, he doesn't trust me. I've never cheated on any boyfriends or anything, but HE has been married before, and he recently disclosed to me that he did cheat on his wife, and even though they tried to work things out afterward, she never trusted him and was constantly at him about what he was doing or where he was, etc. THIS IS THE SAME WAY HE TREATS ME NOW. I can't do ANYTHING right. For example, I work nights, and even though I usually wake up when he calls me during the day, on one particular day, he'd called me and I didn't pick up. He made a huge deal out of that. There have been several other similar incidents, but now, tonight, he's tripping again because he called my cell phone while I was at work and I didn't answer. He knew I was at work, and on any other night, he'd call my work number. Tonight was particularly busy and I couldn't answer my cell phone, but I did text him and tell him that it was super busy and that I'd call him back when things slowed down a little. When I tried to call him about an hour later, he didn't answer, and I text him again. He finally text me back at about midnight and made a smart-*** remark like, "Oh, NOW you answer the phone. It wasn't that busy up there." How the hell does he even know HOW busy it was because he didn't even call my work number, he called my cell, and if he HAD called my work number, he'd have known it was busy because he'd have been on hold for a while before he was able to get through. Bottom line is, I'm tired of this. I love him dearly, I really do, but I can't take much more of this. I'm always wrong, can't do anything right, and I've tried to talk to him about this but he always brushes it off. Others I've talked to have told me to just let it go, but I can't bring myself to do it because I really care about him. This is so hard.