How do I handle my husband's issues with his confidence?

I asked about how to regain my husband's trust yesterday. My husband found out about a sort of flirtation I had with a guy at my last job. We've moved away but he found out about it just before the move. Nothing physical happened between me and my colleague and I have no emotional interest or attachment in him. Just some flirting and talk for about 10 months. My husband says I lied by not telling him what was going on, and then I lied some more when I didn't tell him the truth when he started to suspect things. He doesn't trust me I can tell and I know I've hurt him. I think I made it worse by explaining that I had no emotion for my male friend, just physical attraction but my best friend says men never want to hear this. I talked with him last night and my husband says that I have hurt him with comments I've made over the past months. I said he wasn't a good provider and that he isn't really my type at all. I admit this, and that I said he isn't the kind of man that most women think of as the most attractive. But he is attractive to me for other reasons though he has forgotten this. He also says I make conflicting demands like wanting a house and children now (which I do) but also talking about how I miss the days when I could let a man pick me up and have NSA sex that night (which I also miss and I feel like I'm getting older but stuck in the same rut). I guess the sum of it is that he feels both that he can't trust me and that maybe I've shot his ego up a little. How do we get past this? Why does he hold it against me so much--I'm tired of him sort of looking sad every so often when he thinks I'm not looking and I got angry with him this morning for doing it? When can I say he needs to man up and get over it?