I need advice about a issue, somebody please tell me if Im crasy?

My BF & I were making love & for some reason he couldnt .I worried that I was doing something wrong, so I asked.What he told me upset me so badly I freaked out & started crying.He says I overreacted,I say that no one in their right minds would have TOLD their lover this,even if it was true,some things you just dont reveal out of common decency. Here is what he said: "I knew I wasnt going to be able to (O) & I didnt think it was right to fantasize about someone else to make it happen when Im w/you." I could have gone my whole life not knowing this. I think he's horribly insensitive to say this,even if it was true.We all fantasize about other people than our lovers,but thats not a thing you would TELL them.Hes been open about his ual fantasies w/me & tells me he masturbates frequently to diff. 'topics'.I have no problem with that. I didnt need to hear this in so close conjunction with ME. He says he loves me and that Im irrisistable, we have great but this makes me feel bad.