I think that what you are going through can be normal in any marriage especially when dealing with something as stressful and TTC. I would definately stop for a while and put focus back on your marriage. You may also want to seek counseling for yourself.
I wouldnt tell your DH because nothing happened and you will only hurt him by telling.

__________________