My older son was always grouchy in the evenings when he was an infant, too. I have fond memories now of our nightly "dance parties" where we'd dance around the living room while my wife exercised upstairs. Trust me, man; even the difficult stuff will become wonderful memories. :D
You are right. Some how many of those tough memories at the NICU are now fond memories. Never thought that would happen.