You weren't even with your mistress that much...
"This is not a mayor or a governor or a president who's sitting in an ivory tower. I was at Ground Zero as often, if not more, than most of the workers," Rudy Giuliani
According to Salon.com:
By our count, Giuliani spent about 58 hours at Yankees games or flying to them in the 40 days between Sept. 25 and Nov. 4, roughly twice as long as he spent at ground zero in the 90 days between Sept. 17 and Dec. 16. By his own standard, Giuliani was one of the Yankees more than he was one of the rescue workers.