Yesterday my network was subject to one of the most difficult to trace network problems i've seen. Basically at 12:45pm, the "internet died", and I rebooted our ISA server, it worked again. About 20 minutes later, it died again, this happened about 2-3 times more, I knew we had a major issue. Microsoft ISA Server is one of microsoft's best products, and especially ISA 2006 SP1 is very reliable. I put a packet trace outside my ISA firewall, it showed packets leaving my network AND returning, however, my ISA server reported that the packets were leaving, and NEVER returning. VERY ODD, ISA doesn't lie. We did the normal replace hardware, we even swapped out a router, switches and the ISA server hardware to no avail. This was very perplexing. Finally after 14 hours of wanting to tear my hair out, we found something in a packet trace. I had captured a ping of google.com when stuff worked, and when it wasn't working. This was just outside our ISA server(which is a back firewall, not a front one)but on the other side of our router (but this router doesn't do any packet filtering, so I ignored this *bad idea*). We noticed that packet responses LOOKED the same, but when we dug deeper we saw the MAC address on the response packets were different, but with the same correct IP. One Mac was accurate when it worked, and different when it wasn't. Someone had created a machine with the same IP Address of our router outside our ISA firewall. This was causing return packets from the internet to be misdirected to a server, and not to our router. So ISA wasn't lying, and doing some Mac Table lookups showed me which switches and ports to chase until I found the culprit rogue machine. After powering it down(and disconnecting the ethernet cable) all the problems stopped and I could finally go home at 3AM.
God I love IT.