All-nighter

We shifted over from our old Border Manager-based VPN to a new WAN infrastructure mostly provided by our ISP. It was one of the biggest single rounds of changes I have ever done to our network infrastructure. It went very well, down to the very last server.

Then that very last server bit me in the ass. It was our primary SMTP server, the gateway where almost all our external email comes in and out from the Internet. I made a simple change, to the server’s default gateway. I tried to get it to kick in but the smtp gateway agent crashed. I shut down and restarted the server, and the gateway would no longer run. I tried reinstalling it but that corrupted it worse, and broke the web access agent too. I ended up having to reinstal the whole gateway server, which took until 03:30. I got to bed as the sun was coming up.

Today things were working s-l-o-w-l-y. We discovered we had several instances where the Cisco routers from the ISP had autonegotiated conflicting network speed and duplex settings with our HP routers. With the help of our partner Dom Amyotte from Greycon Consulting, we got everything working. The users were skeptical this morning, but now they seem pretty happy.