Another Two Days Lost to Full Mail Server
In the ongoing battle between email and disk space in our company, I lost another round. Our users managed to fill up all the disk space on one node of our mail server cluster this past weekend, and I’ve spent all day trying to get things back to normal. All the wonderful telephone calls from irate users are really motivating. Probably about as motivating to me as all the requests we make to them to please keep their mailboxes cleaned up. It’s amazing how much down time a full disk can cause. Mail servers consist of various components that handle messages in different layers. When one component’s disk space fills up the messages start queuing at a higher level in the system, which causes other bits of disk to start rapidly filling up, and server components to start thrashing. That slows everything down, even on the servers that aren’t full.
It’s fun and games until everything sorts itself out. The most challenging part is trying to explain to users why they can’t have infinite storage for email, because they think disks are so cheap. They don’t see that high-performance fibre-channel disks are an order of magnitude more money than disks for their cheapo home PC, and we also have to pay for equipment to back everything up.
I should have everything back to normal late tonight, with a bit of additional space available to stave off the next crash until we have a working automatic archiving system.