We’ve been trying to figure out other ways to reduce the maintenance effort required with email. On top of all the other stuff we have related to projects in email, we also have discussions that happen between engineers in the various engineering disciplines we practice. They use GroupWise distribution lists for this, but since there is no easy way to archive all the mail from a distribution list, everybody on the list tends to keep everything they ever get emailed to them, and it just adds up to a lot of unnecessary email. The other hassle with that, is that some person has to maintain the mailing list membership.
An alternative to this is to use a mailing list manager with automatic web archiving, so that users can self-subscribe to a list, and when they get messages from the list, they can delete them after reading them, because the mailing list manager will automatically archive all the messages.
I’ve been working with GNU Mailman for this, on a little Ubuntu Server virtual machine. Ubuntu’s wiki has a simple-to-follow procedure on how to set it up using exim4 or postfix as your MTA. I built the virtual machine using the Ubuntu server install CD, apt-get installed Mailman and Postfix, followed the wiki directions, and voila, I had a nice mailing list server. Then, I just added a pointer in my GroupWise route.cfg in the Internet Agent, and it all worked.
I created a few preliminary groups, and everything seems to work, so now I just have to write up some simple user docos, and create a backup script for the list archives, and we’re off to the races. I love open source software.
this week has been exhausting. The email system recovered slowly thorughout the week until it was delivering current messages immediately by Wednesday afternoon, and it finished delivering the backlogged deferred messages by some time today. Users have been complaining, and we have been working to get it stable again, tuning the spam firewal to reduce the load on the actual system as much as possible, and brainstorming about how to get this problem fixed long term.
Users hate dealing with it, and they hate being badgered to clean out old email and file it properly, but they also hate it when the system goes out of service because it’s so stuffed full. If they won’t clean up after themselves, we have to make the system do it for them.
It all comes down to changing our email retention policy. We have to get away from the (everywhere else scorned) idea that we have to keep paper copies of project record-related emails. It works, but it is very labour intensive. Instead, we have to come up with a systematic way of storing them electronically in a robust discoverable way that’s easy to find messages in, and takes minimal work from the end users. Our experience over the past year has shown that users are generally unwilling to take much of a role in managing email. When you look in the mailbox of your biggest email user, who is otherwise very competent and professional, and he has 5000 emails in his inbox, 20% of which are unopened, it becomes clear that user-based management of email is not going to happen.
We’re going to start seeding the idea of an electronic email record retention system within the company, in order to get buy-in from management, and they we’re going to hit them with cost estimates of what we think manual management is costing us now, versus the costs of an elecronic system. Hopefully that will allow us to implement a system that gets rid of the email managment headaches for both our IT staff and our end-users.
We have limited space on our mail server. We have a policy where users are supposed to file any email that’s related to a project, and then delete it off the mail server. They generally don’t do that and the server gets overly full. Friday, it got right full, on both nodes of the cluster, causing mail to stop processing and forcing it to queue up on our backup mail server. This continued over the weekend as we tried to do anything possible to recover some disk space. Once we got things up and running, the backup mail server processed a tonne of mail through to the main system, choking the spam filter, and filling the mail server again.
Then we (repeat first paragraph).
So far the system is bogged down so much that the spam filter keeps crashing (a Barracuda spam firewall) and it’s in a colocation site where we can’t reboot it. The post office servers are so busy trying to deal with the backlog that the users can’t access their mail. The longer it’s offline the bigger the backlog gets, and there’s no end in sight.
I keep getting angry phone calls from users, and all I can tell them is that we’re doing our best to free up space and get things running, and that there’s a reason why we ask them every week to clean out their mailboxes.