Submitted a support request to Novell for GroupWise

Well, I couldn’t resolve the password-mangling problem we’re having with GroupWise user moves from NetWare to Linux servers, so I submitted a tech support ticket today to Novell Support using our premium support account. Hopefully they can help us with this.

Problems with GroupWise User Moves

We’ve built a new set of GroupWise servers. They are running on SUSE Linux servers, and our existing GroupWise servers run on NetWare. We are planning to live-move the user accounts from the NetWare servers to the Linux servers. Both the NetWare servers and the Linux servers are running the same version of GroupWise 7, SP1.

I started moving some accounts experimentally last week, and had some problems. The users who had been moved were not able to login to their mail accounts until we went in as the administration user, and changed their password. I struggled with that for a couple of days, and asked James for help. He found that there were some eDirectory operations that were stuck, and clearing those out helped the problem with GroupWise passwords somewhat. After fixing the directory, moving GroupWise users now only breaks the caching copy of the online password, but not the actual online password, so users who have been moved can still login to online mode and WebAccess, but not in caching mode. They can fix their own accounts by going into the caching mailbox, and setting the online password in the cache to match the online password using the Accounts preferences dialog box.

I don’t really want to impose that requirement on the users, because a lot of them would struggle with it, so we’re going to work some more to see if we can’t get the process to be more seamless.

GroupWise 7 Progress

I’ve finished building the four new GroupWise servers, which are VMware virtual machines in our IBM bladecenter. Three of the servers will be post office servers where users’ mailboxes reside, and the fourth will be a gateway server with the GroupWise Internet agent, Monitoring agent and WebAccess agent. I already have the post offices built, and I have successfully transferred a couple of users into one of the new post offices. Once I have the gateway done, I’ll start moving users over en masse.

We’re using an interesting approach to storing our mail in order to allow mininmum downtime backups and to ease the act of doing backups and restores. I’ve configured a blade running Sun Solaris, as a file server that serves storage to the post office severs, which are running SLES 9. The Solaris server is configured with three ZFS filesystems, one for each post office, and it serves those via NFS. The SLES 9 GroupWise post office servers then mount the storage over NFS and run the post offices from there. It’s slightly slower than using native SAN partitions for GroupWise, but it allows us to host the mail on ZFS, which I’ve raved about previously. To do a post office snapshot, using this approach, takes seconds. A simple shell script like this allows a snapshot to be made with minimal GroupWise downtime and no specialized backup tools.

/etc/init.d/grpwise stop po1.poa
ssh -i ssh_key storageserver "zfs snapshot data/po1@snapshot"
/etc/init.d/grpwise start po1.poa

Once this snapshot completes, GroupWise is up and running and we have a complete coherent snapshot that we can synchronize to our disk-to-disk backup server, while users continue to use the system unimpeded. It works great.

GroupWise 7 Update Progress

Last night I started working on our GroupWise 6.5 to GroupWise 7 upgrade. We are moving from a two-node NetWare cluster running GroupWise 6.5 with a third box running WebAccess and the Internet agent to a four node configuration on IBM Blades with SAN storage running SUSE Linux Enterprise Sserver 9 and GroupWise 7 virtualized under VMware hosted on SLES9.

The first step was to update the existing GroupWise back-end running on NetWare to version 7. That will allow me to create new GroupWise domains and post offices on Linux and add them to the existing system. Then I will be able to live-move the users from the NetWare post offices to the Linux ones, and finally decommission the NetWare post offices. That’s the plan.

The update of the back-end stuff from GroupWise 6.5 on NetWare to GroupWise 7 SP1 on NetWare took place last night and was entirely successful. The only glitch was that when I sent out a notice to the users at the end of the process early this morning using my Evolution client, the message got stuck somehow in my outbox and got sent to everyone three times. I just told them all that we had made our GroupWise system three times more reliable.

GroupWise 7 Deployment Preparation

I’m getting ready to deploy new servers for GroupWise 7, updating our existing GW 6.5 servers running in a NetWare cluster to a set of virtual machines in VMware Server running SUSE Linux Enterprise Server on the Bladecenter, with GroupWise 7.01 for Linux.

The first step, which I’m planning for Tuesday night next week, is to upgrade the back-end NetWare boxes to GroupWise 7 so that the primary GroupWise domain will be version 7, allowing me to then add Linux GroupWise 7 domains to the sysetm. Then, I will create the new GroupWise secondary domains for GroupWise 7 on the Linux virtual machines, create post offices on them, and start migrating users over. The hundred gigabytes of live email on our existing GroupWise systems is going to take a while to move over.

One Email Per Minute Over and Over

I just had a user call me to complain that he was getting the same email over and over and had received over a thousand over night. The message is a legitimate message from an actual employee of one of our business partners. The sender sent the message last night, and we have received it about once every minute since then. I just looked at our spam firewall, and it is showing me that the sender’s SMTP server is connecting to it every minute or so and sending another copy of the message. I blocked the sender’s email address at the spam firewall, so at least my user won’t get anymore. We’re getting in touch with the sender to let them know their mail system is malfunctioning. Hopefully they’ll be able to stop it. Since I set the firewall to block the sender’s mail address a little while ago, the firewall has blocked 22 copies of the message.

It’s very weird, and I’ve never seen it happen before. According to my logs, the sender used Microsoft Outlook, Build 10.0.6626, and there a Postfix server in the sender’s outgoing mail path, but there’s no real indication of where the source of the problem is. Fun and games.

What’s Up With All The Spam?

I don’t know if it is just our company or what, but in the last 30 days, the spam being filtered by our Barracuda spam filter (which is doing a great job by the way) has doubled, from about 5,000 per day in early September to nearly 10,000 per day now. We have about 600 users, so that’s 16 spams per user per day. Luckily for some users and not for others, it’s not evenly distributed like that. Our amount of legit mail hasn’t increased, so I can only conclude that the spammers are on a concerted attack. I wish they would all bugger off and go to jail.

Internal Mailing Lists with GNU Mailman

We’ve been trying to figure out other ways to reduce the maintenance effort required with email. On top of all the other stuff we have related to projects in email, we also have discussions that happen between engineers in the various engineering disciplines we practice. They use GroupWise distribution lists for this, but since there is no easy way to archive all the mail from a distribution list, everybody on the list tends to keep everything they ever get emailed to them, and it just adds up to a lot of unnecessary email. The other hassle with that, is that some person has to maintain the mailing list membership.

An alternative to this is to use a mailing list manager with automatic web archiving, so that users can self-subscribe to a list, and when they get messages from the list, they can delete them after reading them, because the mailing list manager will automatically archive all the messages.

I’ve been working with GNU Mailman for this, on a little Ubuntu Server virtual machine. Ubuntu’s wiki has a simple-to-follow procedure on how to set it up using exim4 or postfix as your MTA. I built the virtual machine using the Ubuntu server install CD, apt-get installed Mailman and Postfix, followed the wiki directions, and voila, I had a nice mailing list server. Then, I just added a pointer in my GroupWise route.cfg in the Internet Agent, and it all worked.

I created a few preliminary groups, and everything seems to work, so now I just have to write up some simple user docos, and create a backup script for the list archives, and we’re off to the races. I love open source software.

Email Back to Normal but…

this week has been exhausting. The email system recovered slowly thorughout the week until it was delivering current messages immediately by Wednesday afternoon, and it finished delivering the backlogged deferred messages by some time today. Users have been complaining, and we have been working to get it stable again, tuning the spam firewal to reduce the load on the actual system as much as possible, and brainstorming about how to get this problem fixed long term.

Users hate dealing with it, and they hate being badgered to clean out old email and file it properly, but they also hate it when the system goes out of service because it’s so stuffed full. If they won’t clean up after themselves, we have to make the system do it for them.

It all comes down to changing our email retention policy. We have to get away from the (everywhere else scorned) idea that we have to keep paper copies of project record-related emails. It works, but it is very labour intensive. Instead, we have to come up with a systematic way of storing them electronically in a robust discoverable way that’s easy to find messages in, and takes minimal work from the end users. Our experience over the past year has shown that users are generally unwilling to take much of a role in managing email. When you look in the mailbox of your biggest email user, who is otherwise very competent and professional, and he has 5000 emails in his inbox, 20% of which are unopened, it becomes clear that user-based management of email is not going to happen.

We’re going to start seeding the idea of an electronic email record retention system within the company, in order to get buy-in from management, and they we’re going to hit them with cost estimates of what we think manual management is costing us now, versus the costs of an elecronic system. Hopefully that will allow us to implement a system that gets rid of the email managment headaches for both our IT staff and our end-users.

Email Broken, Users Angry

We have limited space on our mail server. We have a policy where users are supposed to file any email that’s related to a project, and then delete it off the mail server. They generally don’t do that and the server gets overly full. Friday, it got right full, on both nodes of the cluster, causing mail to stop processing and forcing it to queue up on our backup mail server. This continued over the weekend as we tried to do anything possible to recover some disk space. Once we got things up and running, the backup mail server processed a tonne of mail through to the main system, choking the spam filter, and filling the mail server again.

Then we (repeat first paragraph).

So far the system is bogged down so much that the spam filter keeps crashing (a Barracuda spam firewall) and it’s in a colocation site where we can’t reboot it. The post office servers are so busy trying to deal with the backlog that the users can’t access their mail. The longer it’s offline the bigger the backlog gets, and there’s no end in sight.

I keep getting angry phone calls from users, and all I can tell them is that we’re doing our best to free up space and get things running, and that there’s a reason why we ask them every week to clean out their mailboxes.