Scott’s Blog

I’m building a corporate email archiving system for work. It will basically clean out user’s GroupWise accounts into an archive storage system as messages age. We haven’t established firm policies yet about which messages of which age will be migrated into the archive system. First, I’m building a working system that we can demonstrate in our suggested configuration to the managers’ committee, so that they can see what it will look like and set the policies.

To demonstrate the system, I was reluctant to use my own email account, because while I’m working with the system, I don’t want to accidentally bugger up my own email, and also because my mail load doesn’t look much like a typical user’s mail, since I’m not working much as a project engineer. I set up a testing user called Archivist instead.

Next, I needed to get a bunch of mail into the account so that I could demonstrate the archive system in action. Luckily I have a ready source of endless emails. I just reconfigured our Barracuda spam firewall to quarantine all spam into a specific account instead of deleting it, and pointed that account at my test Archivist mailbox. After a few hours, I had hundreds of test messages at my disposal. A quick review to clean out any x-rated subject lines, and I was good to go.

I recently switched one of my machines to Ubuntu 7.10 64-bit edition. I use 64-bit Linux on my desktop so I can run VMware Server and use 64-bit guest machines.

We use GroupWise for email and calendaring, and I’ve never had anything but brutal performance out of Evolution in my GroupWise account so I decided I wanted to use the real GroupWise client. Unfortunately, it’s a 32-bit rpm package built for SLED and there’s no deb package of it, and no 64-bit build.

To get it working on 64-bit Ubuntu took a few steps. First, I needed a deb package, and GroupWise comes as an rpm, so I installed “alien” which is a utility that can convert between package formats.

sudo apt-get install alien

I tried using alien on my 64-bit machine to convert the rpm to a deb like this:

sudo alien -c novell-groupwise-client-7.0.2-20060607.rpm

This resulted in a “wrong architecture” error like this:

dpkg-gencontrol: error: current build architecture amd64
does not appear in package’s list (i386)

Since I couldn’t figure out how to make alien try to build an i386 deb package on my x64 machine in 30 seconds, I gave up, used scp to copy the rpm file to the i386 machine next to me in my office, and used alien on it to convert the rpm to a 32-bit deb package. Then I copied the resulting deb package back to the 64-bit machine.

Next, I knew I needed a 32-bit Java environment for the GroupWise client. Ubuntu has a lot of 32-bit packages built for the 64-bit platform under the deb package moniker of ia32-xxx.deb. The 32-bit java package I needed was ia32-sun-java6-bin, which provides a 32-bit java6 jre.

sudo apt-get install ia32-sun-java6-bin

Next, I tried installing the GroupWise deb I created before using dpkg:

sudo dpkg -i novell-groupwise-client-7.0.2-20060607.deb

The install failed complaining of missing libraries libstdc++5, libasound2 and libgcc1. I removed GroupWise, installed the dependencies, and reinstalled GroupWise. This time it installed.

sudo apt-get remove novell-groupwise-client
sudo apt-get install libstdc++5 libasound2 libgcc1
sudo apt-get install novell-groupwise-client-7.0.2-20060607.deb

After that, I had to change the jre that GroupWise comes with for the official Sun jre I installed earlier. To do that, I just went like this:

sudo rm -r /opt/novell/groupwise/client/jre
sudo ln -s /usr/lib/jvm/ia32-java-6-sun/jre \
/opt/novell/groupwise/client/jre

The next step was to try running the client using /opt/novell/groupwise/client/bin/groupwise.sh which of course failed. This time the error message indicated a missing library ksuperwin.so and some libgtk and libglib components. Some googling indicated that I needed libglib1.2 and libgtk1.2. Remembering this is a 32-bit application meant I needed the 32-bit versions of these libraries. Unfortunately, there are no ia32 packages of libgtk1.2 or libglib1.2 in the Ubuntu 64-bit repositories.

I had to go to Ubuntu’s web repository for Ubuntu 7.10 and download libgtk1.2 and libglib1.2 from here and here respectively. I needed the 32-bit versions, not the 64-bit versions.

I saved the two deb files locally, and then unpacked them into a temporary directory and copied the libraries to the 32-bit library directory.

mkdir temp
dpkg -X libglib1.2_1.2.10-17build1_i386.deb temp
dpkg -X libgtk1.2_1.2.10-18_i386.deb
sudo cp temp/usr/lib/* /usr/lib32/

At that point, all the library dependencies were settled, and I was able to run GroupWise without problems. To get a GroupWise icon on my desktop, I just copied the .desktop file to the ~/Desktop folder:

cp /opt/novell/groupwise/client/gwclient.desktop ~/Desktop/

Good luck if you are trying this yourself.

Here’s screenshot-proof that you can run the GroupWise cross-platform client on Ubuntu Linux 7.10 on an x86_64 kernel.

Also, notice my empty inbox, which I manage using Merlin Mann’s Inbox Zero strategy.

See if you can tell the difference between these two algorithms:

For each new email I receive, do this:
if the sender is in my address book,
create a reply to the sender with a canned message,
send it out.
done

Pretty simple, eh? Now how about this one:

For every new or old item in my entire mailbox, do this:
create a new canned message,
add everyone in my address book in the “To” field
send it out
done

Interestingly, the first one produces a fairly well behaved Vacation rule. The second one produces 133,000 messages per hour in our mail system until you delete the offending account after 90 minutes.

By the way, it takes about 10 hours to delete 200,000 email messages from three GroupWise post offices using gwcheck with the itempurg=subject switch.

This post is based on a true story.

Now that our GroupWise 7 deployment is completely moved onto Linux servers with Solaris back-ending the storage, I have finally got a consistent backup and restore solution that allows me to recover accidentally deleted users without pulling out all my hair.

Our GroupWise post office directories live on Solaris zfs partitions exported to the Linux post office servers via nfs. When I want to back them up, I stop the poa for five seconds and do a zfs snapshot. Then I use zfs send and zfs receive to replicate the snapshot to another Solaris box with oodles of SATA disks. I snapshot the replica every day and keep daily snapshots for 60 days. The snapshots are very space-conservative so it’s relatively cheap to do this.

When one of my administrators deletes a GroupWise account that he or she shouldn’t have, I can use this system to recover the deleted accounts quickly. On the Linux post office servers, I created /mnt/restore/po directories for each post office. Then using ConsoleOne, I defined restore areas pointing to those directories.

When I have to restore a user’s data that was deleted on a certain day, I do a zfs clone operation on the backup server to clone the zfs snapshot made the day before the user was deleted. Then I use zfs set sharenfs= to export the clone via nfs and make it mountable on the Linux post office server. Back on the Linux post office server, I mount the exported post office under /mnt/restore/po, and I now have a working restore area for that day. Choosing a different day is as easy as doing a new zfs clone, exporting it, and unmounting and remounting on the post office server.

If I have to restore a deleted user, instead of just deleted mail, I just have to do the same clone, export and import operation with the GroupWise primary domain directory backup, and then use ConsoleOne, select Restore Deleted User, browse to the mounted clone, and pick the user to restore. Then I follow the above procedure to get the data back for that user.

This is so much easier than dealing with our former backup service provider, where I would have to get a full tape restore done, see if the user was on it, wash, rinse and repeat. That process sometimes took days if the administrator wasn’t sure which day the deletion happened on.

Thanks to a lot of hard work by Denys, and no thanks to Novell support, we’ve finished moving all our GroupWise users to our new GroupWise server architecture.

Our old system consisted of six post offices hosted on a two-node NetWare cluster using Novell Cluster Services. There were six post offices as part of a history that included corporate political influences that helped create a functional but sub-optimal technical design. We had an opportunity to upgrade to GroupWise 7, migrate to new hardware, and consolidate to an architecture that included one post office per server on three servers, all in one operation. We built new servers, consisting of three virtual machines running SLES9, hosting one post office each, but with the storage for those post offices held on a fourth server running Solaris, with the post office data on ZFS filesystems, and remote-mounted on the SLES boxes. The beauty of this is that we can do backups with about 10 seconds or less of downtime per post office per day, and we can do speculative changes with near-instant rollback if the changes do anything unexpected. ZFS is your friend.

Anyways, the thanks for Denys is for all the effort he put in to move more than 600 users, many resources and several distribution lists from the six old post offices to the three new ones, and dealing with the aftermath of broken passwords that ensued.

The no-thanks to Novell support is for the lack of help fixing a bug that causes mailbox caching passwords to break when you move a user from a NetWare-created GroupWise post office to a Linux-hosted GroupWise 7 post office. Despite us filing a Premium Support ticket, and Novell recognizing a bug (GroupWise defect 239947) and refunding our support ticket, they never fixed the bug, and after months of us waiting, they even stopped responding for our requests for a status update on the bug. GroupWise 7 SP2 came out without the bug fixed, and we had to proceed and fix 600 broken passwords manually. Hence the ensuing aftermath.

Anyways, we’re finally done with the migration, and our new architecture is much more scalable than the former one. I expect that by adding post office virtual machines on additional blades, plus adding storage management Solaris blades as required, we’ll be able to scale up to several thousand users, which should work for us for the foreseeable future.

In the ongoing battle between email and disk space in our company, I lost another round. Our users managed to fill up all the disk space on one node of our mail server cluster this past weekend, and I’ve spent all day trying to get things back to normal. All the wonderful telephone calls from irate users are really motivating. Probably about as motivating to me as all the requests we make to them to please keep their mailboxes cleaned up. It’s amazing how much down time a full disk can cause. Mail servers consist of various components that handle messages in different layers. When one component’s disk space fills up the messages start queuing at a higher level in the system, which causes other bits of disk to start rapidly filling up, and server components to start thrashing. That slows everything down, even on the servers that aren’t full.

It’s fun and games until everything sorts itself out. The most challenging part is trying to explain to users why they can’t have infinite storage for email, because they think disks are so cheap. They don’t see that high-performance fibre-channel disks are an order of magnitude more money than disks for their cheapo home PC, and we also have to pay for equipment to back everything up.

I should have everything back to normal late tonight, with a bit of additional space available to stave off the next crash until we have a working automatic archiving system.

I wrote before about implementing internal mailing lists in our company using GNU Mailman. That never got too far because of the user interface. Our test group of users didn’t like it. Since then I played with a few other mailing list managers, but having been very busy, I didn’t get anything implemented for real. We’ve also been having problems with our GroupWise upgrade and simultaneous migration to new mail server hardware, for which I’ve opened a support ticket with Novell. Because of this the list manager server went onto the back burner.

Recently in my blog meanderings I came across a mention of Dada Mail, another open-source mailing list manager. I thought I’d give it a try. I’ve also been thinking that a mailing list manager would make a decent virtual machine appliance, so I’ve decided that in order to satisfy our corporate need and provide some give-back to the open source community, I’d try to learn about rPath Linux, a special Linux distribution for building software appliances, and rBuilder, a free service to help you bundle applications as application server appliances. I don’t know how much time I’ll have to put to this, but right now I’m thinking I’m going to try to build a virtual appliance that uses Dada Mail as a mailing list manager and is mostly pre-configured right out of the box. If it works, and provided the Dada Mail maintainer is cool with it, I’ll make it available on rBuilder Online.

I have been trying and trying to get to like a Blackberry as my phone and organizer. I got the thing at Brainshare last year. I have failed miserably, and I’m going back to a regular non-smart phone and separate Palm organizer. I’ve used Palms for 7 years, and I just find them easier to use than Blackberry. I also like using my Palm as an e-reader. I got a new Tungsten T/X, which has a big screen and built-in wi-fi. The included Versamail email software works great with GroupWise in IMAP mode. Not to mention, my neighbor has a nice unsecured wireless access point and broadband, which makes it easy to get mail.

We’ve been working with Novell on a support ticket regarding problems we’ve been having with moving GroupWise users to new post offices on Linux from old ones on NetWare in GroupWise 7. Everything goes fine, except that the user has to re-enter the online password in the caching GroupWise client before it will sync up after the move. This is annoying because the password is buried several clicks deep in the GroupWise client, and is beyond some of our users to pull off reliably. That means we’d have to visit a lot of desks to get the move done, and we have nearly 600 GroupWise mailboxes (with about 520 left to move).

Novell couldn’t duplicate the problem in their labs. They asked for move logs, which I provided. They couldn’t determine the problem from that, so they asked me to copy a user database before and after the move and send those to them. I have to find a willing user to move, backup and move their account, have it exhibit the problem, and then collect the moved database. So far I haven’t been able to meet those criteria. With it being so close to Christmas I have a feeling I’m going to have to wait until January before I can provide Novell with the next data they need to work on the problem.