Scott’s Blog

I recently switched one of my machines to Ubuntu 7.10 64-bit edition. I use 64-bit Linux on my desktop so I can run VMware Server and use 64-bit guest machines.

We use GroupWise for email and calendaring, and I’ve never had anything but brutal performance out of Evolution in my GroupWise account so I decided I wanted to use the real GroupWise client. Unfortunately, it’s a 32-bit rpm package built for SLED and there’s no deb package of it, and no 64-bit build.

To get it working on 64-bit Ubuntu took a few steps. First, I needed a deb package, and GroupWise comes as an rpm, so I installed “alien” which is a utility that can convert between package formats.

sudo apt-get install alien

I tried using alien on my 64-bit machine to convert the rpm to a deb like this:

sudo alien -c novell-groupwise-client-7.0.2-20060607.rpm

This resulted in a “wrong architecture” error like this:

dpkg-gencontrol: error: current build architecture amd64
does not appear in package’s list (i386)

Since I couldn’t figure out how to make alien try to build an i386 deb package on my x64 machine in 30 seconds, I gave up, used scp to copy the rpm file to the i386 machine next to me in my office, and used alien on it to convert the rpm to a 32-bit deb package. Then I copied the resulting deb package back to the 64-bit machine.

Next, I knew I needed a 32-bit Java environment for the GroupWise client. Ubuntu has a lot of 32-bit packages built for the 64-bit platform under the deb package moniker of ia32-xxx.deb. The 32-bit java package I needed was ia32-sun-java6-bin, which provides a 32-bit java6 jre.

sudo apt-get install ia32-sun-java6-bin

Next, I tried installing the GroupWise deb I created before using dpkg:

sudo dpkg -i novell-groupwise-client-7.0.2-20060607.deb

The install failed complaining of missing libraries libstdc++5, libasound2 and libgcc1. I removed GroupWise, installed the dependencies, and reinstalled GroupWise. This time it installed.

sudo apt-get remove novell-groupwise-client
sudo apt-get install libstdc++5 libasound2 libgcc1
sudo apt-get install novell-groupwise-client-7.0.2-20060607.deb

After that, I had to change the jre that GroupWise comes with for the official Sun jre I installed earlier. To do that, I just went like this:

sudo rm -r /opt/novell/groupwise/client/jre
sudo ln -s /usr/lib/jvm/ia32-java-6-sun/jre \
/opt/novell/groupwise/client/jre

The next step was to try running the client using /opt/novell/groupwise/client/bin/groupwise.sh which of course failed. This time the error message indicated a missing library ksuperwin.so and some libgtk and libglib components. Some googling indicated that I needed libglib1.2 and libgtk1.2. Remembering this is a 32-bit application meant I needed the 32-bit versions of these libraries. Unfortunately, there are no ia32 packages of libgtk1.2 or libglib1.2 in the Ubuntu 64-bit repositories.

I had to go to Ubuntu’s web repository for Ubuntu 7.10 and download libgtk1.2 and libglib1.2 from here and here respectively. I needed the 32-bit versions, not the 64-bit versions.

I saved the two deb files locally, and then unpacked them into a temporary directory and copied the libraries to the 32-bit library directory.

mkdir temp
dpkg -X libglib1.2_1.2.10-17build1_i386.deb temp
dpkg -X libgtk1.2_1.2.10-18_i386.deb
sudo cp temp/usr/lib/* /usr/lib32/

At that point, all the library dependencies were settled, and I was able to run GroupWise without problems. To get a GroupWise icon on my desktop, I just copied the .desktop file to the ~/Desktop folder:

cp /opt/novell/groupwise/client/gwclient.desktop ~/Desktop/

Good luck if you are trying this yourself.

Here’s screenshot-proof that you can run the GroupWise cross-platform client on Ubuntu Linux 7.10 on an x86_64 kernel.

Also, notice my empty inbox, which I manage using Merlin Mann’s Inbox Zero strategy.

I have an IBM Thinkpad from 2003 or so. It came with Windows XP on it. When I got it, I booted it once and went ooh, XP sucks, and then reformatted it and installed some distribution of Linux on it. It was probably Fedora. For most of the time I’ve had it, it has either had Novell Linux Desktop 9 or Ubuntu on it. I also have a desktop with OpenSUSE 10.2 on it, and I figured it would be handy to have a real Windows box instead of a virtual machine, to run the odd Windows application on.

I installed Windows XP from our corporate disks onto my Thinkpad, and used the license key on the sticker that came with the laptop when it asked for a key. I also downloaded the dozen or so driver files that are required because Windows doesn’t include most hardware drivers. I got the thing all set up and I had started installing my applications on it, while it ran Windows Update in the background. After a couple of Windows Updates were installed and rebooted, it installed Windows Genuine (dis)Advantage. Then it said that this copy of Windows wasn’t legitimate, and offered to let me buy Windows for about $200. I stewed at it for a few minutes, said fuck it, and went back to Ubuntu. I hate Microsoft.

We are deploying Deltek Vision 4.1 as our new financial management system in March. We started work a while back on this project. I built the infrastructure in VMware Server running on top of SuSE Linux Enterprise Server 9. We are using a three-box Vision implementation, with a separate VM running Windows 2003 Server Standard Edition on dedicated VMware Server hosts for each of Vision Web, Vision Reporting and Vision SQL Server. The virtualization is to allow for disaster recovery and portability of hardware. The database analyst and programmer guys got started quite a while ago getting the reports that our project managers rely on in our old system working in Vision. We’ve also been testing and troubleshooting Vision and training the accounting staff during this time.

A problem started manifesting itself with Vision and SQL Server after some of the data was imported into SQL Server and we started doing queries on it. The problem would occur particularly often whenever nested select statements were used in a query. SQL Server would fail to execute the query and error out with one of four different errors: error 5243, 5242, 823, or 682. All of these errors have multiple meanings, but a common thread is I/O problems to do with physical disks or storage drivers, when SQL Server does lots of writes in TEMPDB. In our case because we are using SQL Server on a virtual machine, it implied some kind of problem with virtual disks or with the VMware virtual storage controller driver, or possibly with the underlying filesystem on the VMware host.

Several VMware Server knowledgebase and discussion posts mentioned similar problems regarding SQL Server 2000 and SQL Server 2005 on VMware Server.

To confirm that the problem was a VMware problem, which was just a suspicion initially, I built a physical Windows 2003 Server that was otherwise identically configured to the virtual one. On the physical server the queries never failed in our tests.

That made us fairly confident that we had a problem with VMware Server somewhere. When I initially created the virtual machine, I built a reiserfs partition on our IBM DS4300 SAN to store it. When I built the VM, I created a 100 GB virtual disk that was configured in 2 GB chunks, and I did not preallocate all the space at build time. I thought that perhaps the I/O problems were occurring when the VM writes the TEMPDB and new storage was allocated as the virtual disk expanded. I decided to convert the virtual disk to a fully preallocated disk using vmware-vdiskmanager, which is a command-line tool that comes with VMware Server. I did that conversion and then tested the new VM on non-production hardware that was very similar to the production blade server, except that it had locally attached storage instead of a SAN LUN. The problem almost went away. It went from erroring out more than half the time to erroring out about once in 15 or 20 runs. That indicated that I was on the right track.

We had a momentary lapse of reason and thought that the VM might run better on a Windows VMware Server host. I moved the 100 GB preallocated disk version of the VM to a Windows XP Pro workstation running VMware Server. The error occurred nearly every time, so we abandoned that ill-conceived path.

Next, I thought that either reiserfs couldn’t cut it, or VMware Server couldn’t cut it. Since I had just received a new workstation from the vendor, I configured it with OpenSUSE 10.2 and formatted the disk in ext3. I also built an ESX Server 3 evaluation server in Engineering, on an IBM x334 pizza box connected to a fibre-channel SAN. I copied the 100 GB VM to both my new workstation and the ESX Server.

On my workstation, the moderately demanding test query ran 50 times in a row without failing until I gave up on it, proving out that ext3 works better as an underlying filesysetm for VMware Server, at least when the VM you are hosting is Windows 2003 Server with SQL Server 2005.

On the ESX server, the query also worked every time, which was fully expected.

Finally, I decided that even if the ext3 and preallocated disk fix didn’t fix the problem 100% of the time, it was worth applying it to the production system, so that the problems would be reduced during training. It would also buy us time to decide whether or not to buy ESX Server for about $10,000CDN including one year of support.

I shut down the production Vision database server after hours. I moved the existing non-preallocated VM off the SAN LUN that it used for production. Then I reformatted the reiserfs partition on the SAN LUN to ext3. After the format I was surprised to find that the available space was smaller than it was with reiserfs. I had to resize the SAN LUN up a few gigabytes to allow me to convert the non-preallocated disk to a fully preallocated one back on the SAN LUN. After the resize, I recreated the ext3 partition and used vmware-vdiskmanager to convert the non-preallocated disk to a preallocated one. The VM booted and ran fine after the disk conversion.

On the converted production VM, all errors appear to have ceased and performance may have improved slightly as well. We have decided to proceed to deployment on VMware Server using this configuration.

Take Away Points

• The problem referenced in this article occurs on SQL Server 2000 and SQL Server 2005. We discovered this after the fact while working on something else.
• It is a good idea to run VMware Server on Linux, not on a Windows host.
• It is a good idea to use ext3 instead of reiserfs as the filesystem to store your virtual machines. Other Linux filesystems might be suitable as well, but were not tested.
• Filesystems formatted with ext3 use more space for overhead than reiserfs.
• VMware Server is similar in performance to VMware ESX server for Windows 2003 virtual machines running SQL Server under light to moderate loads.
• In the future I will try very hard to not have to move a 100 GB virtual machine all over my network. It takes a long time to repeatedly move 100 GB worth of files from system to system. (duh!)
• When working with troubleshooting on large virtual machines, it is great to have lots and lots of fast storage nearby on the network. Speculative changes are much less hair-raising if you have lots of room to backup your virtual machines.
• You can do awesome stuff with virtual machines that you just can’t even consider unless you have lab hardware coming out your ears and an army of lab monkeys to help you.

My new Sun Ultra 20 M2 workstation arrived yesterday. It came preinstalled with Solaris, which as awesome as it is, I am not able to use. I need to be able to run VMware Server for my work, which means Linux, not Solaris (at least right now). I booted Solaris and poked around a bit just to satisfy curiosity, and then popped in my OpenSUSE 10.2 DVD and started the install.

I’m not running yet, so you can guess that there were some challenges. The workstation came with an NVidia Quadro FX 560 dual head video card. James has the same box, and he managed to install SLED 10 without any challenges, so I figured OpenSUSE 10.2 would be no problem. Unfortunately, that wasn’t the case. I started with the 32 bit version. The install went fine, but then at first boot, Xwindows started and showed a blank screen. I thought I maybe needed to use the other port on the Quadro FX 560. That didn’t work.

Then I looked at the supported OS list for the Ultra 20, and it listed SLES 9 SP3 and very explicitly stated 64-bit version only. I decided to try the 64-bit version even though I had had some stability issues with it previously. I repeated the install, with the same result. Then just because James had gotten SLED 10 working I tried the 64-bit version of that. It worked fine, and the video was nice and smooth in Xwindows. I scratched my head for a while, and then came to the conclusion that the proprietary NVidia X driver was at fault. That driver is installed automatically in SLED, but in OpenSUSE it isn’t because it’s proprietary.

I’m retrying the OpenSUSE 10.2 64-bit install right now and at the end I’m going to manually install the proprietary NVidia driver and see if that works.

I wrote before about implementing internal mailing lists in our company using GNU Mailman. That never got too far because of the user interface. Our test group of users didn’t like it. Since then I played with a few other mailing list managers, but having been very busy, I didn’t get anything implemented for real. We’ve also been having problems with our GroupWise upgrade and simultaneous migration to new mail server hardware, for which I’ve opened a support ticket with Novell. Because of this the list manager server went onto the back burner.

Recently in my blog meanderings I came across a mention of Dada Mail, another open-source mailing list manager. I thought I’d give it a try. I’ve also been thinking that a mailing list manager would make a decent virtual machine appliance, so I’ve decided that in order to satisfy our corporate need and provide some give-back to the open source community, I’d try to learn about rPath Linux, a special Linux distribution for building software appliances, and rBuilder, a free service to help you bundle applications as application server appliances. I don’t know how much time I’ll have to put to this, but right now I’m thinking I’m going to try to build a virtual appliance that uses Dada Mail as a mailing list manager and is mostly pre-configured right out of the box. If it works, and provided the Dada Mail maintainer is cool with it, I’ll make it available on rBuilder Online.

I started building a VMware Server host for the new web server mentioned previously last week. The VM host is just running OpenSUSE 10.2, because it’s temporary. The VM for the web server will be SLES9, our current standard platform. I started that today. It’s mostly built, but it needs some tlc, patching and configuration before it’s ready.

A whole bunch of other crap is occupying my time now too. We’re getting ready for our switchover to Deltek Vision, and I have to build some VMs with w2k3 server and SQL Server 2000, that we will use to convert our data from CFMS to Vision. We need a few of those. Luckily I can build one and clone it.

I have a new workstation on the way, and according to our vendor, it should be here Friday. It’s a Sun Ultra 20 M2, and I can’t wait to get it.

I was having inexplicable problems with OpenSUSE 10.2 64-bit version on my present desktop, so I backed it up to the thumper box with rsync and rebuilt it in 32-bit mode today. So far so good.

Here is what looks to me like the first indication of Microsoft extortion for other Linux vendors besides Novell. Steve Ballmer is offering to help other Linux vendors reach patent agreements with them too, but to me it sounds like he is essentially saying Novell thought it necessary to protect its customers from patent lawsuits by Microsoft. That validates that we might have something to sue Linux companies and users over. Now all the other Linux vendors better get in line and sign up for “protection” too.

I admit that I don’t like Windows or the way Microsoft does business, and my views are slanted with my own experience working in a mixed network environment. I’m concerned about what this business agreement between Microsoft and Novell might do to the open source community, even though it will probably benefit our company overall, at least in the near term.

Update to this post: Wow, please forgive the horrible language of this post. I wrote it while watching the webcast live, and apparently my journalistic skills are not up to the task of simultaneous watching, listening and writing blog entries while building SLES virtual machines on another face of the cube on my desktop.

I got an email that essentially said Come to a Novell CEO Update Webcast today at 5 PM EST. I was confused slightly by this since it is now Daylight Savings Time, which made my brain break trying to figure out when to connect.

I went to the URL www.novell.com/webcast at 15:00 MDT, and there was no stream available yet. I waited a few minutes, reloaded and found a flash stream, which I connected to.

The stream showed the Microsoft Press Announcement. WTF?

I waited with bated breath…

At 15:22, a bunch of pointy-hairs from Novell and Microsoft came on and started off talking about a Microsoft / Novell joint announcement. Steve Ballmer was there as well as John Dragoon, Jeff Jaffe, Ron Hovesapien and several big-giant customers. The announcement is a big mystery so far.

The first speaker was Steve Ballmer. He started with thanks to a bunch of different people, including customers. He announced a set of agreements that he said would enhance interoperability between Windows and Linux. As CEO of Microsoft I recognize that Linux does and will continue to play an important role in IT infrastructure.

Microsoft and Novell are joining together on a series of solutions including virtualization, management and document compatibility.

Second, the new collaboration creats an intellectual property bridge between the open source and proprietary model. Microsoft and Novell have found a solution they say will respect both models.

Thirdly they are announcing a business agreement to promote and sell their products together as an interoperable stack. They focussed on Virtualization and Management.

Steve said that Microsoft wants customers who are choosing to use Linux and Windows to choose Novell’s SuSE Linux. He said if you asked him the right answer would still be Windows Windows Windows from his perspective, and something different from Jeff Jaffe’s perspective.

I have to say at this point it’s freaking me out to see Steve Ballmer speaking in front of a Novell backdrop. I can hear the ranting of the Linux zealots already. “Novell’s sold out!”

The first Novell speaker was Ron Hovesapien. He said that he initiated the discussion with Microsoft to discuss customers and the needs of the customers. He said he was impressed with the committment of Microsoft to the initiative. He said Microsoft will redistribute about 70,000 SLES licenses in order to help customers see they have a choice. They have also dedicated a team of Microsoft marketing people to help push the initiative. Microsoft has also partnered with Microsoft to collaborate on software patents in order to facilitate interoperability solutions.

Next was Jeff Jaffe who was there to present some of the technical aspects of the agreement. Novell’s opinion is that the agreement will strengthen the open source community in three ways. First Microsoft is going to work on interoperability between Linux and Windows. Next, Microsoft is announcing a covenant “not-to-sue” on patent infringement for open source developers. Third, Novell and Microsoft have agreed to collaborate on certain open source projects in management and document compatibility.

There are three areas of technology collaboration. First is virtualization. SLES on Windows and Windows on SLES are going to be worked on in the collaboration using paravirtualization. Second is web services management. They are going to collaborate on web services management technologies and in eDirectory / Active Directory. Third is document compatibility. They are going to be jointly developing translators between ODF and Microsoft Offce formats.

Next was Brad Smith, an IP lawyer from Microsoft. He talked about patent issues with Microsoft and Novell. The issues were about building a bridge between the two models. Both Novell and Microsoft have large and important patent portfolios. They worked out a bunch of arrangements that remove barriers from the two companies collaborating without compromising their portfolios, so that they could jointly develop open source projects that help interoperability.

One of the agreements is that customers who purchase support from Novell for SUSE Linux will also get a patent covenant from Microsoft. There will be a financial exchange between the companies as part of this agreement, but they didn’t explain how it would work.

He also mentioned that they need to address the needs of developers. The first is a promise that they won’t assert patents against individual developers in a non-commercial role. The second thing is that they promise that they won’t assert patents against developers for OpenSUSE.org. That allows open source developers to work in areas that may have Microsoft patents and get that code into Linux without worrying about infringing on patents.

John Dragoon came back and introduced some partners to discuss their perspective. The first partner up was Shane Robinson from HP. He spoke about how excited HP was about virtualization and interoperability between Novell and Microsoft for future datacenters.

The last remarks were from Randy Cowan from Goldman-Sachs. He said he was grateful to industry leaders working together on interoperability solutions. He is pleased that the burden of ensuring interoperability for their stratigic platforms was being moved from their shoulders to Microsoft and Novell’s.

At the end John Dragoon entertained questions.

The first question was to Steve Ballmer. Someone asked if they are considering making a similar deal with Redhat. Steve said that the deep work involved was with the patent portfolio. They were pressured by their customers to deal with Linux vendors to resolve the patent issues and provide interoperability. They started directly communicating about it in April. It sounds like Redhat hasn’t been in the picture at all.

The next question was when did this agreement start and who initiated it. Ron Hovesapien said he initiated teh discussions with the CIO at Microsoft, around April.

The next question was about how it has been going from competitors to collaborators. Steve said first things first, we’re still competitors. Microsoft recognizes Novell as a sort-of proxy for the customer, and as a way to work with the open source community that is respectful of that community. Ron H said that the agreement should accellerate the uptake of both Novell and Microsoft products.

Next question was regarding if the agreement would resolve the Microsoft / Novell Word Perfect lawsuit. The answer was no.

Next was what’s a short summary of this agreement. Steve said that they have agreed to work together on the technical levels of interoperability, and that they have agreed on a way to ensure both sides’ intellectual property is protected. Microsoft doesn’t license their IP to Linux users, but they have agreed to this agreement to provide patent peace-of-mind to customers.

There were more customer touchy feely questions as well that I haven’t written about.

Update: Novell has put up a large FAQ regarding this deal.

Yesterday I wrote an article about using Syslog-NG as a centralized loghost on SUSE Linux, and how to configure SUSE Linux machines to log to the loghost. I emailed it to Novel Cool Solutions last night. This morning, this article showed up in my news aggregator, posted on Linux Today, about using Syslog-NG as a centralized log host on Debian. Due to the approximately one-week turnaround on CoolSolutions, my article will be published after this one, which will end up making mine seem like somewhat of a rip-off. Dang it!