Scott’s Blog

We are deploying Deltek Vision 4.1 as our new financial management system in March. We started work a while back on this project. I built the infrastructure in VMware Server running on top of SuSE Linux Enterprise Server 9. We are using a three-box Vision implementation, with a separate VM running Windows 2003 Server Standard Edition on dedicated VMware Server hosts for each of Vision Web, Vision Reporting and Vision SQL Server. The virtualization is to allow for disaster recovery and portability of hardware. The database analyst and programmer guys got started quite a while ago getting the reports that our project managers rely on in our old system working in Vision. We’ve also been testing and troubleshooting Vision and training the accounting staff during this time.

A problem started manifesting itself with Vision and SQL Server after some of the data was imported into SQL Server and we started doing queries on it. The problem would occur particularly often whenever nested select statements were used in a query. SQL Server would fail to execute the query and error out with one of four different errors: error 5243, 5242, 823, or 682. All of these errors have multiple meanings, but a common thread is I/O problems to do with physical disks or storage drivers, when SQL Server does lots of writes in TEMPDB. In our case because we are using SQL Server on a virtual machine, it implied some kind of problem with virtual disks or with the VMware virtual storage controller driver, or possibly with the underlying filesystem on the VMware host.

Several VMware Server knowledgebase and discussion posts mentioned similar problems regarding SQL Server 2000 and SQL Server 2005 on VMware Server.

To confirm that the problem was a VMware problem, which was just a suspicion initially, I built a physical Windows 2003 Server that was otherwise identically configured to the virtual one. On the physical server the queries never failed in our tests.

That made us fairly confident that we had a problem with VMware Server somewhere. When I initially created the virtual machine, I built a reiserfs partition on our IBM DS4300 SAN to store it. When I built the VM, I created a 100 GB virtual disk that was configured in 2 GB chunks, and I did not preallocate all the space at build time. I thought that perhaps the I/O problems were occurring when the VM writes the TEMPDB and new storage was allocated as the virtual disk expanded. I decided to convert the virtual disk to a fully preallocated disk using vmware-vdiskmanager, which is a command-line tool that comes with VMware Server. I did that conversion and then tested the new VM on non-production hardware that was very similar to the production blade server, except that it had locally attached storage instead of a SAN LUN. The problem almost went away. It went from erroring out more than half the time to erroring out about once in 15 or 20 runs. That indicated that I was on the right track.

We had a momentary lapse of reason and thought that the VM might run better on a Windows VMware Server host. I moved the 100 GB preallocated disk version of the VM to a Windows XP Pro workstation running VMware Server. The error occurred nearly every time, so we abandoned that ill-conceived path.

Next, I thought that either reiserfs couldn’t cut it, or VMware Server couldn’t cut it. Since I had just received a new workstation from the vendor, I configured it with OpenSUSE 10.2 and formatted the disk in ext3. I also built an ESX Server 3 evaluation server in Engineering, on an IBM x334 pizza box connected to a fibre-channel SAN. I copied the 100 GB VM to both my new workstation and the ESX Server.

On my workstation, the moderately demanding test query ran 50 times in a row without failing until I gave up on it, proving out that ext3 works better as an underlying filesysetm for VMware Server, at least when the VM you are hosting is Windows 2003 Server with SQL Server 2005.

On the ESX server, the query also worked every time, which was fully expected.

Finally, I decided that even if the ext3 and preallocated disk fix didn’t fix the problem 100% of the time, it was worth applying it to the production system, so that the problems would be reduced during training. It would also buy us time to decide whether or not to buy ESX Server for about $10,000CDN including one year of support.

I shut down the production Vision database server after hours. I moved the existing non-preallocated VM off the SAN LUN that it used for production. Then I reformatted the reiserfs partition on the SAN LUN to ext3. After the format I was surprised to find that the available space was smaller than it was with reiserfs. I had to resize the SAN LUN up a few gigabytes to allow me to convert the non-preallocated disk to a fully preallocated one back on the SAN LUN. After the resize, I recreated the ext3 partition and used vmware-vdiskmanager to convert the non-preallocated disk to a preallocated one. The VM booted and ran fine after the disk conversion.

On the converted production VM, all errors appear to have ceased and performance may have improved slightly as well. We have decided to proceed to deployment on VMware Server using this configuration.

Take Away Points

• The problem referenced in this article occurs on SQL Server 2000 and SQL Server 2005. We discovered this after the fact while working on something else.
• It is a good idea to run VMware Server on Linux, not on a Windows host.
• It is a good idea to use ext3 instead of reiserfs as the filesystem to store your virtual machines. Other Linux filesystems might be suitable as well, but were not tested.
• Filesystems formatted with ext3 use more space for overhead than reiserfs.
• VMware Server is similar in performance to VMware ESX server for Windows 2003 virtual machines running SQL Server under light to moderate loads.
• In the future I will try very hard to not have to move a 100 GB virtual machine all over my network. It takes a long time to repeatedly move 100 GB worth of files from system to system. (duh!)
• When working with troubleshooting on large virtual machines, it is great to have lots and lots of fast storage nearby on the network. Speculative changes are much less hair-raising if you have lots of room to backup your virtual machines.
• You can do awesome stuff with virtual machines that you just can’t even consider unless you have lab hardware coming out your ears and an army of lab monkeys to help you.

In the Pearl Jam Concert review article I mentioned that PJ was making concert recordings available for purchase and download in beautiful non-copy-protected non-DRM standard mp3 format. I forgot to mention that you just have to go to bootlegs.pearljam.com to buy them. You can even pre-purchase shows that aren’t available/haven’t happened yet and they will automatically download when they become available.

Last night Mark from work and I went to Pearl Jam at Rexall Place. It was my first Pearl Jam show, and even now the next day I’m virtually speechless. It was spectacular. As you will be able to tell from this post, I’m a major PJ fan.

We went to the pre-show party at the Stonehouse, organized by Genevieve from the PJ Message Pit. She, with some other hard-core fans, worked hard to give us a good party, with nothing but Pearl Jam music, drink specials, fan camaradarie, and even some cool stuff to auction off for charity. The best item was a signed guitar donated by the band. The charity they chose was the Crohn’s and Colitis Foundation of Canada, presumably because Mike McReady, the guitarist, suffers from Crohn’s. It was especially satisfying for me, as I am a colitis sufferer, and I can’t say how much I’d love to see a cure. Thanks loads, Genevieve!

At the show, the first thing that was amazing to me was the crowd. I still have a bit of tinnitus going, mostly from the crowd noise. The screaming, clapping and whistling during the short break after the first set was pushing my hearing pain threshold. Everyone was into the whole show, singing along with both heavily-played radio hits and the more obscure stuff that you only hear if you own the albums. That proved the crowd was full of real fans and not just radio fans. I read various reviews this lunch time, and some people said the crowd energy was less than in Calgary, especially people who were sitting (standing) in floor seats. Others said the Edmonton crowd was more energetic and appreciative. I know that nobody at Rexall Place sat down throughout the show. The entire coliseum stood up for the whole time. It sounded deafening where we were, so I have a hard time imagining a more energetic crowd.

The set list they chose was from the whole history of the band, right back to the earliest stuff. There were lots of the popular songs, and some of the more obscure ones. It was like since PJ hadn’t played here for over a decade, they wanted to give the fans a bit of all the great stuff we’d missed in live performances. Most of the way through the show, they started playing Evolution, and Mark said that they had played all the songs he really wanted to hear. My favorite was Elderly Woman Behind a Counter in a Small Town, and Crazy Mary was a huge highlight for me. BEST. CRAZY MARY. EVER. The feeling I got when the opening song Release started up was incredible. I thought “Awesome, it’s going to be a fan’s show!” That sense carried throughout the show, and I’m sure I won’t be pulling out any other music for weeks besides my large PJ collection.

I’m still kicking myself that I didn’t splurge and go to the Calgary show too on the 4th. It would have been worth two little three hour drives in two days and the price of another ticket. Lots of people there had done so. As a huge bonus to the fans, though, Pearl Jam is making each concert available for downloadable purchase just hours after the performance. I’ve already purchased the Edmonton show for $9.99 USD and am listening to it as I write this. I’m admiring the set list from the Calgary show too and I’m sure in a couple of days it will join my collection as well. There are a couple of tracks on the Calgary show, like Insignificance and Red Mosquito that I really wanted to hear last night but didn’t make it into the Edmonton setlist. The downloads are well produced and the sound quality is top-notch, mixed right off the sound board. The download packages consist of the complete sets in plain mp3 format, without any onerous attempt at copy protection that just makes it hard to legitimately use your music. They also come with CD sticker images, jewel case images customized for each show with location, date and setlist, and a handful of high quality concert photos. Very worth it.

For the awesome show, and the easy and affordable downloads, which will enhance my memories from the concert, thanks a lot Pearl Jam. Please come back this way again!

Go me! Today is one year since I started blogging. There are 241 blog entries in my blog so far, which averages out to one entry every 1.5 days. That’s pretty good, considering I didn’t know how I would be able to keep writing frequently enough to make this useful when I started.

Blogging is pretty fun when you get into it, and I haven’t had too many dry spells when I didn’t have anything to write about. It’s rather the reverse. My job involves lots of different stuff, all the time, and my family life is busy also, so it has not been possible for me to establish a regular shcedule for blog-writing. Often times stuff I want to write about never gets written because I can’t make time for it. Other times, I end up storing up several things I want to say, and then writing a bunch of entries in a row. The quality suffers from this, I think.

Anyways, I’m going to keep on blogging, and some day, after our vacation, hopefully, I’m going to upgrade my blog server so I can securely enable comments. I’d like some feedback. I know there are a fair number of visitors, thanks to my Apache logs, but comments that weren’t all spam would be a good blogging motivator.

Anyways, here’s to the next year.

We have some servers in our network that we don’t trust to be able to connect to just any host on the network. We also have some services that we don’t necessarily trust all our users to be able to connect to. We need to hive off these services so that they can run safely and users can’t threaten them, and the servers they run on are at minimal risk of compromise, or are at minimal risk of compromising other machines. To do this they need to be firewalled.

In the past we have had packet filtering firewall services available at many points in our network but with some changes that we are making to simplify management of things and take advantage of some new broadband connections, we are losing some of the internal firewalling luxury that we have had in the past. That means that we have to implement some localized firewalls if we want to have these services properly protected. One way to do this is to take advantage of firewalling built into the operating system of the servers we are running. On Linux, there is a bidirectional packet filter built right into the OS with sufficient configuration granularity to meet any of our internal firewalling needs. However, on Windows, the included firewall software is inadequate. It only blocks incoming traffic, not outgoing traffic, which allows an infected server to still attemtpt to infect other machines on the network. Second, in XP, it is possible for a third-party application to disable the firewall without any indication to the user. To the user it can look as if the XP firewall is running even when it is not. Therefore for Windows machines, we need another solution.

One possiblility for solving the need for firewalled Windows machines is to install a host-based software firewall on each Windows server. This would work OK, but each server would need a license for the firewall software, and we would have to configure the firewalls on every server running Windows. In our environment we have a nice architectural feature that we can take advantage of to provide packet filtering firewalls without resorting to third-party firewalling products on top of Windows. That feature is that we run many of our Windows servers as virtual machines hosted in VMWare. This has allowed me to create a VMWare packet filtering firewall to stick in front of the Windows virtual machines, and run both the firewall and the hosted Widows virtual machines on one VMWare server.

The rest of this article is a bit of a tutorial on how to build a network-invisible OpenBSD packet filtering firewall on a virtual machine, to protect other virtual machines on the virtual server.

In my case I have implemented this in VMWare GSX Server 3.1. I’m not going to go over setting up VMWare, because there is great documentation for doing that on VMWare’s site. The firewall VM that I am using is based on OpenBSD version 3.7, the latest release as of this writing. I have built the firewall VM as a transparent packet filter, meaning that it has no IP addresses of it’s own, but rather bridge from the virtual segment where the Windows VMs are connected to the real network, packet filtering on the way. This protects the firewall from remote tampering, because there is really no way to access the firewall’s operating system directly over the network.

I’m assuming some basic knowledge of building virtual machines in VMWare. First, I did a basic installation of OpenBSD without XWindows on a VM setup with a 2 GB hard drive, two virtual nics, and 128 MB of virtual RAM. I used the OpenBSD mini-CD iso file cd37.iso as a virtual CD-ROM to boot the virtual machine. I have a DHCP server in my lab to provide IP addresses so I configured the first ethernet card with DHCP and then did a network installation of OpenBSD using ftp.openbsd.org as a source repository, since Calgary is close to me. Once the installation was finished, I added the emacs editor, because I can’t get the hang of vi. To do that I did this:

setenv PKG_PATH ftp://ftp.openbsd.org/pub/OpenBSD/packages/i386
pkg_add ${PKG_PATH}/emacs-21.3p1-no_x11.tgz

Once I had emacs I was ready to edit. I based my installation on the article I found in Daemon News from July 2002. The article has some slight errors, particularly on the direction of the packet filters, and it does the firewalling on the internal interface rather than the external one as I have done. I’ll explain the reason for my choice later.

First I enabled IP forwarding between my two network cards. This is disabled by default. To enable it, you have to edit /etc/sysctl.conf and change the line that says net.inet.ip.forwarding=0 to read net.inet.ip.forwarding=1. That causes OpenBSD to route packets between it’s network interfaces.

Next, I removed the IP configuration I used for the installation and added startup commands for my two network interfaces, so that they would start without loading IP addresses for each card. I also added a command to add bridging between the two cards. Note in VMWare my two ethernet interfaces came up as le1 and le2. If using a different architecture (ie real hardware rather than virtual machines) the interface names could be different. You can find out what your network interfaces are called in OpenBSD by typing ifconfig -a. Just for reference, in my configuration, In set up a bridged virtual NIC connected to /dev/vmnet0, which showed up i nOpenBSD as le1, and I set up another virtual NIC connected to /dev/vmnet2, which is a private virtual network, and that interface shows up in OpenBSD as le2. When I add virtual machines that are intended to be firewalled by this packet filtering firewall, I will configure them to be connected to /dev/vmnet2.

Here are the commands I used to setup my network interfaces without IP addresses.

Note: Some of the example text may appear truncated but if you select it you will still be able to copy and paste it, if desired.

ifconfig le1 delete  # This removes any configuration
ifconfig le2 delete  # information about the interfaces
echo 'up' > /etc/hostname.le1
echo 'up' > /etc/hostname.le2
echo 'add le1 add le2 up' > /etc/bridgename.bridge0

After rebooting, the two interfaces should be up, and bridging should be enabled between them. To see if the bridge is enabled, type ifconfig -a. The following line should appear within the output.

bridge0: flags=41  mtu 1500

Next, I reconfigured an existing Windows virtual machine, so that it’s network interface was set to be /dev/vmnet2, and booted it up. The machine could talk to the network via the bridge running on the OpenBSD VM, and it worked completely normally. That meant that the bridge on the firewall virtual machine was working, and next I could implement packet filtering.

The packet filter, called pf in OpenBSD, is disabled by default. To enable it, change the line in /etc/rc.conf that says pf=NO to pf=YES. That will cause pf to load on the next reboot, and use the default packet filter configuration file /etc/pf.conf. The default packet filtering rules allow everything in, so after making the changes to /etc/rc.conf and rebooting, the virtual machine that is behind the firewall should still be able to communicate on the network normally.

For packet filtering rules, it is important to understand the TCP/IP communications you are trying to handle. You should have a good idea of the protocols (TCP, UDP, ICMP) and the port numbers or ICPM message types that you want to allow or block. I wanted everything to be blocked by default, but to allow some specific things through. My virtual machines behind the firewall needed to be able to obtain an IP address assignment from a DHCP server outside the firewall. I wanted to be able to ping across the firewall in both directions, and to access the terminal server behind the firewall using the rdp protocol from clients on the network. I also wanted to be able to hit the server behind the firewall with ssh so I could do rsync backups through the firewall.

Note that OpenBSD’s firewall supports the notion of inbound and outbound traffic, so rules for the packet filter include the directives “in” or “out”. To understand this properly, from the perspecive of the firewall box, “in” means traffic coming from any network segment into the firewall box. “Out” means traffic leaving the box. Therefore, traffic going from one network, through the firewall to another network, goes “in” through the first interface, and then “out” through the second one. Since packet filters are applied to an interface, and we don’t need to filter all traffic twice, it makes sense allow all inbound and outbound traffic on one interface, and then block all traffic on the other interface except for the traffic we want to pass. If the firewall was in the middle of multiple network segments, it might make sense to have packet filter rules on more than one interface.

Below is a commented ruleset for allowing the items discussed above.

# /etc/pf.conf

ext_if="le1" # external interface
int_if="le2" # internal interface

# Allow all traffic in and out on the internal interface.
# Traffic only needs to be filtered on one interface in
# a bridge.  Filter on the external interface so that
# "in" and "out" directives make sense in terms of in and
# out of protected network.
pass in quick on $int_if all
pass out quick on $int_if all

# Block all traffic on external interface by default
block in log on $ext_if all

# Rules below allow particular services

# Allow PING in both directions
pass in log on $ext_if inet proto icmp all\
   icmp-type 8 code 0 keep state
pass out log on $ext_if inet proto icmp all\
   icmp-type 8 code 0 keep state

# Allow protected boxes to obtain dhcp address from dhcp
# server on umprotected network
pass out log on $ext_if proto udp from any\
   to any port = 67 keep state

# Allow GroupWise Messenger and Mail clients to
# communicate with server
pass out log on $ext_if proto tcp from any\
   to any port = 8300 keep state
pass in log on $ext_if proto tcp from any\
   port = 8300 to any keep state
pass out log on $ext_if proto tcp from any\
   to any port = 1677 keep state

# Allow ssh clients to come in to boxes behind firewall
pass in log on $ext_if proto tcp from any\
   to any port = 22 keep state

# Allow remote desktop clients to talk to terminal server
# behind firewall.  Also allow the terminal server to
# print to network printers using raw protocol
pass in log on $ext_if proto tcp from any\
   to any port = 3389 keep state
pass out log on $ext_if proto tcp from any\
   to any port = 9100 keep state

The ruleset must end with a blank line or it won’t load properly in pf. This ruleset is fairly straightforward, and by doing all packet filtering on the external interface (the interface connected to the unprotected network) it is easy keep straight that rules that contain “in” mean traffic incoming to the protected network, and rules that contain “out” mean traffic leaving the protected network.

Other services are easy to add provided that the protocols and ports are known. If you are tweaking your rules, I suggest that you write a couple of scripts to help you during your work. First, to quickly reload the filter rules after you have edited them, write a script that you can run from the command line on your firewall that looks like this.

#!/bin/sh
/sbin/pfctl -F all
/sbin/pfctl -f /etc/pf.conf
/sbin/pfctl -s rules

The other thing you might want to be able to do is parse the log file (note my pf.conf examples include the directive “log” in all the rules) to show you what rules are being hit by your network traffic. This script will dump out a list of rules that have been hit, numbered according to the order that they show up in /etc/pf.conf.

#!/bin/sh
tcpdump -n -e -ttt -r /var/log/pflog

Thanks for reading!