Scott’s Blog

I’ve been getting questions in the comments on my blog about our Deltek Vision system now that it’s been in and in production for a while. Instead of answering them in the comments, I thought I’d do a new post.

First, I’ll describe what we’ve settled on for our hardware deployment. We run Vision completely virtualized on two different VMware platforms, in a three-server configuration. We have a SQL Server 2005 server running Windows 2003 Server Enterprise edition as a VM on VMWare ESX Server on a dedicated blade server. We have a Vision reporting server (Actuate) running Windows 2003 Server Standard edition on ESX Server on a blade server that is shared with other workloads. We have a Vision Web Tier server running Windows 2003 Server Standard edition on VMWare Server (the free server virtualization product) on a SuSE Linux Enterprise Server host that is shared with other workloads. All physical servers use shared storage in a fibre-channel SAN.

The reason we settled on ESX Server for SQL Server 2005 is that under load, we couldn’t get SQL to behave properly in VMware Server. We tried various different setups, different filesystems on the VMware Server host, and in every instance SQL Server behaved in a flaky unpredictable manner, failing in various ways.

The reason we settled on ESX Server for the reporting tier was because we wanted to have a second ESX Server to run the SQL Server VM on in case the first ESX Server died, so we needed it anyways. Otherwise the Vision Reporting tier runs fine in VMware Server.

The end user interface is provided by the web tier which is just an IIS server in VMware Server. It is plenty fast enough for us. We have about 600 users give or take a few. This server is pretty lightweight, all told, and we could probably get away with running the web tier and the reporting tier on the same box, but that would limit our expandability, and we’re growing fast.

All in all, everything has worked great for the last year. My caveats are that you want to strongly consider running SQL Server either in ESX Server or on bare iron. The rest of Vision, at least for a company of our size, works great virtualized on moderately powerful server gear (ours is a couple of years old already).

You don’t generally run into performance issues, unless your accounting people are running big batch jobs or you are running maintenance routines or backups. If you can run all that off-hours, then you can get away with smaller hardware than if you have to do stuff like that during the day. If you are small (a couple of hundred users) you could probably run on one box, but I would recommend server-grade gear, not repurposed desktops. SQL Server is finicky regarding disk I/O and you want to be sure you have a fast disk subsystem. I would also consider the Vision hardware recommendations to be minimums. We doubled the recommended RAM and processor speed and it has worked out well for us. Plus it gives us room to expand. We were under 400 users when we started this.

I have a nice big IBM server box with ESX serving as my entire Engineering lab (Engineering is my name for the IT lab we have here for Star Trek-related historical reasons). The box has a terabyte of RAID5 SATA disks on board for virtual machine storage, but that’s not quite enough for what we’re doing. To get more storage for virtual machines, I decided that I could take advantage of our Sun x4500 server with Solaris, to allocate another terabyte of storage. Solaris supports iSCSI targets in ZFS, so that seemed like the way to go. The x4500 has four gigabit network interface cards on board, three of which I haven’t been using so far, so I decided a good way to go would be to add a gigabit network interface to my IBM ESX server and use a crossover CAT6 cable to direct-connect them and provide a dedicated gigabit storage “network” for iSCSI.

Here are my lab notes from getting that setup.

Setting up iSCSI storage on Solaris and then getting it mounted on ESX Server

First, add a physical nic to VMware Server if required. iSCSI is only supported on Gigabit ethernet in VMware ESX 3.x. If you don’t know how to do this, get your hardware dude to take care of it, and you go take a remedial computer class.

Next, add a physical nic to Solaris box if required. This was not required in my case because my x4500 server had a spare port. Hook the Solaris box nic to the ESX box nic with a crossover cable or via a gigabit switch.

Next configure the nic in Solaris like this: (assuming you know the device name). On the x4500 the built in ethernet interfaces are called e1000g0, e1000g1, e1000g2, and e1000g3. I was already using e1000g0 for the main interface to the network.

ifconfig e1000g1 plumb
ifconfig e1000g1 192.168.254.2 netmask 255.255.255.0 up

Check to see if it worked with this:

ifconfig -a

You should see that e1000g1 now has an IP address assigned to it.

Next, add the file for the new interface’s IP address under /etc

cat > /etc/hostname.e1000g1
192.168.254.2
CTRL-D

Next, edit /etc/inet/hosts and add a line for your new IP address. You might have to chmod +w hosts before editing and chmod -w hosts after. Follow the format in the file. Set a new hostname for your new interface

For example,

192.168.254.2 ss003stor.corporate.ae.ca ss003stor

Make sure the file ends with a blank line.

Next, edit the /etc/inet/netmasks file. You might have to do the chmod ting again because the file is normally set to read only. Add this line:

192.168.254.2 255.255.255.0

Make sure the file ends with a blank line.

Next, add a new zfs volume that we will use as an iSCSI target. This example assumes that the zfs filesystem data/vols already exists.

zfs create -V 1T data/vols/vs-vmfs01

This creates a 1 Terabyte data volume. It is a volume, and not a zfs filesystem, so you can’t see it with the ls command, but you can see it with zfs list -r data/vols.

Next, make the volume shared via iSCSI.

zfs set shareiscsi=on data/vols/vs-vmfs01

Next, create an iSCSI target portal group (tpgt), and set it up to only listen for iSCSI connections on your new dedicated storage network card.

iscsitadm create tpgt 1
iscsitadm modify tpgt -i 192.168.254.2 1

Make sure the group got created:

iscsitadm list tpgt -v
TPGT: 1
      IP Address: 192.168.254.2

Add the target we already created to the group.

iscsitadm modify target -p 1 data/vols/fs-vmfs01

Make sure that worked.

iscsitadm list target -v

Target: data/vols/vs-vmfs01
    iSCSI Name: iqn.bla.bla.bla
    Alias: data/vols/vs-vmfs01
    Connections: 1
        Initiator:
            iSCSI Name: iqn.bla.bla
            Alias: vs200.bla.bla
    ACL list:
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 0
            GUID: blabla
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0T
            Backing store: /dev/zvol/rdsk/data/vols/vs-vmfs01
            Status: online

Now the target is ready to get mounted and formatted by the ESX Server.

The next step is to configure the ESX Server.

First, go to the virtual infrastrucutre client and select the server’s configuraion tab. Click on Network Adapters and make sure your network card shows up in the list. Click on Networking, and click Add Networking.

In the dialog, select the VMkernel radio button and click Next. Select “Create a Virtual Switch” and make sure your new network interface is selected. Click next. Under Network Label, give it a name like vmkernel storage, and select Next and then Finish.

On your new virtual switch, click Properties. In the Ports dialog, click Add, and select VMKernel. Call the network label VMkernel Storage, and give it an IP address on the same subnet as your Solaris dedicated storage server network card, with the same network mask. Click Next and Finish.

Next, go to the console command line via ssh or on the actual console. Login as root. Type the following commands to enable iSCSI software initiator, open a port in the firewall, and setup and scan the iSCSI target:

/usr/sbin/esxcfg-swiscsi -e
(enables iSCSI software initiator)
/usr/sbin/esxcfg-firewall -e swISCSIClient
(opens iSCSI client port in firewall)
/usr/sbin/vmkiscsi-tool -D -a 192.168.254.2 vmhba40
(tells the iSCSI software adapter to request iSCSI targets from the storage box)
/usr/sbin/esxcfg-rescan vmhba40
(tells the iSCSI initiator to scan for new targets)

Next, go to the Storage tab and click Add Storage. Choose the Disk/LUN radio button and click Next. Select your new iSCSI target from the list and click Next. Follow the prompts, and ESX server will format the volume with vmfs and mount it. You should now be able to start using it right away.

A few weeks ago I cloned our entirely virtual Vision system into Engineering, and our DB / Analyst guy Bart and our accounting people went through a run-through of upgrading Vision from 4.1 to 5.1. They were pretty successful, and after doing a bunch of testing, they figured out which of our particular customizations would have to be tweaked to get working in 5.1, and declared themselves ready.

On Saturday night, I took a last backup of the whole virtual environment, and Sunday morning, Bart upgraded Vision to 5.1. He called me at the snowboard hill by 10:40 AM to let me know he was done and everything was back up and running. I love it when an upgrade goes off so well. We’ve been running on Vision 5.1 for a couple of days, and so far, so good.

I’m building new DNS servers for our corporate DNS due to our recent hardware problems with one of our old servers. We have had good success with publicly exposed BSD servers on the Internet, including OpenBSD and FreeBSD. My experience with FreeBSD is much more recent than with OpenBSD so for the new servers I am going with FreeBSD 7.0. We also have been building almost everything new as virtual machines on ESX server for the last year or so, so the DNS servers will be no exception.

FreeBSD is not officially supported by VMware on ESX server, but it runs on ESX 3.x without problems. My ESX server is configured with ESX 3.02. My first step was to build a template FreeBSD virtual machine. I’m going to use that as the basis for a master DNS server that we hide away and use to make changes on, and some slaves that are public-facing.

The first thing I didwas to get the FreeBSD iso files and put them somewhere where the ESX server can see them. I have an ISO library on my Sun x4500 that can be nfs mounted by my ESX servers, so that’s where I put the FreeBSD iso files (there are 3 iso files for FreeBSD 7.0 RC1).

Next, I set up a new virtual machine with 512 MB of RAM (probably overkill for a dedicated authoritative DNS server in charge of a few zones) and an LSI scsi adapter. For the guest OS, I chose Other/Other (not 64-bit). I pointed the VM’s CD-ROM drive at the FreeBSD disk1 iso file, and booted it. I’m not going to go over the partitioning and installation details of FreeBSD here. The install of FreeBSD is fairly straightforward if you have some UNIX/Linux experience. I chose to install the “Developer” package set, and to install the ports collection. I also added emacs and bash via the packages at the end of the installation because that’s how I roll. During the install it’s a good idea to add another user besides root, and to make them a member of the wheel group, so that they can run su to become root. In FreeBSD you have to be a member of the wheel group in order to be allowed to run su.

Once the install was completed, I rebooted, and logged in as root. To make the FreeBSD system a good virtual citizen on your ESX server, it’s a good idea to install the vmware tools. Unfortunately there is no vmware tools installer for FreeBSD on ESX 3.x. Luckily, the vmware tools for FreeBSD from VMware Server work in ESX Server. To install them on my virtual machine, I downloaded the tgz package for VMware Server 1.04, extracted it, and copied the file vmware-server-distrib/lib/isoimages/freebsd.iso to my ISO library. Then I connected that iso file as the CD-ROM of the virtual machine. Then from inside the VM, I used the ports collection to install the vmware tools. To do that, I logged as root (or become root with su) and then ran the following commands:

cd /usr/ports/emulators/vmware-tools6
make install clean distclean

Once the install completed, I rebooted to startup the vmware tools guest daemon. I logged in after the reboot and ran ps ax | grep vmware to verify that the vmware guest daemon was running, and I saw output like this:

570 ?? Ss 0:03.52 /usr/local/sbin/vmware-guestd --background /var/run/vmware_guestd.pid --h

The final thing to get the FreeBSD VM ready to go is to replace the default ethernet device, which is a lance virtual nic. There are many anecdotes about the lance ethernet driver in FreeBSD dropping packets under load. It’s a good idea to change to either the vmxnet device or the e1000 device instead of lance. The FreeBSD GENERIC kernel (the default kernel) in FreeBSD 7.0 does not have a driver for the vmxnet device, but it does have one for the e1000 device. Either can be used, but if you choose the vmxnet device, you have to build a new kernel (waaay beond the scope of this post).

To replace the default lance virtual nic with the e1000 one, I first shut down and powered off the vm. Then, I accessed my ESX server as root with a shell (via ssh or on the console) and used vi to edit the .vmx file of the FreeBSD virtual machine. I found a bunch of lines beginning with ethernet0 and inserted a line at the top of those lines like this:

ethernet0.virtualDev="e1000"

Then, I saved the file and started up the virtual machine. When it came up, there was no IP address bound to the ethernet interface in the VM, and I had to login to it using the VMware ESX server Virtual Infrastructure Client console or Virtual Center. I logged in as root to the VM, and edited the /etc/rc.conf file. I found the line that said ipconfig_le0=... and changed it to ipconfig_em0=... and then rebooted. The new virtual nic was then started up with the same settings as the old one had before I replaced it.

That’s it. If you follow this procedure, you will have a VM that is ready to be configured for whatever workload you want to put on it.

Tonight while applying Windows patches I wanted to make a backup of all our Vision servers so that I can install them on an ESX server in engineering and do a few test upgrades. The total size of all the VMs involved is about 200 GB. So far I’m pushing 3 hours copying all that stuff on the 2GB/s fibre-channel SAN.

I can’t wait to see how long it will take to copy all that stuff over the WAN from our colocation site to our corporate office. I should be ready to start testing in a week or so.

Our company is drinking the virtualization kool-aid more and more. We have three ESX server licenses in our Bladecenter now, and tonight I migrated the second of four of our Deltek Vision servers (the Reporting tier) from VMware Server hosted on Linux to ESX server (The first was the Deltek Vision SQL Server tier).

First I used my normal rsync backup script and zfs snapshot to create the last backup of the VMware Server version of the report server. Then I just mounted the snapshot via NFS on the ESX server, and used

vmkfstools -i nfsvol/source.vmdk vmfsvol/dest.vmdk

to clone the virtual disk onto one of my SAN vmfs3 datastores. Then I created a new virtual machine in one of the ESX server hosts, pointed it at the imported virtual disk, booted, re-installed vmware tools and rebooted again. It wasn’t too hard and everything works.

The longest part was the vmkfstools import operation, which took about 45 minutes for the 80 GB disk. The SAN was under heavy load at the time doing mail system backups, so I can’t complain.

I got back from VMworld last Friday. I had intended to blog throughout the sessions as I have in the past with Brainshare, but unfortunately the internet connectivity in the presentation rooms at the Moscone Center was not good enough to maintain a decent connection. I suppose I could have done offline blog posts and then uploaded them during breaks, but it was too much fiddling and I didn’t bother.

However, there was lots of great information at the conference. I learned some immediately applicable information about securing ESX servers and services that run on VMware servers. I also learned some practical stuff about network attached storage in a virtualized environment, including NFS and iSCSI.

There was also a lot of stuff that got my brain stewing about future approaches both in our datacenter back-end stuff and in the regular office server rooms and even out on the floor. In particular, I think future shared storage that we buy will probably be moving away from fibre-channel and towards iSCSI, which is faster (when done right,) more flexible, and way cheaper. Also, I think that a lot more of our desktop workloads are virtualizable than we thought, and also we could probably achieve significantly higher hardware consolidation than we have been expecting.

The next year or two will probably see much more of our infrastructure becoming virtualized, that much is certain.

I’m at VMworld in San Francisco. Jenn’s here with me. I’ve already attended my first hands-on lab. It was about securing ESX Server. The lab was very well put together and the lab manual was very accurate. Well done, and I learned some useful stuff that we can put into practice right away.

I went on vacation with my family, starting with summer swimming provincials August 18, 19 and 20, up until September 2. In all that time the only writing I did was to send a few Facebook messages to family and update my Facebook profile. I’ve been back at work a week, and as usual I’m finding it slow to get back into the writing habit, both personally and professionally. I’m not sure why this is, but I’m guessing it’s just getting out of the writing habit. It doesn’t happen to me on long business trips, but that’s probably because I always blog during them. I have a big business trip coming next week, where I’m going to VMWorld in San Francisco. I imagine that will get me back into the writing habit, because there’ll be tons of interesting stuff to write about and I’m the only one from my work attending the conference so there’ll be no other real way of discussing what I’m learning.

I use virtualization extensively at work to run multiple virtual computers on one physical machine. We also use it to disconnect the operating system and application environment from the physical hardware for disaster recovery and hardware agnosticism. Our platforms of choice are VMware Server and VMware ESX server. The first is great because it’s free, and the second is great because it’s amazingly fast and reliable.

Almost all our virtualized workloads run fine in VMware Server in production, which is great because there are no license costs. The only workload that works like crap on VMware server that we use, is SQL Server. It dies like a dog because of I/O latency or something, and the only thing we could do to get it working in a virtualized environment is to run it in ESX server. SQL Server is so flaky that it returns random query results (when it works) or one of several unrelated errors (when it fails) when run in VMware Server or VMware Workstation.

Since I’ve become a switcher I’ve been looking to run virtual machines on my Mac at home. The likely choice for me is VMware Fusion, which is still in beta, even though Parallels is more mature on the Mac platform. The advantage of VMware is that my work virtual machines will run at home. In Beta 3, it seems that 64-bit VMs are not supported, even though my Mac is a Core2-Duo. The website says you can run 64-bit VMs, but a Solaris VM I built at work won’t run in 64-bit mode in Beta3. I haven’t updated to Beta4 yet, but apparently it has a new feature called Unity, which allows you to sort-of disappear a Windows virtual machine desktop so that the application windows running inside the virtual machine just appear as windows on your Mac desktop. that’s kind-of cool, I guess. I’ll update to Beta4 and see if my Solaris 64-bit VM works.