Scott’s Blog

I’ve been getting questions in the comments on my blog about our Deltek Vision system now that it’s been in and in production for a while. Instead of answering them in the comments, I thought I’d do a new post.

First, I’ll describe what we’ve settled on for our hardware deployment. We run Vision completely virtualized on two different VMware platforms, in a three-server configuration. We have a SQL Server 2005 server running Windows 2003 Server Enterprise edition as a VM on VMWare ESX Server on a dedicated blade server. We have a Vision reporting server (Actuate) running Windows 2003 Server Standard edition on ESX Server on a blade server that is shared with other workloads. We have a Vision Web Tier server running Windows 2003 Server Standard edition on VMWare Server (the free server virtualization product) on a SuSE Linux Enterprise Server host that is shared with other workloads. All physical servers use shared storage in a fibre-channel SAN.

The reason we settled on ESX Server for SQL Server 2005 is that under load, we couldn’t get SQL to behave properly in VMware Server. We tried various different setups, different filesystems on the VMware Server host, and in every instance SQL Server behaved in a flaky unpredictable manner, failing in various ways.

The reason we settled on ESX Server for the reporting tier was because we wanted to have a second ESX Server to run the SQL Server VM on in case the first ESX Server died, so we needed it anyways. Otherwise the Vision Reporting tier runs fine in VMware Server.

The end user interface is provided by the web tier which is just an IIS server in VMware Server. It is plenty fast enough for us. We have about 600 users give or take a few. This server is pretty lightweight, all told, and we could probably get away with running the web tier and the reporting tier on the same box, but that would limit our expandability, and we’re growing fast.

All in all, everything has worked great for the last year. My caveats are that you want to strongly consider running SQL Server either in ESX Server or on bare iron. The rest of Vision, at least for a company of our size, works great virtualized on moderately powerful server gear (ours is a couple of years old already).

You don’t generally run into performance issues, unless your accounting people are running big batch jobs or you are running maintenance routines or backups. If you can run all that off-hours, then you can get away with smaller hardware than if you have to do stuff like that during the day. If you are small (a couple of hundred users) you could probably run on one box, but I would recommend server-grade gear, not repurposed desktops. SQL Server is finicky regarding disk I/O and you want to be sure you have a fast disk subsystem. I would also consider the Vision hardware recommendations to be minimums. We doubled the recommended RAM and processor speed and it has worked out well for us. Plus it gives us room to expand. We were under 400 users when we started this.

Today is Exploit Wednesday, the day after Microsoft’s Patch Tuesday. We have a maintenance window on some of our stuff the day after Patch Tuesday so that we can get everything patched up to snuff.

I was doing patches on a bunch of Windows 2003 servers tonight, one of which had already received IE7 and the rest that had not. After installing mandatory updates on the first couple that were still on IE6, they couldn’t get onto the Internet anymore in IE6. Every time I tried I got some error that said something about being unable to access a key that wasn’t registered or something. I think it was talking about a registry key but I’m not sure.

I couldn’t run Windows Update to get IE7, because Windows Update uses IE and IE6 was broken. I had to go to a different server, download a standalone IE7 installer for Windows 2003 server, and install IE7 manually. After that the server with the broken IE worked again.

The rest of the servers showed IE7 as a high priority update, with borg-like insistance that resistance was futile. I caved and allowed the IE7 install to go through on all the servers. Each server that got IE7 installed needed four reboots to get all the IE7 patches and the latest Server 2003 patches installed.

Meanwhile, while I was repeatedly rebooting the Windows servers, Mac Software Update popped up on my Mac. It wanted to do an OS update. I let it, and it finished and went away. No reboot. That doesn’t always happen, mind you. I think there have been at least two firmware updates since I got my 20 inch iMac, and those definitely require a reboot. You can’t get away from patches, but at least I’ve never seen a Mac patch run that required more than one reboot.

Our users started using Deltek Vision last week. Despite a long time planning, preparing, and porting, there were still many long days and late nights to get everything working on time. Other than some problems with one very large set of reports that we are still trying to troubleshoot, it seems like it’s working OK. It is quite slow compared to our old in-house custom web-based reporting that we did off of CFMS, but that’s not unexpected, because what we had before was wicked-fast.

There are a few little things to clean up, like making sure backups are coordinated with accounting’s large report runs, and stuff like that. There are also lots of other IT initiatives that I’m looking forward to working on now that this project is complete.

I blogged before about the US Department of Homeland Security warning about updating Windows. Today we got email from MacAfee, our virus software provider, saying we should update all our Windows machines because something really bad is about to happen:

“Folks,

Respectfully,
Robert Embree
Corporate Account Manager
McAfee, Inc.”

I’m nervous about all those blissfully unaware home computer users with their un-patched home computers that haven’t seen a security fix since they were bought for Christmas three years ago. You know there are loads of those out there.

I spent this morning patching Windows servers for the latest hack of the day. It was a serious enough security vulnerability that the US Department of Homeland Security actually issued a statement that users should patch immediately.

Now I definitely don’t want to help terrorists use compromised Windows machines to plot terrorist acts, so personally I avoid running Windows and Microsoft software wherever possible. To do otherwise is to support terrorism (this is semi-tongue-in-cheek for you literalists). Terrorism aside, using nothing but UNIX or Linux machines makes life much easier without viruses, spyware, malware, and phishing attacks.

In the event that we have to run a server-side application that needs Windows (which seems to be happening more and more lately) we try to keep the Windows servers contained and up-to-date. Today I was patching Windows machines, and noted with dismay that we actually have eight Windows servers now. Seven of them are virtual machines, and one runs on bare iron. They tend to multiply somehow. Anyways, I patched them all, and out of the eight, one of them crashed during the Windows Update process. It crashed trying to update Windows Media Player. Why Windows Media Player can even be installed on a SERVER box is a mystery to me, but it’s there, so it has to be updated. The crash was repeatable, so I had to do a custom update, and deselect that particular patch. I’m going to see if I can uninstall Windows Media Player from that server altogether so it doesn’t need the patches.

If Microsoft ever gets Windows more secure it sure will make life a lot easier. On the other hand, I’ll just continue to run more secure systems on my own machines, and avoid all that crap altogher.

I got Deltek Vision installed in a 3-tier setup today, with one SQL server, one Reporting server and one Web/Application server. The application runs, and you can login to the web interface.  I suppose that’s a big step. I had to contact Deltek tech support to get registered on their support portal, because the login data they sent us had a couple of typos in it, but other than that, and the disappointing IE requirements they have, everything seems to be going smoothly. Tech support was responsive and helpful.

This is our second big application server setup in virtual machines. This one, while it’s still in development for the next several months, will be running on GSX server, but we may either move it to ESX or partly to ESX and partly to the new free VMware Server platform when the latter ships in production-ready code.

Hopefully the next release of Vision will work in Firefox and not require IE. I notice that they also only support IE6, but I expect they’ll have IE7 support fairly soon now that IE7 is out in public beta.

Work has been coming along with the Bladecenter. I now have a fully working Zenworks Linux Management server running, and have deployed either SLES9, OES Linux, or ESX server to all the blades. I have automatic updating working for the Linux OSes from Novell’s update servers via the ZLM server, and I have Zenworks Imaging working, to take backup images of the server boot disks.

I have also used ZLM to automatically deploy VMWare GSX Server or VMware Server Beta to several of the blades. That works well, but I haven’t figured out how to automatically configure it on each blade, so I still have to go and run vmware-config.pl after ZLM deploys the RPM files.

I have mastered the multipath-tools enough to get my SAN working the way I want it, so I can have shared-access to volumes containing virtual machines, and mount them as required on whichever blade I want. That way I can suspend a VM, unmount it on one blade, mount it on another and resume it, all within a minute or two. The only caveat I have found is that even though you can move virtual machines between the new VMware Server Beta and VMware GSX server 3.2, you can’t suspend the VM on one and resume it on the other. Suspending on one GSX server and resuming on another works as expected.

I have configured my first Windows 2003 server for the financial management system as a VM, and set up active directory. I’ll be creating the other three Windows 2003 server VMs as soon as I figure out the system requirements of Deltek Vision, and as soon as I figure out how multipathing works in ESX server.

James and I are attending Windows 2003 Server and Active Directory school this week. Today we’re learning about Active Directory. It’s similar to eDirectory but it seems like kind of a hack of Domains into eDirectory’s structure. It’s integration into DNS makes more sense than having it’s own name services (like WINS). We’ll see how things develop this week. We’ve already learned a thing or two about AD, anyways.

The objective is to allow us to design our own AD layout properly for the financial management system.