Scott’s Blog

We are entering the final phases before rolling out Deltek Vision, and we’ve hit a showstopper. We’re running the entire Deltek Vision system as four virtual machines on four dedicated hosts, in VMware Server on top of SuSE Linux. Each virtual machine is a W2K3 server, with one piece of Vision on it. One is a PDC, one is the web tier, one is the report server, and one is an SQL2005 server. All have dual Xeon processors and tons of RAM.

The issue we are having is that many queries against the SQL server, if they return more than trivial results, stop with an error that says TEMPDB is full. Other times the same queries will work as expected. The size of TEMPDB configured by the DBA doesn’t matter.

We thought that initially it was because we had upgraded from SQL Server 2000 to 2005, so we uninstalled and reinstalled SQL Server 2005. That didn’t fix the problem. Then, I built a standalone Deltek Vision sysetm on a single workstation, our guys imported the data and tried their tests, and it didn’t experience the problem. Right now, I’m building a standalone Vision install on a brand new virtual machine, to try it again. If that works, I’ll move it to the production hardware and see what happens there.

If it turns out that it is a problem with running the VM on our production hardware, I may have to rebuild one of the production machines as a native W2K3 server rather than a VMware Server host. Then I lose the ability to fail the machine over to different hardware in the event of an emergency.

Monday when we got up Jenn said that she had heard there was a big car accident on my regular route to work. I decided to be smart and take an alternate route. As I was going along the little main street of our small city towards an intersection with another main road, a big Chevy Impala made an ill-advised left turn directly into my path. I had enough time to slam on the brakes and shave about 10 km/hr off my not overly speedy 50, before I crashed into the Impala. The driver was an elderly man, and his elderly wife was with him in the car. Nobody was injured (although yesterday and today I’ve had a sore neck and blinding headache barely staved off by Super Motrin), but my truck and the other car were smashed up. A City police officer was at the light going the other way and offered his information as a witness. My insurance company has already stated that they think it was clearly the other guy’s fault and they’ll waive my deductible.

I’m hoping my insurance company won’t try to write off my truck. The damage is fairly superficial, limited to the hood, bumper and front fender. My truck is older (a 1994 Toyota) but it only has about 130,000 km on it. The blue book value is probably somewhat lower than it’s replacement cost would be. As with any time this kind of thing happens, I’m totally not prepared to have to pay for a new vehicle.

My insurance covers a rental vehicle. The rental place issued me what I’m calling a ridiculously huge truck. It’s a Dodge Ram 1500 4×4 quad cab. I feel like an enviro-villain driving it to work in traffic, towering over the other typically huge Albertan SUVs. I’m going to see if they can change it for a small car or something.

One of my pet peeves, and something that comes up often because I admin our SPAM firewall, is the dumb behavior of vendor and service provider websites in sending our users email with the “From:” field set to an address in our own domain. This is called “spoofing”. An example of its use is when one of our staff goes to a vendor site, and fills out an online order form for a purchase order. The vendor’s ordering site sends an email to my boss requesting approval of the order. The email that the vendor’s site sends has the “From:” field set to the email address of the user who submitted the order, instead of an email address belonging to the vendor. According to section 3.6.2 of rfc2822, which is the document that defines how email addresses are supposed to work, this behavior is wrong.

This excerpt says that essentially, the “From:” address should always belong to the entity who sends the message, namely the vendor.

In all cases, the “From:” field SHOULD NOT contain any mailbox that
does not belong to the author(s) of the message. See also section
3.6.3 for more information on forming the destination addresses for a
reply.

Vendors do this so that if the recipient replies to the message, the reply will go to the person who made the order or whatever. Spammers do this because it is common practice to automatically trust senders from your own domain or from your own address book with your spam filters, so it helps spam get in. Many people who run enterprise spam firewalls block messages from their own domain name if they don’t come from their own mail server, which is the case when a vendor does this.

Fortunately, rfc2822 provides an alternative means to accomplish what the vendor wants, while allowing spoof protection to be enabled and still not block the vendors’ messages. It’s called the “Reply-To:” field, and the rfc defines it like this:

When the “Reply-To:” field is present, it indicates the mailbox(es)
to which the author of the message suggeststhat replies be sent.

All the vendor has to do is put something like “orderingsystem@vendor.com” in the “From:” box, and the address of the customer in the “Reply-To:” box. When a user replies to the message, the reply goes to the “Reply-To:” address. Unfortunately, too many people who build ordering systems or service providing systems are either too lazy or too stupid to read the bloody standard and do it properly.

Instead, the vendors go to great lengths to write FAQs and create help forums and “compatibility assessments” in order to explain to their customers how to “fix” their spam filters to prevent the vendors’ malformed messages from getting blocked. Mail administrators have to often tweak their spam firewalls to allow these damn things through. We have about a dozen whitelist entries in our spam firewall to allow various vendors’ broken messages through.

I was unexpectedly asked to provide a PHP and MySQL-enabled web server with phpMyAdmin for one of our subsidiary companies to host their brand new web content, that they contracted to be developed by a marketing outfit. The server was requested last week around Wednesday. I had no hardware, no host environment, no software selected, and was basically not ready for this request. We were planning on revamping our public web hosting infrastructure in the late spring after all the wrinkles get worked out of our Deltek Vision deployment, and after we’re done updating our GroupWise hardware and decommissioning the old GroupWise hardware. Right now the old and new GroupWise systems are running in tandem in our colo rack, so there’s not really any spare hardware capacity or even much rack space down there.

Anyways, I quickly dumped some lab services I had running in Engineering on an ML370, stuffed some ram into it, and started building a virtual server host last week. I set it up in OpenSUSE 10.2, with VMware Server 1.01, and then built a web server VM with SLES9, Apache, MySQL, phpMyAdmin, an FTP server, a firewall, user IDs, and all the latest patches. I deployed it to the public internet, with the virtual server host’s interface hidden behind the firewall, and just the appropriate services exposed outside on the VM. I notified the web developers that it was ready yesterday mid-afternoon.

The web developers were barking at me about how long it was taking and how I might cause them a production delay. I got the server out as fast as I could, considering it was spur of the moment and I had to reshuffle a bunch of other work and hardware to deal with it. I assumed that because they were dancing around waiting for it that they were ready to upload content to it right away. Now it’s over a day since I notified them it was ready, and there’s still no content. They haven’t even tried to login yet. It just goes to show you that a lack of planning on other people’s part shouldn’t constitute an emergency on your part. Despite the fact that I take that as a fundamental axiom, I don’t follow my own advice too well and allow other people to impose artificial urgency to too many things I do. I should learn my lesson, but in a service role in our company, it’s tough, and often the squeakiest (and most annoying) wheel gets the oil.

I wrote before about implementing internal mailing lists in our company using GNU Mailman. That never got too far because of the user interface. Our test group of users didn’t like it. Since then I played with a few other mailing list managers, but having been very busy, I didn’t get anything implemented for real. We’ve also been having problems with our GroupWise upgrade and simultaneous migration to new mail server hardware, for which I’ve opened a support ticket with Novell. Because of this the list manager server went onto the back burner.

Recently in my blog meanderings I came across a mention of Dada Mail, another open-source mailing list manager. I thought I’d give it a try. I’ve also been thinking that a mailing list manager would make a decent virtual machine appliance, so I’ve decided that in order to satisfy our corporate need and provide some give-back to the open source community, I’d try to learn about rPath Linux, a special Linux distribution for building software appliances, and rBuilder, a free service to help you bundle applications as application server appliances. I don’t know how much time I’ll have to put to this, but right now I’m thinking I’m going to try to build a virtual appliance that uses Dada Mail as a mailing list manager and is mostly pre-configured right out of the box. If it works, and provided the Dada Mail maintainer is cool with it, I’ll make it available on rBuilder Online.

I’ve got a little home network like all geeks, and it contains three computers plus various networking devices, like a router/firewall and a switch and wireless access point. The computers are two windows machines for Jenn and the kids, and one FreeBSD box, which is both my workstation and the network storage server and domain controller for the windows machines. This situation is not ideal for a couple of reasons. First, my machine always has to be on in order for the other machines to login, and second, I can’t freely tinker with my machine for fear of breaking the carefully constructed SAMBA configuration that allows my machine to be the domain controller.

I’d like to be able to format my machine and install a different operating system on it whenever I want, without breaking the network for Jenn and the kids. To that end, I want to move the network storage and hopefully the authentication off to a different machine, that is easy to build, configure, back up and admin. Since I have no budget, and because I want some flexibility in what the server machine does, I haven’t been looking at network attached storage devices from storage vendors, even though there are several of those on the market that target home users. Instead, I’ve been looking at free and preferably open source NAS software.

I basically need shared storage, some way of sharing USB printers, and some kind of backup mechanism. I also want a DHCP server, but my firewall/router does that for me. As I see it, there are really three choices for what I want to do. First, I could take an old PC and build a full-bore server, running FreeBSD or some other network-server type OS, configure SAMBA and NFS, and CUPS, and some backup software, and manage that thing. Second, I could buy pre-made devices that do network printer sharing and file serving. Third, I could take an old PC and install a NAS appliance software package on it.

The full server thing has its appeal, except that it is complex to build and manage and I don’t have a lot of time for that. The pre-made devices are too much money. I’ll be happy if I can scrape enough dough together to buy some hard drives. That leaves a NAS software appliance, of which there are a few to choose from, but none of them that I have found do printer sharing.

I looked at what’s out there and had initially decided to try FreeNAS, because it requires very little resources, and it is based on FreeBSD, and I have a disk full of data on my existing FreeBSD system that I could just plug into the FreeNAS server and share out. Unfortunately, FreeNAS has a big drawback that makes it unusable for me: You can’t apply different access rights to the same shared filesystem for different users or groups, at least via the web interface. I have shared filesysetms that are read-only accessible to the kids but read-write accessible for me and Jenn, and I’d like to keep that capability.

Now I’m looking at OpenFiler, another open source NAS based on Linux. It has much more capable access controls, and supports software RAID with storage pools, allowing disks to be added and capacity of existing volumes to be grown o the fly. It doesn’t act as a SAMBA domain controller, but I’m thinking we can live without that. It also doesn’t have USB printer sharing, but I might be able to find a cheap hardware widget that does that instead. I just have to find an older PC to run it on.

I have been blogging with Movable Type for quite a while. My posts from my MT blog date back to August 2004. My recent switchover to a hosted blog on WordPress has actually been pretty fun. I was writing some posts today and I noticed the stats pages for my blog. Considering my nonexistent readership, I was reminded of this old Dilbert cartoon, the punch-line of which was something like “Never has so little been measured so much.”

The management interface lets you see how many page views there were for each post, how many clicks there were on different URLs in your posts, and what search terms are resulting in links to your blog pages. Very neat. There’s also a stats page for the subscription feed to the blog, so you can tell how many people syndicate your blog.

I have been trying and trying to get to like a Blackberry as my phone and organizer. I got the thing at Brainshare last year. I have failed miserably, and I’m going back to a regular non-smart phone and separate Palm organizer. I’ve used Palms for 7 years, and I just find them easier to use than Blackberry. I also like using my Palm as an e-reader. I got a new Tungsten T/X, which has a big screen and built-in wi-fi. The included Versamail email software works great with GroupWise in IMAP mode. Not to mention, my neighbor has a nice unsecured wireless access point and broadband, which makes it easy to get mail.

I started building a VMware Server host for the new web server mentioned previously last week. The VM host is just running OpenSUSE 10.2, because it’s temporary. The VM for the web server will be SLES9, our current standard platform. I started that today. It’s mostly built, but it needs some tlc, patching and configuration before it’s ready.

A whole bunch of other crap is occupying my time now too. We’re getting ready for our switchover to Deltek Vision, and I have to build some VMs with w2k3 server and SQL Server 2000, that we will use to convert our data from CFMS to Vision. We need a few of those. Luckily I can build one and clone it.

I have a new workstation on the way, and according to our vendor, it should be here Friday. It’s a Sun Ultra 20 M2, and I can’t wait to get it.

I was having inexplicable problems with OpenSUSE 10.2 64-bit version on my present desktop, so I backed it up to the thumper box with rsync and rebuilt it in 32-bit mode today. So far so good.

Consultant that we didn´t know was working for one of our subsidiaries for whom we host web content: ¨What is the login ID, password, database id, login and password for the PHP-enabled web server we will be using to host the new web content that we´ve developed for the subsidiary?¨

Me: ¨What PHP enabled web server with a database? Our public-facing web servers all host static content.¨

Consultant:¨Oh… crap.¨

Now I have to build a new web server VM (which I was planning on doing later this spring) and find some spare capacity at a reasonably high-availability server in our infrastructure to host it (which I was also planning to do later this spring) and I have to do it in a couple of days. Crud