Scott’s Blog

I did a live upgrade on our Sun x4500 today from build 64 to build 89. I wanted to use the native CIFS (windows) server capability for zfs in the newer Solaris build.

The live upgrade was essentially a mirror of the process I undertook to get to b64 in the first place, where I created a boot environment in the alternate disk, copied my existing boot environment to it, and used a Solaris build 89 dvd image to upgrade it. Details on that can be found in this old post.

I had one problem, where the local filesystem service failed to start after the reboot into the new version of Solaris. It was complaining that some of the directories that my zfs filesystem uses for mount points were not empty. The directories it complained about were root filesystems I had created for some Solaris zones inside the zfs. I exported the zpool, which unmounted everything, and indeed, the zone subdirectory still existed in the zfs directory hierarchy. I moved those directories out of the way, and rebooted with init 6. The system came up as expected, and everything booted up normally.

It appears that what happened was that when I ran lucreate to create the duplicate of my b64 environment prior to upgrading it to b89, it erroneously copied my zone roots into it from the zfs filesystem. Then when I upgraded the system and rebooted, those files were in the way of my zfs pool, preventing if from mounting cleanly.

Other than that, the upgrade went swimmingly. I will have to check to see if the zone in question also managed to get upgraded, since I moved the files created by lucreate after the upgrade to get back to the files stored in the zfs pool.

Update - the zone I mentioned is still on build 64. I’ll have to figure out how to upgrade that.

I have a lot of disk-based snapshots of NetWare and OES Linux servers on a big Sun x4500. We get the data onto there using rsync. We let our admins get data back from Solaris by using a web interface based on Apache. However, this doesn’t work very well for getting data from Solaris to NetWare in bulk in a hurry. Neither does rsync, because with datasets in the several hundred gigabyte range, rsync is dog slow.

Here’s a quick way: Install the nfs part of the NetWare Native File Access Pack on your NetWare box, export the volume via nfs from NetWare, mount it on your Solaris box, and just copy. I’m getting about 45 GB per hour over a gigabit connection, which is fast enough for right now, because I just want to get the data over the weekend. I could probably tune it up to about 100 GB per hour with a little effort.

How to do it:

First get the NFAP from Novell for NetWare. Uncompress it somewhere on the NetWare server or put it on a CD. Use the installer on the NetWare gui to install it.

Next, edit sys:\etc\exports, and add a line like the following:

/volume/folder -rw -root solarisbox.mydomain.com

This shares the folder “folder” on the NetWare volume “volume” via nfs, and gives the Solaris machine solarisbox.mydomain.com full access to the share. If you want to share the whole volume, just omit the “/folder” part.

Next, add a line in the sys:\etc\hosts file for the IP address of the Solaris machine. If you don’t do this then the Solaris machine will be unable to mount the nfs share.

192.168.1.10 solarisbox.mydomain.com solarisbox

The last step on NetWare is to stop and restart nfs.

nfsstop
nfsstart

Next, on the Solaris machine, mount the NetWare nfs share, and go to town. As root on Solaris, do this:

mkdir /mnt/mountpoint
mount netwarebox.mydomain.com:/volume/folder /mnt/mountpoint

This will make the NetWare folder available on the Solaris box at /mnt/mountpoint.

Now you can use cp, or even rsync natively on the Solaris machine to get data stored on your Solaris volumes onto the NetWare volume.

I bought an HP EX470 Home Server for Jennifer and her workmates to use as a backup target for their systems at work. It looked carefully factory-packed in the box. I took it out and went to set it up, and noticed that the tape holding the plastic bag wrapped around the actual server box had been un-stuck previously. Uh-oh, I thought.

I booted the thing up, connected an XP virtual machine to it (I have all Macs at home), and sure enough, instead of asking me what to name the server and what I wanted the password to be, it asked me to login to the server that was already configured. The London Drugs dudes had already had the machine out, and set it up, and then when they repackaged it, they didn’t reset it to factory settings. Now I have to recover it back to factory with the restore disk before I can configure it the way I need it.

Since I got some Macs at home I’ve been wanting to use Time Machine systematically for backups. Previously I have had a FreeBSD server at home to store bulk and shared data and backups. Initially, I had anticipated getting more drives for the server, formatting them as ZFS, and using that with Time Machine. Then, when Mac OSX 10.5 Leopard released, it turned out that Time Machine only supports external USB or firewire disks. That meant my server idea wouldn’t work.

I borrowed a USB drive from work to test out with time machine, and it worked for a while, to the point that I even used it to restore a large chunk of my system because I had screwed it up by messing around with it. Then, something funny happened and the USB disk wouldn’t mount anymore. I did a bunch of troubleshooting and discovered that the journal had gotten buggered up on the HFS+ volume on the USB disk. I found a little utility that I ran to disable journaling, which then allowed me to use disk utility to repair the volume.

Recently I was using the USB disk to shuffle some data around, and at the one point that the only copy of some digital video files from the video camera I have were on the disk, it died again. I’ve been trying to repair the volume for four days with various tools. Nothing is working so far. It appears that there is an actual hardware failure somewhere in the device. It’s a Maxtor One Touch III with two hard drives configured in a RAID0, so I can’t recover the data from the disks one at a time by direct-connecting them to another machine.

I was considering going out and buying a big external USB disk for backups until this event occurred. On the other side of my desk, there’s a FreeBSD box with a nice set of mirrored ZFS drives where the data really should be. I’m taking this as a slap upside the head, and I’m not risking any of my data on external USB disks anymore. I’m going to get a couple more disks for the server, RAIDZ them, and use that instead.

I won’t be able to use Time Machine, but what the hell, I can setup a different backup software for Jenn’s and my machines.

Over the weekend last weekend James and I had planned to go to the colocation and add a SATA shelf to our SAN, increasing our storage down there by about 11 TB. The install required firmware updates for every piece of hardware that touched the SAN, including blades, the blade chassis, the SAN controllers, all the existing storage shelves, and every disk. We were pretty concerned about the upgrade, since we have some critical systems that use that storage, including our financial management system and email.

Unfortunately I wasn’t able to go because of the funeral for Reagan’s dad, so James went on his own and I helped him remotely. Friday James took some big SCSI disks down and plugged them into one of the pizza boxes we have down there, and I spent all night backing up data and virtual machines while the new storage shelf spent the night burning in on the test bench in the colocation staging area. Saturday morning I got up early and made sure all the backups were done, and then shut everything down. James spent the day installing firmware updates, and then installing the storage shelf. Once he was done, I helped him get everything up and running again. After all is said and done, between the colocation and our corporate office, we have about 50 TB of spinning bits in production.

The install was successful and no data was lost, and everything came back up as expected. There was one bugbear though. On late Monday, the battery in one of the SAN controllers failed, which kicked the whole SAN into write-cache-disabled mode. When write caching is off, the performance of the SAN is inadequate to running our mail and the financial system, and users experience long delays and connection failures. We have a new pair of batteries coming tomorrow, but in the mean time we had to force the controllers to enable write-caching even without the batteries. We risk data corruption in the event of a power loss, but the colocation has battery and diesel backup, plus redundant power feeds from the utility company, so for a day or so it’s an acceptable level of risk. It’ll be fixed tomorrow.

Do not consolidate free space on the SAN during the work day.

I now have iSCSI working, and have managed to resolve the problems I was having in my previous post. The problem was caused by me trying to use the wrong syntax in iscsiadm, the command in open-iscsi that you use to manage iscsi on your Linux system.

If you try to login to an iscsi target using iscsiadm with the following syntax, it locks up your Linux system. While it is my fault I was using the incorrect syntax, a sane tool should not be able to hang an entire system just because the user uses the wrong syntax with the command, obviously. This command causes the crash:

iscsiadm -m node -p 10.0.0.1:3260 --targetname target --login

Note that the --targetname parameter is supposed to have an equals sign between the switch and the parameter, and I was forgetting the equals sign in favour of a space. The proper command would be this:

iscsiadm -m node -p 10.0.0.1:3260 --targetname=target --login

I have fixed my scripts and now iscsi is working for me on Linux initiators, and I have also submitted this as a bug to the open-iscsi mailing list, so that in the future, other dummies won’t crash their servers because of a typo. Check the link to see if any discussion develops on the bug report.

Since VMworld, and all the talk I heard there about iSCSI, I’ve become interested in learning more about iSCSI in the various environments we use at work. To that end, I’ve set up an iSCSI target on our Solaris-based x4500 (because with zfs zvols, iSCSI targets are pretty trivial to set up). I’ve been trying to get various software initiators to talk to it. Here’s my track record so far:

• A physical machine running OpenSUSE Linux 10.2 with open-iscsi: discovery works, but logging into the iscsi portal on the x4500 locks up the OpenSUSE box completely, forcing a power-off reboot.
• A VMware Server VM running Solaris Express DE: just works.
• A VMware Server VM running Windows XP with the Microsoft iSCSI initiator: just works.
• A physical machine running Ubuntu 7.04 with open-iscsi: discovery works, but logging into the iscsi portal locks the box up completely forcing a power-off reboot.
• Netware: just works.

Conclusion: Either find out what’s wrong with open-iscsi on my versions of the Linux kernel or the conflict between open-iscsi and Solaris’s iscsi target, or find another iscsi tool chain on Linux.

I’ve got a little home network like all geeks, and it contains three computers plus various networking devices, like a router/firewall and a switch and wireless access point. The computers are two windows machines for Jenn and the kids, and one FreeBSD box, which is both my workstation and the network storage server and domain controller for the windows machines. This situation is not ideal for a couple of reasons. First, my machine always has to be on in order for the other machines to login, and second, I can’t freely tinker with my machine for fear of breaking the carefully constructed SAMBA configuration that allows my machine to be the domain controller.

I’d like to be able to format my machine and install a different operating system on it whenever I want, without breaking the network for Jenn and the kids. To that end, I want to move the network storage and hopefully the authentication off to a different machine, that is easy to build, configure, back up and admin. Since I have no budget, and because I want some flexibility in what the server machine does, I haven’t been looking at network attached storage devices from storage vendors, even though there are several of those on the market that target home users. Instead, I’ve been looking at free and preferably open source NAS software.

I basically need shared storage, some way of sharing USB printers, and some kind of backup mechanism. I also want a DHCP server, but my firewall/router does that for me. As I see it, there are really three choices for what I want to do. First, I could take an old PC and build a full-bore server, running FreeBSD or some other network-server type OS, configure SAMBA and NFS, and CUPS, and some backup software, and manage that thing. Second, I could buy pre-made devices that do network printer sharing and file serving. Third, I could take an old PC and install a NAS appliance software package on it.

The full server thing has its appeal, except that it is complex to build and manage and I don’t have a lot of time for that. The pre-made devices are too much money. I’ll be happy if I can scrape enough dough together to buy some hard drives. That leaves a NAS software appliance, of which there are a few to choose from, but none of them that I have found do printer sharing.

I looked at what’s out there and had initially decided to try FreeNAS, because it requires very little resources, and it is based on FreeBSD, and I have a disk full of data on my existing FreeBSD system that I could just plug into the FreeNAS server and share out. Unfortunately, FreeNAS has a big drawback that makes it unusable for me: You can’t apply different access rights to the same shared filesystem for different users or groups, at least via the web interface. I have shared filesysetms that are read-only accessible to the kids but read-write accessible for me and Jenn, and I’d like to keep that capability.

Now I’m looking at OpenFiler, another open source NAS based on Linux. It has much more capable access controls, and supports software RAID with storage pools, allowing disks to be added and capacity of existing volumes to be grown o the fly. It doesn’t act as a SAMBA domain controller, but I’m thinking we can live without that. It also doesn’t have USB printer sharing, but I might be able to find a cheap hardware widget that does that instead. I just have to find an older PC to run it on.

Our x4500 server arrived today, after a couple of courier kerfuffles last week. It was much smaller than expected, and it’s amazing that 48 standard 3.5 inch drives fit in a 4U chassis. We’re still rubbing our hands together around here thinking about what we’re going to be able to do with 24 TB of disk space. Unfortunately, we’re still only thinking about it, because (not unexpectedly) we still have to get plugs to allow us to plug the 15A 240VAC power supplies into our 30A 240VAC outlets (left over from when we had the bladecenter in our office). Until then, we can’t power it up, we can only sit and admire it’s shiny silver bezel with the lovely hexagonal ventilation holes.