Scott’s Blog

I did a live upgrade on our Sun x4500 today from build 64 to build 89. I wanted to use the native CIFS (windows) server capability for zfs in the newer Solaris build.

The live upgrade was essentially a mirror of the process I undertook to get to b64 in the first place, where I created a boot environment in the alternate disk, copied my existing boot environment to it, and used a Solaris build 89 dvd image to upgrade it. Details on that can be found in this old post.

I had one problem, where the local filesystem service failed to start after the reboot into the new version of Solaris. It was complaining that some of the directories that my zfs filesystem uses for mount points were not empty. The directories it complained about were root filesystems I had created for some Solaris zones inside the zfs. I exported the zpool, which unmounted everything, and indeed, the zone subdirectory still existed in the zfs directory hierarchy. I moved those directories out of the way, and rebooted with init 6. The system came up as expected, and everything booted up normally.

It appears that what happened was that when I ran lucreate to create the duplicate of my b64 environment prior to upgrading it to b89, it erroneously copied my zone roots into it from the zfs filesystem. Then when I upgraded the system and rebooted, those files were in the way of my zfs pool, preventing if from mounting cleanly.

Other than that, the upgrade went swimmingly. I will have to check to see if the zone in question also managed to get upgraded, since I moved the files created by lucreate after the upgrade to get back to the files stored in the zfs pool.

Update - the zone I mentioned is still on build 64. I’ll have to figure out how to upgrade that.

I have a lot of disk-based snapshots of NetWare and OES Linux servers on a big Sun x4500. We get the data onto there using rsync. We let our admins get data back from Solaris by using a web interface based on Apache. However, this doesn’t work very well for getting data from Solaris to NetWare in bulk in a hurry. Neither does rsync, because with datasets in the several hundred gigabyte range, rsync is dog slow.

Here’s a quick way: Install the nfs part of the NetWare Native File Access Pack on your NetWare box, export the volume via nfs from NetWare, mount it on your Solaris box, and just copy. I’m getting about 45 GB per hour over a gigabit connection, which is fast enough for right now, because I just want to get the data over the weekend. I could probably tune it up to about 100 GB per hour with a little effort.

How to do it:

First get the NFAP from Novell for NetWare. Uncompress it somewhere on the NetWare server or put it on a CD. Use the installer on the NetWare gui to install it.

Next, edit sys:\etc\exports, and add a line like the following:

/volume/folder -rw -root solarisbox.mydomain.com

This shares the folder “folder” on the NetWare volume “volume” via nfs, and gives the Solaris machine solarisbox.mydomain.com full access to the share. If you want to share the whole volume, just omit the “/folder” part.

Next, add a line in the sys:\etc\hosts file for the IP address of the Solaris machine. If you don’t do this then the Solaris machine will be unable to mount the nfs share.

192.168.1.10 solarisbox.mydomain.com solarisbox

The last step on NetWare is to stop and restart nfs.

nfsstop
nfsstart

Next, on the Solaris machine, mount the NetWare nfs share, and go to town. As root on Solaris, do this:

mkdir /mnt/mountpoint
mount netwarebox.mydomain.com:/volume/folder /mnt/mountpoint

This will make the NetWare folder available on the Solaris box at /mnt/mountpoint.

Now you can use cp, or even rsync natively on the Solaris machine to get data stored on your Solaris volumes onto the NetWare volume.

I have a nice big IBM server box with ESX serving as my entire Engineering lab (Engineering is my name for the IT lab we have here for Star Trek-related historical reasons). The box has a terabyte of RAID5 SATA disks on board for virtual machine storage, but that’s not quite enough for what we’re doing. To get more storage for virtual machines, I decided that I could take advantage of our Sun x4500 server with Solaris, to allocate another terabyte of storage. Solaris supports iSCSI targets in ZFS, so that seemed like the way to go. The x4500 has four gigabit network interface cards on board, three of which I haven’t been using so far, so I decided a good way to go would be to add a gigabit network interface to my IBM ESX server and use a crossover CAT6 cable to direct-connect them and provide a dedicated gigabit storage “network” for iSCSI.

Here are my lab notes from getting that setup.

Setting up iSCSI storage on Solaris and then getting it mounted on ESX Server

First, add a physical nic to VMware Server if required. iSCSI is only supported on Gigabit ethernet in VMware ESX 3.x. If you don’t know how to do this, get your hardware dude to take care of it, and you go take a remedial computer class.

Next, add a physical nic to Solaris box if required. This was not required in my case because my x4500 server had a spare port. Hook the Solaris box nic to the ESX box nic with a crossover cable or via a gigabit switch.

Next configure the nic in Solaris like this: (assuming you know the device name). On the x4500 the built in ethernet interfaces are called e1000g0, e1000g1, e1000g2, and e1000g3. I was already using e1000g0 for the main interface to the network.

ifconfig e1000g1 plumb
ifconfig e1000g1 192.168.254.2 netmask 255.255.255.0 up

Check to see if it worked with this:

ifconfig -a

You should see that e1000g1 now has an IP address assigned to it.

Next, add the file for the new interface’s IP address under /etc

cat > /etc/hostname.e1000g1
192.168.254.2
CTRL-D

Next, edit /etc/inet/hosts and add a line for your new IP address. You might have to chmod +w hosts before editing and chmod -w hosts after. Follow the format in the file. Set a new hostname for your new interface

For example,

192.168.254.2 ss003stor.corporate.ae.ca ss003stor

Make sure the file ends with a blank line.

Next, edit the /etc/inet/netmasks file. You might have to do the chmod ting again because the file is normally set to read only. Add this line:

192.168.254.2 255.255.255.0

Make sure the file ends with a blank line.

Next, add a new zfs volume that we will use as an iSCSI target. This example assumes that the zfs filesystem data/vols already exists.

zfs create -V 1T data/vols/vs-vmfs01

This creates a 1 Terabyte data volume. It is a volume, and not a zfs filesystem, so you can’t see it with the ls command, but you can see it with zfs list -r data/vols.

Next, make the volume shared via iSCSI.

zfs set shareiscsi=on data/vols/vs-vmfs01

Next, create an iSCSI target portal group (tpgt), and set it up to only listen for iSCSI connections on your new dedicated storage network card.

iscsitadm create tpgt 1
iscsitadm modify tpgt -i 192.168.254.2 1

Make sure the group got created:

iscsitadm list tpgt -v
TPGT: 1
      IP Address: 192.168.254.2

Add the target we already created to the group.

iscsitadm modify target -p 1 data/vols/fs-vmfs01

Make sure that worked.

iscsitadm list target -v

Target: data/vols/vs-vmfs01
    iSCSI Name: iqn.bla.bla.bla
    Alias: data/vols/vs-vmfs01
    Connections: 1
        Initiator:
            iSCSI Name: iqn.bla.bla
            Alias: vs200.bla.bla
    ACL list:
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 0
            GUID: blabla
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0T
            Backing store: /dev/zvol/rdsk/data/vols/vs-vmfs01
            Status: online

Now the target is ready to get mounted and formatted by the ESX Server.

The next step is to configure the ESX Server.

First, go to the virtual infrastrucutre client and select the server’s configuraion tab. Click on Network Adapters and make sure your network card shows up in the list. Click on Networking, and click Add Networking.

In the dialog, select the VMkernel radio button and click Next. Select “Create a Virtual Switch” and make sure your new network interface is selected. Click next. Under Network Label, give it a name like vmkernel storage, and select Next and then Finish.

On your new virtual switch, click Properties. In the Ports dialog, click Add, and select VMKernel. Call the network label VMkernel Storage, and give it an IP address on the same subnet as your Solaris dedicated storage server network card, with the same network mask. Click Next and Finish.

Next, go to the console command line via ssh or on the actual console. Login as root. Type the following commands to enable iSCSI software initiator, open a port in the firewall, and setup and scan the iSCSI target:

/usr/sbin/esxcfg-swiscsi -e
(enables iSCSI software initiator)
/usr/sbin/esxcfg-firewall -e swISCSIClient
(opens iSCSI client port in firewall)
/usr/sbin/vmkiscsi-tool -D -a 192.168.254.2 vmhba40
(tells the iSCSI software adapter to request iSCSI targets from the storage box)
/usr/sbin/esxcfg-rescan vmhba40
(tells the iSCSI initiator to scan for new targets)

Next, go to the Storage tab and click Add Storage. Choose the Disk/LUN radio button and click Next. Select your new iSCSI target from the list and click Next. Follow the prompts, and ESX server will format the volume with vmfs and mount it. You should now be able to start using it right away.

I have an older IBM Thinkpad R40. It has a build of OpenSolaris on it from the fourth quarter last year. I have been trying to get the Intel Centrino wireless card to work on it, and playing around with nwam. I enabled nwam last week, had no success (it detects a jillion wireless access points out the window of my office but won’t connect to any of them) and I disabled it again and switched back to manual network configuration. Now, when I’m on a wired connection, it works for a while and then forgets which device is the gateway device and I lose my network connection. Going into the network administration panel and setting the gateway device usually gets it back for a couple of minutes, but then it goes away again. It’s very frustrating.

So far, I’ve used svcadm to ensure that the nwamd is disabled and the svc:/network/physical:default service is enabled, which is how it was setup before I enabled nwam initially. I’m not sure where I should go next. I was just about ready to do a live upgrade on this machine anyways, so I might just do that and see if the problem fixes itself. I’m running Solaris on this box to try to become more familiar with the intricacies of Solaris, since we have a few storage servers in production running Solaris, but I need this machine to at least work on a basic level, otherwise it’ll be back to Ubuntu on it.

I have been following the progress of Solaris Express DE, and I’m even running it on my x4500, because I needed a particular ZFS feature before it became available in the official Solaris. Recently the latest drop of SXDE came out, and it reportedly has the new Solaris installer in it. I downloaded a DVD iso and tried to boot it in a VM in VMware Server, without any luck. I thought maybe the iso file was buggered up somehow, despite the MD5 checksum being correct, so I burned the iso to a physical disk and tried booting it on my laptop. I got this frustrating ever-so-close-but-not-quite message from the installer:

This installer requires 768 MB of RAM to run. This machine has 767 MB of RAM.

Doh!

I’ll try again on my home server.

I’ve managed to upgrade the OS on our x4500 server from Solaris 10 06/06 to Solaris Express Developer Edition build 64 using Live Upgrade. Here’s the
entire process from beginning to end.

Before I initially set up the x4500, I had done a minimal amount of research on using Live Upgrade that suggested that it would be a really good idea to leave some empty space on my system’s boot disk. I made sure to do this during the initial configuration.

It’s also important to note that if you are running zones (Solaris virtual machines) and you are starting with Solaris 10 06/06 or earlier, you have to stop your zones, uninstall them and delete their configurations before you do a live upgrade. At some point after Solaris 10 06/06 this became unnecessary, but I don’t know which build number became capable of live upgrading a system with active non-root zones.

First, to see where we were at, I looked at the contents of /etc/release, which were like this:

                Solaris 10 6/06 s10x_u2wos_09a X86
   Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                Use is subject to license terms.
                     Assembled 09 June 2006

That indicates the June 2006 release of Solaris 10.

Next, I needed to figure out which disk was the boot disk. The x4500 has an amazing 8 sata controllers, and 48 disks in it, most of which are part of a big giant zfs pool, but two of which are not, and one of those two on my system is the boot disk. To figure out the boot disk, I just ran the mount command and looked for the name of the disk holding the root partition, “/”. It turned out for some inexplicable reason to be c6t0d0s0.

Next, I needed to prepare another partition on the boot disk in the empty space I had prepared earlier (like a tv chef). I ran the format command to do
this. The session looked like this:

format /dev/dsk/c6t0d0
partition> format
format> 3
Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[0]: 14652
Enter partition size[0b, 0c, 14652e, 0mb, 0gb]: 30gb
partition> quit
format> label
format> quit

The parameter “new starting cyl” is the next cylinder after the highest occupied cylinder on your disk, ignoring the partition called backup, which doesn’t actually use the cylinders it seems to be using.

Now, there is a 30 GB partition called c6t0d0s3 ready and waiting for the Live Upgrade process.

Next, I had to patch the snot out of my server to get it right up to date with all the current patches. I used smpatch to do that, but you can also use the gui patch management console. There’s probably a more streamlined way to do it than I did, buy my way had the advantage that I could figure it out, and it worked. Here’s what I did:

smpatch analyze > list.sh

That puts a list of all patches that the patch management system thinks this machine should install into a file called list.sh. I then edited list.sh to remove all the descriptions of the patches so that it just contained patch numbers like 121119-12. I made each line into a command line for installing each patch number, so after editing the file looked like this:

#!/usr/bin/bash
smpatch upgrade -i 111111-11
smpatch upgrade -i 222222-22
smpatch upgrade -i 333333-33
etc.

Then I made the file executable by going chmod +x list.sh and then ran it. It spent a long time installing patches and then completed nicely. I then rebooted the server by typing init 6.

Next, after the server rebooted, went to Sunsolve and looked up document #72099, which describes necessary patches for using Live Upgrade safely. I
downloaded them all for the x86/64 architecture, unzipped them, and applied them in order according to the document, using a shell script. Each line of
the shell script looked like this:

patchadd /data/patches/118816-03

Once that completed, I rebooted again with init 6.

Next, after the server rebooted, I mounted the build 64 dvd iso and used it to install the Live Upgrade utilities from the version of Solaris that I was going to upgrade to. This is necessary for whatever version of Solaris you will be upgrading to. To mount an iso image, you do this:

lofiadm -a /data/solaris-dvd.iso
mount -F hsfs /dev/logi/1 /mnt

This will mount your dvd image at /mnt. Next I used the pkgrm command to remove the old versions of the Live Upgrade utilities and the pkgadd command to add the new ones.

pkgrm SUNWlur
pkgrm SUNWluu
pkgrm WUNWluzone
pkgadd -d /mnt/Solaris_11/Product SUNWlur
pkgadd -d /mnt/Solaris_11/Product SUNWluu
pkgadd -d /mnt/Solaris_11/Product SUNWluzone

Next I created a duplicate of my boot environment using lucreate. I called the existing boot environment 0606 and the new one b64.

lucreate -c 0606 -n b64 -m /:/defv/dsk/c6t0d0s3:ufs

This takes a long time and essentially copies your boot environment onto the disk you specified. When it’s done you have a copy of your working environment that you can then upgrade, patch or whatever, while the system is running, without affecting your working conifguration.

Next, I used the luupgrade command to upgrade my b64 boot environment to Solaris Express DE build 64 using my dvd image, which again was mounted at the /mnt mount point.

luupgrade -u -n b64 -s /mnt/

This takes quite a while. After the first little while, a percent progress indicator shows up to tell you how it’s doing. When it’s done, you just have to activate the new boot environment and reboot, and you will be running your new version.

luactivate b64
init 6

Once the server reboots, there is a new grub menu selection to boot the new environment. The new environment is the default so if you do nothing the server boots into the new version of Solaris. Once mine was done, I checked the /etc/release file, and it said the following:

                    Solaris Nevada snv_64a X86
   Copyright 2007 Sun Microsystems, Inc.  All Rights Reserved.
                Use is subject to license terms.
                      Assembled 18 May 2007

That indicates Solaris Express Developer Edition build 64, which is what I was hoping for. I also checked my customized services that I had configured in Solaris 10, and everything was still working, so the upgrade was successful.

In the continuing saga to update Solaris, I’ve added the first successful chapter. I need to get to a Solaris build of about b55 or higher, so that the command zfs receive supports the rollback feature prior to the receive operation.

On some of my Solaris machines, the boot disks are small and there is not enough room to do an upgrade over top of the system. I probably have to rebuild those completely, and until I can do that, synchronizing between them will have to be via rsync rather than zfs send / zfs receive. However, our x4500 has a big boot disk, with unpartitioned space, which means I can use Solaris Live Upgrade to update it. Live Upgrade allows you to make a duplicate of your working boot environment on a new partition or disk, upgrade the duplicate to a new version (or even just apply patches to it) and then reboot into that duplicate. It also allows you to revert to the original system in the event you have problems with the upgraded one.

Before I do this on the x4500, I wanted to try it on a non-production machine. To do that, I built a VMWare Server VM with Solaris Express DE build 55, and then yesterday, with the help of these instructions, I did a live upgrade to Solaris Express DE build 64. That worked fine, so I’m encouraged to try a live upgrade on the x4500. It’s running Solaris 11/06, however, not Solaris Express DE, so it’s a bigger step to get to build 64. I think I’m going to start my VM over at the same level as the x4500, and re-do the live upgrade to b64 directly, so I can see what all the steps will need to be on the x4500.

Previously I talked about updating Solaris 11/06 to Solaris Express b55 in order to have the newer zfs receive option that forces a rollback just prior to doing an incremental receive. I tried a few weeks ago, before my time off when my dad passed away, to update one of our storage servers to Solaris Express DE build 55 from Solaris 11/06. The update didn’t go too well, because it stopped saying there wasn’t enough disk space. When I had originally built that machine, I used Solaris’ default configuration of the boot drive, which included a small (4GB or so) root partition. I didn’t have a lot of time at that point to figure out an alternate approach, so I stopped the update, and kept Solaris 11/06.

I’m back to trying to figure that out now, so I installed Solaris 11/06 using the default installation settings for disk partitions, onto a test box. Now I’m running the installer of Solaris Express DE b55 on it to see if I can upgrade it. The first try, I chose the GRUB menu selection that would install Solaris Express plus the developer tools. That resulted in a funny error message that stated something like This installer requires 768 MB of RAM to run. This machine has 767 MB of RAM. Please choose a different method of installation. I laughed between curse words when I read that.

Next I tried installing just Solaris in the GRUB boot menu. The installer tested the installation profile, determined that it would have to reconfigure the partitions on disk to fit this version of Solaris, and asked for a writable backup area so that it could backup my stuff. I specified a shared zfs partition on our Thumper box, and away it went. It is now in the process of doing the upgrade. When it’s done I’ll check it out to see if my stuff is all still there. Wait, crap, it has failed. It says that the share point on the NFS server must have o+rw rights. Doh. I’ll fix that and try again.

Now the machine is working through the installer. It’s an amazing 1% done so far.

Well, it failed again. So far not too good. I may have to backup important stuff in my boot disks to zfs, export my zfs filesystems and reinstall Solaris from scratch on the boxes I’m interested in. Luckily the Thumper box has a bigger boot partition and it should upgrade with no problems.

We’re using Solaris 10 with zfs as a target for disk-to-disk backups off of our production networks. Our old system used Linux and the filesystems we tried (ext3 and reiserfs) were awkward, slow and poorly able to handle millions of files on a single filesystem. We should probably have used XFS, but by the time we were strongly considering changing what we were using, zfs showed up on the scene. We needed more storage, and instead of buying more of the crappy HP MSA20 SATA enclosures we’d been using and having lots of problems with (weekly firmware updates and inexplicable occurances of the storage just unmounting and refusing to remount without rebooting), we decided to buy our nice Sun x4500 server with 24 TB of disk space.

We’re using rsync to synchronize from our production servers to the primary backup box, the x4500 server, in zfs. Then, we make zfs snapshots and make those available to the network administrators in our offices over http. That way, they can get files back from any day pretty easily. It works well and is very reliable.

We also have a second Solaris 10 storage server in our colocated site, with about 6 TB of storage in SATA shelves in an IBM DS4300 SAN. We want to make replicas of backup data on our x4500 onto this Solaris box so that we can purge any vestiges of local backups from our system. The zfs filesystem has a neat feature for doing this, called zfs send and zfs receive. These commands allow you to stream a whole zfs filesystem out to another medium and then retrieve it back and recreate that zfs filesytem again. If you pipe the output of zfs send through ssh to another box and then to zfs receive, you end up with a duplicate zfs filesystem created on another server. Sweet! Then, you can use a differential zfs send to update a snapshot, so that instead of re-sending the entire data set again the next day, you can just send the differences between yesterday and today and get another full snapshot. Unfortunately, in the build of Solaris we are using, which is Solaris 10 Enterprise 11/06, zfs send and receive for differential snapshots over the network doesn’t work, because the receiving side always makes some insignificant change to the filesystem before the receive starts, and then the differential receive fails because the two endpoints don’t match.

Later builds of OpenSolaris have a new option on zfs receive that forces it to do a rollback immediately prior to accepting the sent data. Our build doesn’t have that option. I’m now faced with the prospect of either a) unmounting the destination filesystem during the send/receive, which makes my backup snapshots unavailable, or b) updating Solaris to an OpenSolaris build. I need to figure out which is more appropriate. I also need to figure out how to nondestructively upgrade from Solaris 10 11/06 to Solaris Express DE or some other newer build.

I read today about an easy-to-use exploit on Solaris to allow login via telnet without even a password. I tried it out on a couple of our Solaris boxes and it worked. The exploit works as follows: Add -l “-rusername” to the telnet commandline, where username is the name of the user you want to become as you telnet into the Solaris box. For example, this will login as “bob” on a server called “solaris.box.com”:

telnet -l "-rbob" solaris.box.com

The exploit doesn’t work for root (-l “rroot” doesn’t work) but it worked on my Solaris boxes for any other users that I tried.

Here’s how you turn off Telnet on Solaris 10: Become root and then run this command.

svcadm disable telnet

I have two questions: In 2007, why the heck is telnet service enabled by default on Solaris 10? and Why didn’t I turn that off when I set up those boxes. The answer to the second question is that I stupidly expected it to be disabled by default.