Home > Sun Solaris, Virtualization, zfs > Solaris iSCSI Target with ESX 3.02 Server

Solaris iSCSI Target with ESX 3.02 Server

I have a nice big IBM server box with ESX serving as my entire Engineering lab (Engineering is my name for the IT lab we have here for Star Trek-related historical reasons). The box has a terabyte of RAID5 SATA disks on board for virtual machine storage, but that’s not quite enough for what we’re doing. To get more storage for virtual machines, I decided that I could take advantage of our Sun x4500 server with Solaris, to allocate another terabyte of storage. Solaris supports iSCSI targets in ZFS, so that seemed like the way to go. The x4500 has four gigabit network interface cards on board, three of which I haven’t been using so far, so I decided a good way to go would be to add a gigabit network interface to my IBM ESX server and use a crossover CAT6 cable to direct-connect them and provide a dedicated gigabit storage “network” for iSCSI.

Here are my lab notes from getting that setup.

Setting up iSCSI storage on Solaris and then getting it mounted on ESX Server

First, add a physical nic to VMware Server if required. iSCSI is only supported on Gigabit ethernet in VMware ESX 3.x. If you don’t know how to do this, get your hardware dude to take care of it, and you go take a remedial computer class.

Next, add a physical nic to Solaris box if required. This was not required in my case because my x4500 server had a spare port. Hook the Solaris box nic to the ESX box nic with a crossover cable or via a gigabit switch.

Next configure the nic in Solaris like this: (assuming you know the device name). On the x4500 the built in ethernet interfaces are called e1000g0, e1000g1, e1000g2, and e1000g3. I was already using e1000g0 for the main interface to the network.

ifconfig e1000g1 plumb
ifconfig e1000g1 192.168.254.2 netmask 255.255.255.0 up

Check to see if it worked with this:

ifconfig -a

You should see that e1000g1 now has an IP address assigned to it.

Next, add the file for the new interface’s IP address under /etc

cat > /etc/hostname.e1000g1
192.168.254.2
CTRL-D

Next, edit /etc/inet/hosts and add a line for your new IP address. You might have to chmod +w hosts before editing and chmod -w hosts after. Follow the format in the file. Set a new hostname for your new interface

For example,

192.168.254.2 ss003stor.corporate.ae.ca ss003stor

Make sure the file ends with a blank line.

Next, edit the /etc/inet/netmasks file. You might have to do the chmod ting again because the file is normally set to read only. Add this line:

192.168.254.2 255.255.255.0

Make sure the file ends with a blank line.

Next, add a new zfs volume that we will use as an iSCSI target. This example assumes that the zfs filesystem data/vols already exists.

zfs create -V 1T data/vols/vs-vmfs01

This creates a 1 Terabyte data volume. It is a volume, and not a zfs filesystem, so you can’t see it with the ls command, but you can see it with zfs list -r data/vols.

Next, make the volume shared via iSCSI.

zfs set shareiscsi=on data/vols/vs-vmfs01

Next, create an iSCSI target portal group (tpgt), and set it up to only listen for iSCSI connections on your new dedicated storage network card.

iscsitadm create tpgt 1
iscsitadm modify tpgt -i 192.168.254.2 1

Make sure the group got created:

iscsitadm list tpgt -v
TPGT: 1
      IP Address: 192.168.254.2

Add the target we already created to the group.

iscsitadm modify target -p 1 data/vols/fs-vmfs01

Make sure that worked.

iscsitadm list target -v

Target: data/vols/vs-vmfs01
    iSCSI Name: iqn.bla.bla.bla
    Alias: data/vols/vs-vmfs01
    Connections: 1
        Initiator:
            iSCSI Name: iqn.bla.bla
            Alias: vs200.bla.bla
    ACL list:
    TPGT list:
        TPGT: 1
    LUN information:
        LUN: 0
            GUID: blabla
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 1.0T
            Backing store: /dev/zvol/rdsk/data/vols/vs-vmfs01
            Status: online

Now the target is ready to get mounted and formatted by the ESX Server.

The next step is to configure the ESX Server.

First, go to the virtual infrastrucutre client and select the server’s configuraion tab. Click on Network Adapters and make sure your network card shows up in the list. Click on Networking, and click Add Networking.

In the dialog, select the VMkernel radio button and click Next. Select “Create a Virtual Switch” and make sure your new network interface is selected. Click next. Under Network Label, give it a name like vmkernel storage, and select Next and then Finish.

On your new virtual switch, click Properties. In the Ports dialog, click Add, and select VMKernel. Call the network label VMkernel Storage, and give it an IP address on the same subnet as your Solaris dedicated storage server network card, with the same network mask. Click Next and Finish.

Next, go to the console command line via ssh or on the actual console. Login as root. Type the following commands to enable iSCSI software initiator, open a port in the firewall, and setup and scan the iSCSI target:

/usr/sbin/esxcfg-swiscsi -e
(enables iSCSI software initiator)
/usr/sbin/esxcfg-firewall -e swISCSIClient
(opens iSCSI client port in firewall)
/usr/sbin/vmkiscsi-tool -D -a 192.168.254.2 vmhba40
(tells the iSCSI software adapter to request iSCSI targets from the storage box)
/usr/sbin/esxcfg-rescan vmhba40
(tells the iSCSI initiator to scan for new targets)

Next, go to the Storage tab and click Add Storage. Choose the Disk/LUN radio button and click Next. Select your new iSCSI target from the list and click Next. Follow the prompts, and ESX server will format the volume with vmfs and mount it. You should now be able to start using it right away.

About these ads
  1. Bernhard
    2008-05-01 at 13:21 | #1

    Hi Scott
    just testing a VM3.5.0 with a netapp 3020 as storage device. I did the rtfm over and over, googled trough the web, but I couldnt setup an iscsi connection to the filer though I did it with w2003 fifty times before. Finally found your blog and with little hope (Oh that want help, that is shurley done in background via Infrastructure client) opened up a console and keyed in anyway:

    /usr/sbin/esxcfg-firewall -e swISCSIClient

    And all of a sudden without rescanning or something like that my test LUN popped up.
    (sort of “beam me up, Scotty)

    That really saved my day (at least the rest of it)!
    Using the Infrastructure Client for configuring ISCSI opening the ports in the firewall isn´t done automatically at all and that is not mentioned in manuals or other pdf´s by vmware.

    great Job!
    kind regards
    Bernhard (Stuttgart/Germany)

  2. 2008-05-20 at 13:56 | #2

    I am about to set up similar in a home lab. Currenly using two PE1650s as ESX hosts and a PE2500 with OpenFiler. I planning on installing Solaris x86 on the PE2500 and using ISCSI / ZFS.

    thanks.

  3. 2008-06-01 at 17:14 | #3

    …and if you’re looking for completely FREE and powerful iSCSI Target for Microsoft Windows you’re welcomed to give a try to StarWind iSCSI Target from Rocket Division Software.

    Just in case here’s an URL:

    StarWind iSCSI Target

    Arigato!

    -ichiro

  4. Chris
    2008-09-25 at 11:19 | #4

    After following all these directions ESX can see the LUNS and everything is fine; however when I add it to the storage datastore it begins formatting and I get the following error:

    “The request failed because the remote server took too long to respond.”

    A while after the error ESX claims the format is complete; however there is definitely something wrong with the storage and it is unusable.

    There is no firewall on my solaris box and my ZFS volume that is shared via ISCSI is 1TB

    Anyone else run into this problem?

  5. Giang nguyen
    2009-04-22 at 07:24 | #5

    Did you test more than 1 LUN per Target with ESX? I can only see LUN 0 per target from Solaris 10 iscsi target. If you have work around that’s great

  6. Phil Forrest
    2009-05-17 at 12:35 | #6

    You may have a typo…..

    in the command “iscsitadm modify target -p 1 data/vols/fs-vmfs01″ aren’t you intending to call the volume data/vols/vs-vmfs01 ?

    Thanks,
    Phil

  7. Ken
    2009-06-26 at 15:20 | #7

    I’m having a problem running the “iscsitadm modify target -p 1 data/vols/vs-vmfs01″.
    When ever I run that command I receive a “iscsitadm: Error Target not found” message.
    Here is what is in my zfs list

    NAME USED AVAIL REFER MOUNTPOINT
    data 168K 2.00T 19K /data
    data/vols 37K 2.00T 19K /data/vols
    data/vols/vs-vmfs01 18K 2.00T 18K /data/vols/vs-vmfs01

    I have also tried creating a target using “iscsitadm create target -u 0 -b data/vols/vs-vmfs01 vmware” but I receive an error of “Error for type ‘disk’ backing must be a regular file”

    Any suggestions?

    • Ken
      2010-04-02 at 08:22 | #8

      Sorry, I was going through my old notes and found that I never posted the solution to the problem that I was having.

      Based on
      http://osdir.com/ml/os.solaris.opensolaris.storage.general/2007-12/msg00067.html
      I found that ZFS shareiscsi=on does not store iscsi settings (anymore?). To fix this you need to use the iscsiadm to create your iscsi targets.

      Now as I posted above I was having problems with creating iscsi targets using iscsitadm due to backing errors. This was caused by incorrectly built ZFS volumes for iscsi. When creating a ZFS volume for iscsi you need to use the -V option and give it a volume size.
      ex:
      zfs create -V 1T data/vols/vs-vmfs01
      (This creates a 1 TB ZFS volume)
      iscsitadm create target -b /dev/zvol/rdsk/data/vols/vs-vmfs01 san1
      (This creates a iscsi target called san1. Please note that you need to use the full path to the ZFS volume (/dev/zvol/rdsk/data/vols/vs-vmfs01) rather than the link mount point of data/vols/vs-vmfs01)
      Once this is done you can modify your iscsi target and add in your tpgt
      I hope this helps someone else in the future. I have full build instructions if any wants them, just reply to this post.

      • Jon
        2010-10-27 at 10:47 | #9

        Hi Ken, I’d like to take you up on your offer of detailed instructions. I’m really new to Solaris and ZFS but know ESX well. Thanks

  8. Jamie
    2010-05-12 at 19:40 | #10

    Hi Scott,

    I am having a similar issue and wondering you have any clue about it. I am setting up a solaris 10 virtual machine based on MS hyper-v connecting to windows iscsi target. But there are some connection issue. I cannot see the target in Solaris system. Please send me an email, if you know anything about it.

    Thank you,
    Jamie

  1. 2008-08-31 at 09:48 | #1

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: