Managing VMWare ESX without VMWare's Windows Client Part 1

For a number of years since switching to a Mac the only need for Windows has been for the VMWare client for my home lab. This has worked reasonably well through the XP years, and survived a migration to Windows 7 even, but recently whatever Windows 10 ‘preload’ does has broken my Windows 7 VM’s while leaving them unable to upgrade to Windows 10 and finally the moment has arrived where I need to break free however painful.

VMWare Fusion Pro looks like a good way forward, but given that my next mobile device to replace my aging Macbook Pro is going to be an iPad and keyboard that doesn’t work as a way forward either. I needed something lightweight and command line.

Step forward the libvirt project and its tools - especially virsh. I do remember its inception many years ago as a way to unify management of the then iffy KVM, the likely at the time winner XEN, plus the other smaller players like lxc.

Installing virt-manager

# yum install virt-manager

Connecting to the ESX server… if like myself you don’t have a full SSL certificate authority setup you’ll need the ‘?no_verify=1’. Omit it in a production environment.

$ virsh -c esx://192.168.1.73/?no_verify=1
Enter username for 192.168.1.73 [root]: 
Enter root's password for 192.168.1.73: 
Welcome to virsh, the virtualisation interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

Connecting to the ESX server… if like myself you don’t have a full SSL certificate authority setup you’ll need the ‘?no_verify=1’. Omit it in a production environment. First thing you’re likely to need is to see what VM’s you have. In virsh parlance - ‘domains’.

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 1     switch1                        running
 2     hopper                         running
 3     dns1                           running
 -     dns2                           shut off
 -     ldap1                          shut off
- - - - - - - - - - - -8< - - - - - - - - - - - - - -
 -     solweb2                        shut off
 -     solwlserv1                     shut off
 -     solwlserv2                     shut off
 -     switch2                        shut off

Next thing is you’ll need to start some of them...

virsh # start linc7web1
Domain linc7web1 started

...and of course, stop some of them.

virsh # shutdown linc7lb1
error: Failed to shutdown domain linc7lb1
error: internal error: HTTP response code 500 for call to 'ShutdownGuest'. Fault: ServerFaultCode - Cannot complete operation because VMware Tools is not running in this virtual machine.

If you see the above on a VM, install the vmware tools (e.g. yum install -y open-vm-tools).

virsh # shutdown linc7lb1
Domain linc7lb1 is being shutdown

RAC-in-a-box - Part 1: Building a Linux-based RAC 11gr2 cluster on VMWare

My DBA colleagues have long told me the tale of 'RAC on a rope' - the battered demonstration kit that Oracle have used to show off RAC to customers.

I've long been intrigued by RAC, and puzzled by its apparent instability on our work systems and so decided to take advantage of the OTN accounts ability to download and use Oracle software for trial purposes.  This coincided with a time where VMWare ESXi 4 was released on a free to use basis, and servers without explicit virtualisation features were being dumped on eBay.

All the ingredients for my very own RAC on a rope based on either Linux or Solaris... ideal for proving my suspicion that our preferred vendors OS and hardware were the cause of the problems and not RAC.

First off, the hardware. The Fujitsu Primergy RX220 is a 1U high rackmount server, based around a dual socket 940 system board. With a pair of dual core Opteron 280's and 16Gb RAM installed it packs a lot into its chassis. What does present a problem are the 2 SATA bays, which mean theres not a lot of storage and no scope for splitting storage into data and system.

I originally completed this build in mid2010, so in many ways this is a retrospective look at what I did. Unfortunately my workplace has taken much of my time since then and so details are not as fresh in my mind as they should be.

The first step was to install VMWare ESXi onto a USB key, this is the 'boot disk', and while its pretty slow, it leaves the main SATA disks free for the Virtual Machines we need to define.

The four VM's are:-

• ldns1
• liscsi1
• lrac1
• lrac2  

These were named with expandability in mind, highly available DNS, Multipathed iSCSI, and nodes 3 and 4 within the cluster, however from earlier attempt with Solaris+10g I expected resources to be tight and to this date haven't explored those options.

I started out building a Linux cluster based around Centos 5.5, which if you aren't aware is a freely available RedHat clone. I had originally planned on using Oracles own OEL, however I got bored waiting on the Export Validation to clear. This impatience cost me a little later on, but nothing googling can't work around.

---

RAC-in-a-box - Part 1: Building a Linux-based RAC 11gr2 cluster on VMWare

RAC-in-a-box - Part 2: Building the DNS Server

RAC-in-a-box - Part 3: Building the iSCSI Server

RAC-in-a-box - Part 4: Building the RAC nodes

Using COMSTAR to provide fibre disk to VMWARE ESX or other SAN client

During the day I work as a Unix sysadmin for a large company in the banking sector. Recently we've had a nightmare run of issues from our SAN and the systems attached (all from the same vendor).  Given the chaos we've endured, I ordered some fibre channel HBA's and a string of fibre to learn more about a technology that underpins my work on a daily basis.

Unlike work, my available hosts are UltraSPARC and Opteron based but I figure the fundamentals will be the same. Much of the work here is comparable to the iSCSI COMSTAR tutorials written by the likes of Joerg Moellenkamp and Ben Rockwood, but just omitting the iSCSI element.

My adventures doing this would have been easier if I'd bugged Egg to write this excellent summary of fibre channel disk handling first.

The first step is to install your software and move your HBA into target mode. The software is installed with one command, pkg install storage-server.

For Emulex cards this is nice and simple. Edit /kernel/drv/emlxs.conf and change the following settings:-

target-mode=1;
ddi-forceattach=1

Save the file and reboot. Once restarted you can query the HBA with fcinfo.

root@ramjet:~# fcinfo hba-port
HBA Port WWN: 10000000c95eea8f
        Port Mode: Target
        Port ID: 0
        OS Device Name: Not Applicable
        Manufacturer: Emulex
        Model: LP10000
        Firmware Version: 1.91a1 (T2D1.91A1)
        FCode/BIOS Version: 1.50a4
        Serial Number: VM64819498
        Driver Name: emlxs
        Driver Version: 2.60f (2010.08.31.16.55)
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb
        Current Speed: not established
        Node WWN: 20000000c95eea8f

For Qlogic cards its a bit more involved.

First get the PCI device id's

# prtconf -D | grep qlc
            pci1077,138, instance #0 (driver name: qlc)

            pci1077,138, instance #1 (driver name: qlc)

Note I'm using a none Oracle card here, and one from another vendor to boot (Dell or HP i think). In my case I needed to do this for both pci1077,138 and pci1077,2432. Replace 'pci' with 'pciex' if your cards are PCI Express - I recently found out from re-applying this to a new card setup that it doesn't carry over.

update_drv -d -i 'pci1077,138' qlc
update_drv -a -i 'pci1077,138' qlt

If you are using genuine stock Oracle or QLogic cards you'll only need to do the below.

update_drv -d -i 'pci1077,2432' qlc
update_drv -a -i 'pci1077,2432' qlt

Save the file and reboot. Once restarted you can query the HBA with fcinfo.

HBA Port WWN: 21000024ff2ec367
        Port Mode: Target
        Port ID: 0
        OS Device Name: Not Applicable
        Manufacturer: QLogic Corp.
        Model: QLE2462
        Firmware Version: 0.0.0
        FCode/BIOS Version: N/A
        Serial Number: not available
        Driver Name: COMSTAR QLT
        Driver Version: 20100505-1.05
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb 
        Current Speed: not established 
        Node WWN: 20000024ff2ec367
HBA Port WWN: 21000024ff2ec366
        Port Mode: Target
        Port ID: 0
        OS Device Name: Not Applicable
        Manufacturer: QLogic Corp.
        Model: QLE2462
        Firmware Version: 0.0.0
        FCode/BIOS Version: N/A
        Serial Number: not available
        Driver Name: COMSTAR QLT
        Driver Version: 20100505-1.05
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb 
        Current Speed: not established 

        Node WWN: 20000024ff2ec366

Ensure the stmf service is running, if not, start it with svcadm enable stmf:default

root@ramjet:~# svcs stmf
STATE          STIME    FMRI
online         17:31:06 svc:/system/stmf:default

You can now start to configure your volumes. Check stmf has found your HBA:-

root@ramjet:~# stmfadm list-target
Target: wwn.10000000C95EEA8F

Create some backing ZFS volumes

root@ramjet:~# zfs create rpool/vmware_disk
root@ramjet:~# zfs create -V 10g rpool/vmware_disk/vol_01
root@ramjet:~# sbdadm create-lu /dev/zvol/rdsk/rpool/vmware_disk/vol_01
Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f00003ba4040274d405b8a0001  10737418240          /dev/zvol/rdsk/rpool/vmware_disk/vol_01

root@ramjet:~# zfs create -V 10g rpool/vmware_disk/vol_02
root@ramjet:~# sbdadm create-lu /dev/zvol/rdsk/rpool/vmware_disk/vol_02
Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f00003ba4040274d405ba20002  10737418240          /dev/zvol/rdsk/rpool/vmware_disk/vol_02

Sit back and admire your handiwork for a moment...

root@ramjet:~# stmfadm list-lu -v
LU Name: 600144F00003BA4040274D405B8A0001
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/rpool/vmware_disk/vol_01
    View Entry Count  : 0
    Data File         : /dev/zvol/rdsk/rpool/vmware_disk/vol_01
    Meta File         : not set
    Size              : 10737418240
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN  
    Product ID        : COMSTAR      
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Enabled
    Access State      : Active
LU Name: 600144F00003BA4040274D405BA20002
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/rpool/vmware_disk/vol_02
    View Entry Count  : 0
    Data File         : /dev/zvol/rdsk/rpool/vmware_disk/vol_02
    Meta File         : not set
    Size              : 10737418240
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN  
    Product ID        : COMSTAR      
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Enabled
    Access State      : Active

Of course, such output would be unwieldy in a big environment, this is neater:-

root@ramjet:~# stmfadm list-lu
LU Name: 600144F00003BA4040274D405B8A0001
LU Name: 600144F00003BA4040274D405BA20002

Add a 'view' to allow the target device to let initiators see storage - NOTE: this allows access from any host. There are controls to do this but I've not got that far and have no need of it at present. Update: see my later post about Target and host groups.

root@ramjet:~# stmfadm add-view 600144F00003BA4040274D405B8A0001
root@ramjet:~# stmfadm add-view 600144F00003BA4040274D405BA20002

So, power on a host with a fibre hba…

Jan 26 17:43:35 ramjet fct: NOTICE: emlxs0 LINK UP, portid 2, topology Private Loop,speed 2G

You can now allocate your LUN's as required!