Friday, 17 August 2012

Solaris mpathadm and fibre disks


We all know that SAN's provide an excellent way to centralise your storage, and any SAN in a big company is likely to require a level of redundancy in the connectivity. This entry will run through how to enable multipathing on Solaris.

This was written for Solaris 11 but should be relevant to Solaris 10.

The setup here is a T2000 running Solaris 11 with a Emulex LP10000DC HBA - thats a dual port card, connecting to a homebuilt 2u rackmount server running OpenIndiana, again with an Emulex LP10000DC. The servers are simply cross-connected port to port.


Step 1 - checking your setup


First identify your Fibre Channel devices


root@t2000:~# cfgadm -vl | ggrep -B1 fc
c4                             connected    configured   unknown
unavailable  fc-private   n        /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc
c5                             connected    configured   unknown
unavailable  fc-private   n        /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc


Now you know your controller, you can spot your disks. Since the SAN target has two adapters you can see two devices differing only by one number and one letter, eg c4t10000000C95187CAd0 and c5t10000000C95187CBd0

root@t2000:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
       1. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0
       2. c4t10000000C95187CAd0 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,0
       3. c4t10000000C95187CAd1 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,1
       4. c4t10000000C95187CAd2 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,2
       5. c4t10000000C95187CAd3 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,3
       6. c4t10000000C95187CAd4 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,4
       7. c4t10000000C95187CAd5 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,5
       8. c4t10000000C95187CAd6 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,6
       9. c4t10000000C95187CAd7 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,7
      10. c4t10000000C95187CAd8 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,8
      11. c4t10000000C95187CAd9 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,9
      12. c5t10000000C95187CBd0 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,0
      13. c5t10000000C95187CBd1 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,1
      14. c5t10000000C95187CBd2 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,2
      15. c5t10000000C95187CBd3 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,3
      16. c5t10000000C95187CBd4 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,4
      17. c5t10000000C95187CBd5 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,5
      18. c5t10000000C95187CBd6 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,6
      19. c5t10000000C95187CBd7 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,7
      20. c5t10000000C95187CBd8 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,8
      21. c5t10000000C95187CBd9 <OI-COMSTAR-1.0-10.00GB>
          /pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,9

Step 2 - enable mpxio


The command to enable mpxio is stmsboot. From a quick look at the man page you'll see it applies to fp, mpt, mpt_sas, and iscsi storage. We don't want to multipath c2 as those are our boot disks!

To enable mpxio on the fibre channel devices, stmsboot -D fp -e

root@t2000:~# stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) y
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y) y

As you reboot, you'll see the system reconfigure itself

Boot device: /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a  File and args: 
SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
stmsboot: Root is on ZFS
stmsboot: configuring devices

stmsboot: vfstab has been updated
stmsboot: dump on ZFS, no dumpadm update required
stmsboot: now regenerating boot archive
Hostname: t2000


Step 3 - check the results


Your fibre channel devices remain untouched...

root@t2000:~# cfgadm -vl | ggrep -B1 fc
c4                             connected    configured   unknown
unavailable  fc-private   n        /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc
c5                             connected    configured   unknown
unavailable  fc-private   n        /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc

The format command shows the biggest change...

root@t2000:~# format 
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t600144F0ABA8000000004F311DDD0001d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311ddd0001
       1. c0t600144F0ABA8000000004F311DDF0002d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311ddf0002
       2. c0t600144F0ABA8000000004F311DE10003d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de10003
       3. c0t600144F0ABA8000000004F311DE30004d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de30004
       4. c0t600144F0ABA8000000004F311DE40005d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de40005
       5. c0t600144F0ABA8000000004F311DE60006d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de60006
       6. c0t600144F0ABA8000000004F311DE80007d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de80007
       7. c0t600144F0ABA8000000004F311DE90008d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311de90008
       8. c0t600144F0ABA8000000004F311DEB0009d0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311deb0009
       9. c0t600144F0ABA8000000004F311DEC000Ad0 <OI-COMSTAR-1.0-10.00GB>
          /scsi_vhci/ssd@g600144f0aba8000000004f311dec000a
      10. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@0,0
      11. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
          /pci@780/pci@0/pci@9/scsi@0/sd@1,0

You can use the stmsboot -L command to view the changes in mapping, you can see that the two paths to d0 have both been mapped correctly.


root@t2000:~# stmsboot -L
non-STMS device name                    STMS device name
------------------------------------------------------------------
/dev/rdsk/c5t10000000C95187CBd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0
/dev/rdsk/c5t10000000C95187CBd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0
/dev/rdsk/c5t10000000C95187CBd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0
/dev/rdsk/c5t10000000C95187CBd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0
/dev/rdsk/c5t10000000C95187CBd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0
/dev/rdsk/c5t10000000C95187CBd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0
/dev/rdsk/c5t10000000C95187CBd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0
/dev/rdsk/c5t10000000C95187CBd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0
/dev/rdsk/c5t10000000C95187CBd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0
/dev/rdsk/c5t10000000C95187CBd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0
/dev/rdsk/c4t10000000C95187CAd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0
/dev/rdsk/c4t10000000C95187CAd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0
/dev/rdsk/c4t10000000C95187CAd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0
/dev/rdsk/c4t10000000C95187CAd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0
/dev/rdsk/c4t10000000C95187CAd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0
/dev/rdsk/c4t10000000C95187CAd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0
/dev/rdsk/c4t10000000C95187CAd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0
/dev/rdsk/c4t10000000C95187CAd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0
/dev/rdsk/c4t10000000C95187CAd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0
/dev/rdsk/c4t10000000C95187CAd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0

Step 4 - testing the resiliency


Check your starting point, for my setup I'd expect 2 paths, and for both to be working. 

root@t2000:~# mpathadm list lu
        /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
                Total Path Count: 2
                Operational Path Count: 2

If I unplug one fibre and cause the system to use the disk i/o you'll see it spots one path has gone.

root@t2000:~# mpathadm list lu
        /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
                Total Path Count: 1
                Operational Path Count: 1

Replug the fibre cable

root@t2000:~# mpathadm list lu
        /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
                Total Path Count: 2
                Operational Path Count: 2

No comments: