We all know that SAN's provide an excellent way to centralise your storage, and any SAN in a big company is likely to require a level of redundancy in the connectivity. This entry will run through how to enable multipathing on Solaris.
This was written for Solaris 11 but should be relevant to Solaris 10.
The setup here is a T2000 running Solaris 11 with a Emulex LP10000DC HBA - thats a dual port card, connecting to a homebuilt 2u rackmount server running OpenIndiana, again with an Emulex LP10000DC. The servers are simply cross-connected port to port.
Step 1 - checking your setup
First identify your Fibre Channel devices
root@t2000:~# cfgadm -vl | ggrep -B1 fc
c4 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc
c5 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc
Now you know your controller, you can spot your disks. Since the SAN target has two adapters you can see two devices differing only by one number and one letter, eg c4t10000000C95187CAd0 and c5t10000000C95187CBd0
root@t2000:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
1. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
2. c4t10000000C95187CAd0 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,0
3. c4t10000000C95187CAd1 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,1
4. c4t10000000C95187CAd2 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,2
5. c4t10000000C95187CAd3 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,3
6. c4t10000000C95187CAd4 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,4
7. c4t10000000C95187CAd5 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,5
8. c4t10000000C95187CAd6 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,6
9. c4t10000000C95187CAd7 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,7
10. c4t10000000C95187CAd8 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,8
11. c4t10000000C95187CAd9 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,9
12. c5t10000000C95187CBd0 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,0
13. c5t10000000C95187CBd1 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,1
14. c5t10000000C95187CBd2 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,2
15. c5t10000000C95187CBd3 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,3
16. c5t10000000C95187CBd4 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,4
17. c5t10000000C95187CBd5 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,5
18. c5t10000000C95187CBd6 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,6
19. c5t10000000C95187CBd7 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,7
20. c5t10000000C95187CBd8 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,8
21. c5t10000000C95187CBd9 <OI-COMSTAR-1.0-10.00GB>
/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,9
Step 2 - enable mpxio
The command to enable mpxio is stmsboot. From a quick look at the man page you'll see it applies to fp, mpt, mpt_sas, and iscsi storage. We don't want to multipath c2 as those are our boot disks!
To enable mpxio on the fibre channel devices, stmsboot -D fp -e
root@t2000:~# stmsboot -D fp -e
WARNING: This operation will require a reboot.
Do you want to continue ? [y/n] (default: y) y
The changes will come into effect after rebooting the system.
Reboot the system now ? [y/n] (default: y) y
As you reboot, you'll see the system reconfigure itself
Boot device: /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a File and args:
SunOS Release 5.11 Version 11.0 64-bit
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
stmsboot: Root is on ZFS
stmsboot: configuring devices
stmsboot: vfstab has been updated
stmsboot: dump on ZFS, no dumpadm update required
stmsboot: now regenerating boot archive
Hostname: t2000
Step 3 - check the results
Your fibre channel devices remain untouched...
root@t2000:~# cfgadm -vl | ggrep -B1 fc
c4 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc
c5 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc
The format command shows the biggest change...
root@t2000:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t600144F0ABA8000000004F311DDD0001d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311ddd0001
1. c0t600144F0ABA8000000004F311DDF0002d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311ddf0002
2. c0t600144F0ABA8000000004F311DE10003d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de10003
3. c0t600144F0ABA8000000004F311DE30004d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de30004
4. c0t600144F0ABA8000000004F311DE40005d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de40005
5. c0t600144F0ABA8000000004F311DE60006d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de60006
6. c0t600144F0ABA8000000004F311DE80007d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de80007
7. c0t600144F0ABA8000000004F311DE90008d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311de90008
8. c0t600144F0ABA8000000004F311DEB0009d0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311deb0009
9. c0t600144F0ABA8000000004F311DEC000Ad0 <OI-COMSTAR-1.0-10.00GB>
/scsi_vhci/ssd@g600144f0aba8000000004f311dec000a
10. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@0,0
11. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>
/pci@780/pci@0/pci@9/scsi@0/sd@1,0
You can use the stmsboot -L command to view the changes in mapping, you can see that the two paths to d0 have both been mapped correctly.
root@t2000:~# stmsboot -L
non-STMS device name STMS device name
------------------------------------------------------------------
/dev/rdsk/c5t10000000C95187CBd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0
/dev/rdsk/c5t10000000C95187CBd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0
/dev/rdsk/c5t10000000C95187CBd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0
/dev/rdsk/c5t10000000C95187CBd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0
/dev/rdsk/c5t10000000C95187CBd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0
/dev/rdsk/c5t10000000C95187CBd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0
/dev/rdsk/c5t10000000C95187CBd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0
/dev/rdsk/c5t10000000C95187CBd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0
/dev/rdsk/c5t10000000C95187CBd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0
/dev/rdsk/c5t10000000C95187CBd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0
/dev/rdsk/c4t10000000C95187CAd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0
/dev/rdsk/c4t10000000C95187CAd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0
/dev/rdsk/c4t10000000C95187CAd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0
/dev/rdsk/c4t10000000C95187CAd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0
/dev/rdsk/c4t10000000C95187CAd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0
/dev/rdsk/c4t10000000C95187CAd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0
/dev/rdsk/c4t10000000C95187CAd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0
/dev/rdsk/c4t10000000C95187CAd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0
/dev/rdsk/c4t10000000C95187CAd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0
/dev/rdsk/c4t10000000C95187CAd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0
Step 4 - testing the resiliency
Check your starting point, for my setup I'd expect 2 paths, and for both to be working.
root@t2000:~# mpathadm list lu
/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
Total Path Count: 2
Operational Path Count: 2
If I unplug one fibre and cause the system to use the disk i/o you'll see it spots one path has gone.
root@t2000:~# mpathadm list lu
/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
Total Path Count: 1
Operational Path Count: 1
Replug the fibre cable
root@t2000:~# mpathadm list lu
/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2
Total Path Count: 2
Operational Path Count: 2
/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2
Total Path Count: 2
Operational Path Count: 2
No comments:
Post a Comment