Do Unixnerds Dream of Electric Sleep?: SAN

Showing posts with label SAN. Show all posts

Thursday, 22 November 2012

COMSTAR target groups and host groups

So, after using the storage server I built between multiple servers it soon became apparent that I needed to limit the LUN's each client system could see via iSCSI or Fibre. It turns out that its just a small step onwards from the basic COMSTAR tutorials you see around.

First you need to figure out what groupings you have, if you're running a large company, you might want to split your LUN's between development and production (although you should really be making that split at the storage server level if you really want to avoid resource conflicts between development and production!), you may want to split purely by client machine, or maybe some mixture of both.

Target Groups

These allow the broadest split, for example, at a really granular level, I've split my target groups into two categories, based on whether I expect them to be accessed via iSCSI or via Fibre.

 # stmfadm create-tg san  
 # stmfadm add-tg-member -g san wwn.10000000c95287ca  
 # stmfadm add-tg-member -g san wwn.10000000c95287cb  
 # stmfadm create-tg iscsi  
 # itadm create-target  
 Target iqn.2010-09.org.openindiana:02:92b0fb10-bc5e-6292-9d14-84ea8bad9650 successfully created  
 # stmfadm add-tg-member -g iscsi iqn.2010-09.org.openindiana:02:92b0fb10-bc5e-6292-9d14-84ea8bad9650  
 # svcadm enable -r svc:/network/iscsi/target:default

Host Groups

These allow finer control over what hosts can see what. I take this literally, and define a host group per physical client. However if you need to share LUN's between systems - say for a cluster, then you could specifically set that up here.

 # stmfadm create-hg imac-1  
 # stmfadm create-hg aix-1

Since these are iSCSI hosts, we add an iqn.

 # stmfadm add-hg-member -g imac-1 iqn.<client>  
 # stmfadm add-hg-member -g aix-1 iqn.<client>  
 # stmfadm create-hg rx220-1  
 # stmfadm create-hg t2k-1

Since these are fibre hosts, we add a WWN.

 # stmfadm add-hg-member -g rx220-1 wwn.10000000c9711403  
 # stmfadm add-hg-member -g rx220-1 wwn.10000000c9711404  
 # stmfadm add-hg-member -g t2k-1 wwn.10000000c97e366c  
 # stmfadm add-hg-member -g t2k-1 wwn.10000000c97e366d

Assigning LU's

Now all that remains is to assign your LU's to the correct host and target groups. For example, you may have done something similar to the following to create 8 LU's:-

 # zfs create storagepool/LU_vols  
 > for each in 00 01 02 03 04 05 06 07  
 > do  
 > zfs create -V 100GB storagepool/LU_vols/disk$each  
 > stmfadm create-lu /dev/zvol/rdsk/storagepool/LU_vols/disk$each  
 > done

Heres where the extra step happens:-

 # stmfadm add-view -t iscsi -h imac-1 <LU_disk00>  
 # stmfadm add-view -t iscsi -h imac-1 <LU_disk01>  
 # stmfadm add-view -t iscsi -h smitty-1 <LU_disk02>  
 # stmfadm add-view -t iscsi -h smitty-1 <LU_disk03>  
 # stmfadm add-view -t san -h t2k1-1 <LU_disk04>  
 # stmfadm add-view -t san -h t2k1-1 <LU_disk05>  
 # stmfadm add-view -t san -h rx220-1 <LU_disk06>  
 # stmfadm add-view -t san -h rx220-1 <LU_disk07>

Friday, 17 August 2012

Solaris mpathadm and fibre disks

We all know that SAN's provide an excellent way to centralise your storage, and any SAN in a big company is likely to require a level of redundancy in the connectivity. This entry will run through how to enable multipathing on Solaris.

This was written for Solaris 11 but should be relevant to Solaris 10.

The setup here is a T2000 running Solaris 11 with a Emulex LP10000DC HBA - thats a dual port card, connecting to a homebuilt 2u rackmount server running OpenIndiana, again with an Emulex LP10000DC. The servers are simply cross-connected port to port.

Step 1 - checking your setup

First identify your Fibre Channel devices

root@t2000:~# cfgadm -vl | ggrep -B1 fc
c4 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc
c5 connected configured unknown
unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc

Now you know your controller, you can spot your disks. Since the SAN target has two adapters you can see two devices differing only by one number and one letter, eg c4t10000000C95187CAd0 and c5t10000000C95187CBd0

root@t2000:~# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>

/pci@780/pci@0/pci@9/scsi@0/sd@0,0

1. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>

/pci@780/pci@0/pci@9/scsi@0/sd@1,0

2. c4t10000000C95187CAd0 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,0

3. c4t10000000C95187CAd1 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,1

4. c4t10000000C95187CAd2 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,2

5. c4t10000000C95187CAd3 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,3

6. c4t10000000C95187CAd4 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,4

7. c4t10000000C95187CAd5 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,5

8. c4t10000000C95187CAd6 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,6

9. c4t10000000C95187CAd7 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,7

10. c4t10000000C95187CAd8 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,8

11. c4t10000000C95187CAd9 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0/ssd@w10000000c95187ca,9

12. c5t10000000C95187CBd0 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,0

13. c5t10000000C95187CBd1 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,1

14. c5t10000000C95187CBd2 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,2

15. c5t10000000C95187CBd3 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,3

16. c5t10000000C95187CBd4 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,4

17. c5t10000000C95187CBd5 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,5

18. c5t10000000C95187CBd6 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,6

19. c5t10000000C95187CBd7 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,7

20. c5t10000000C95187CBd8 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,8

21. c5t10000000C95187CBd9 <OI-COMSTAR-1.0-10.00GB>

/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0/ssd@w10000000c95187cb,9

Step 2 - enable mpxio

The command to enable mpxio is stmsboot. From a quick look at the man page you'll see it applies to fp, mpt, mpt_sas, and iscsi storage. We don't want to multipath c2 as those are our boot disks!

To enable mpxio on the fibre channel devices, stmsboot -D fp -e

root@t2000:~# stmsboot -D fp -e

WARNING: This operation will require a reboot.

Do you want to continue ? [y/n] (default: y) y

The changes will come into effect after rebooting the system.

Reboot the system now ? [y/n] (default: y) y

As you reboot, you'll see the system reconfigure itself

Boot device: /pci@780/pci@0/pci@9/scsi@0/disk@0,0:a File and args:

SunOS Release 5.11 Version 11.0 64-bit

stmsboot: Root is on ZFS

stmsboot: configuring devices

stmsboot: vfstab has been updated

stmsboot: dump on ZFS, no dumpadm update required

stmsboot: now regenerating boot archive

Hostname: t2000

Step 3 - check the results

Your fibre channel devices remain untouched...

root@t2000:~# cfgadm -vl | ggrep -B1 fc

c4 connected configured unknown

unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2,1/fp@0,0:fc

c5 connected configured unknown

unavailable fc-private n /devices/pci@7c0/pci@0/pci@1/pci@0,2/lpfc@2/fp@0,0:fc

The format command shows the biggest change...

root@t2000:~# format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

0. c0t600144F0ABA8000000004F311DDD0001d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311ddd0001

1. c0t600144F0ABA8000000004F311DDF0002d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311ddf0002

2. c0t600144F0ABA8000000004F311DE10003d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de10003

3. c0t600144F0ABA8000000004F311DE30004d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de30004

4. c0t600144F0ABA8000000004F311DE40005d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de40005

5. c0t600144F0ABA8000000004F311DE60006d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de60006

6. c0t600144F0ABA8000000004F311DE80007d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de80007

7. c0t600144F0ABA8000000004F311DE90008d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311de90008

8. c0t600144F0ABA8000000004F311DEB0009d0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311deb0009

9. c0t600144F0ABA8000000004F311DEC000Ad0 <OI-COMSTAR-1.0-10.00GB>

/scsi_vhci/ssd@g600144f0aba8000000004f311dec000a

10. c2t0d0 <SEAGATE-ST973401LSUN72G-0556 cyl 14087 alt 2 hd 24 sec 424>

/pci@780/pci@0/pci@9/scsi@0/sd@0,0

11. c2t1d0 <FUJITSU-MAY2073RCSUN72G-0501 cyl 14087 alt 2 hd 24 sec 424>

/pci@780/pci@0/pci@9/scsi@0/sd@1,0

You can use the stmsboot -L command to view the changes in mapping, you can see that the two paths to d0 have both been mapped correctly.

root@t2000:~# stmsboot -L

non-STMS device name STMS device name

------------------------------------------------------------------

/dev/rdsk/c5t10000000C95187CBd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0

/dev/rdsk/c5t10000000C95187CBd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0

/dev/rdsk/c5t10000000C95187CBd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0

/dev/rdsk/c5t10000000C95187CBd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0

/dev/rdsk/c5t10000000C95187CBd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0

/dev/rdsk/c5t10000000C95187CBd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0

/dev/rdsk/c5t10000000C95187CBd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0

/dev/rdsk/c5t10000000C95187CBd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0

/dev/rdsk/c5t10000000C95187CBd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0

/dev/rdsk/c5t10000000C95187CBd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0

/dev/rdsk/c4t10000000C95187CAd9 /dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0

/dev/rdsk/c4t10000000C95187CAd8 /dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0

/dev/rdsk/c4t10000000C95187CAd7 /dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0

/dev/rdsk/c4t10000000C95187CAd6 /dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0

/dev/rdsk/c4t10000000C95187CAd5 /dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0

/dev/rdsk/c4t10000000C95187CAd4 /dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0

/dev/rdsk/c4t10000000C95187CAd3 /dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0

/dev/rdsk/c4t10000000C95187CAd2 /dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0

/dev/rdsk/c4t10000000C95187CAd1 /dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0

/dev/rdsk/c4t10000000C95187CAd0 /dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0

Step 4 - testing the resiliency

Check your starting point, for my setup I'd expect 2 paths, and for both to be working.

root@t2000:~# mpathadm list lu

/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2

Total Path Count: 2

Operational Path Count: 2

If I unplug one fibre and cause the system to use the disk i/o you'll see it spots one path has gone.

root@t2000:~# mpathadm list lu

/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2

Total Path Count: 1

Operational Path Count: 1

/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2

Total Path Count: 1

Operational Path Count: 1

Replug the fibre cable

root@t2000:~# mpathadm list lu

/dev/rdsk/c0t600144F0ABA8000000004F311DEC000Ad0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DEB0009d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE90008d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE80007d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE60006d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE40005d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE30004d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DE10003d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DDF0002d0s2

Total Path Count: 2

Operational Path Count: 2

/dev/rdsk/c0t600144F0ABA8000000004F311DDD0001d0s2

Total Path Count: 2

Operational Path Count: 2

Friday, 27 January 2012

Building a SAN/NAS combo server...

Getting started

I've been mulling over how best to produce a NAS box for backing up the Mac's and xbox360 media, as well as how to best provide some cheap mass storage for the Sun box (only takes expensive SAS and SCA) and the VMWare box. Most NAS boxes offer iSCSI which is good, but only 4 bays, when I realistically needed something that resembled a big cheap box'o'disk with dual LAN ports for link aggregation and a fibre card chucked in.

I decided to go the roll my own route, first looking at the Chenbro ES34069 which offered a mini-itx form factor and 4 built in bays that assuming I could squeeze enough cards in would give me an eSATA port for expansion. Trouble being, after I totted up everything I needed it started to get beyond my cheapo budget for the box itself, without any disk.

Back to the drawing board and much scouring of eBay later I decided a bigger rack mount case (2-3u would do nicely and allow me to use whatever cheap parts I could find). Turns out Chenbro cases are really hard to find in the UK, although xcase hold a good stock.

I stumbled on a RM21406 plus Tyan S2882 mobo going dirt cheap on eBay that got me halfway there. The only disappointing feature of the bundle was that the mobo had 4 onboard SATA ports, while the case had 6 bays. So I'd need a RAID card. Dell CERC's which seem to be some kind of Intel/Adaptec/SiliconImage mashup offered 6 ports at a reasonable price, probably as per the reviews its not a great performer. At the price, and with 64mb cache it would be good enough.

Luckily I already had a Emulex E10000, a 20GB PATA drive, a bag of ECC DDR and a pair of Dual Core Opterons kicking around from an earlier upgrade to the Fujitsu RX220 which filled in the rest. Next stop cooling. A pair of 1u coolers were sourced and fitted after a light dose of thermal paste. Why 1u coolers in a 2u case? Well this board may get retired into a 1u case and replaced with something newer one day as it runs ESX5 which the RX220 doesn't due to a buggy ACPI implementation. After Intel coolers, this were a pig to fit, but certainly felt solid once they were on.

Take 1

With a single 1u PCI-X bracket I fitted the Dell CERC card into the slot, and ran cables from the mobo to the rear brackets to some eSATA ports. The whole lot was a tight fit and I did wonder if the strain on the SATA cables and sockets would cause damage. After a quick test firing up I was really disappointed to find that this Chenbro had no activity lights on the SATA side. A little googling on the CERC suggests that the 8 pin connector on the card holds discrete LED activity lines. So I endeavoured to knock up a cable with a soldering iron, a bag of LED's and a glue gun.

Bending the pins around the board seemed to be a good way to avoid permanently modifying (or damaging) the backplane. A liberal squidge of glue front and back helps to keep the LED's stable as long as they aren't knocked. I picked up a Dell cable assembly to get the 8pin plug to fit the card and discarded the rest (the plug is available but I didn't have the correct crimp tool to make a cable up). I ran this to a small rectangle of veroboard with stake headers on it for each of the eight connections.

I soldered a 2 pin fly lead from each LED which connected to the veroboard. A short flylead was also connected from the veroboard to ground on the backplane. Sadly when I assembled this all back together bays 0,1 and 3 all lit up, either my soldering was naff or theres some signalling on the 8 wire connection.

I abandoned this, and refocussed on what I wanted the box to do.

Take 2

The original plan was to have a vertical riser in the case so I could use full height cards. These turned out to be rare as rocking horse poo and had to be ordered from the US. I also stumbled across the BMC/IPMI card for the board, which sadly uses up one of those 3 ports on the riser. It also fouls the top slot where I'd planned to fit the eSATA brackets. Since I only had two SATA drives in the bays I discarded those until I one day decide to do a Take 3. As you can see, the CERC card at the bottom takes the PCI-X 133 slot, the Fibre card takes a 100 slot (that way my I/O on the card never exceeds what the disk subsystem can manage) and the top slot contains the IPMI card.

I also took the opportunity to fit right angled SATA cables which stopped everything being so tight.

Take 3

I dumped the ARC2-656-C9 riser as it proved to be unreliable, once again proving that if you buy shite you buy twice. A proper Tyan M2044 is in place now, with a HP SAS card.

So a stack of SATA cables later, and about £100 over budget its all done - all to fix this.

I've also finally got 6x WD3000JS hard disks in, not great disks but big enough and good enough to do whats needed.

..and now its ready to go, I've gone for OpenIndiana, a Solaris fork which is ideally suited to serving up ZFS filesystems over iSCSI or Fibre. Its clocking in 95MiB/s over iSCSI/Gigabit ethernet and ~200MiB/s over 2GB fibre, not bad for a thrown together SAN box.