Building a SAN/NAS combo server...

Getting started

I've been mulling over how best to produce a NAS box for backing up the Mac's and xbox360 media, as well as how to best provide some cheap mass storage for the Sun box (only takes expensive SAS and SCA) and the VMWare box. Most NAS boxes offer iSCSI which is good, but only 4 bays, when I realistically needed something that resembled a big cheap box'o'disk with dual LAN ports for link aggregation and a fibre card chucked in.

I decided to go the roll my own route, first looking at the Chenbro ES34069 which offered a mini-itx form factor and 4 built in bays that assuming I could squeeze enough cards in would give me an eSATA port for expansion. Trouble being, after I totted up everything I needed it started to get beyond my cheapo budget for the box itself, without any disk.

Back to the drawing board and much scouring of eBay later I decided a bigger rack mount case (2-3u would do nicely and allow me to use whatever cheap parts I could find). Turns out Chenbro cases are really hard to find in the UK, although xcase hold a good stock.

I stumbled on a RM21406 plus Tyan S2882 mobo going dirt cheap on eBay that got me halfway there. The only disappointing feature of the bundle was that the mobo had 4 onboard SATA ports, while the case had 6 bays. So I'd need a RAID card. Dell CERC's which seem to be some kind of Intel/Adaptec/SiliconImage mashup offered 6 ports at a reasonable price, probably as per the reviews its not a great performer. At the price, and with 64mb cache it would be good enough.

Luckily I already had a Emulex E10000, a 20GB PATA drive, a bag of ECC DDR and a pair of Dual Core Opterons kicking around from an earlier upgrade to the Fujitsu RX220 which filled in the rest. Next stop cooling. A pair of 1u coolers were sourced and fitted after a light dose of thermal paste. Why 1u coolers in a 2u case? Well this board may get retired into a 1u case and replaced with something newer one day as it runs ESX5 which the RX220 doesn't due to a buggy ACPI implementation. After Intel coolers, this were a pig to fit, but certainly felt solid once they were on.

Take 1

With a single 1u PCI-X bracket I fitted the Dell CERC card into the slot, and ran cables from the mobo to the rear brackets to some eSATA ports. The whole lot was a tight fit and I did wonder if the strain on the SATA cables and sockets would cause damage. After a quick test firing up I was really disappointed to find that this Chenbro had no activity lights on the SATA side. A little googling on the CERC suggests that the 8 pin connector on the card holds discrete LED activity lines. So I endeavoured to knock up a cable with a soldering iron, a bag of LED's and a glue gun.

Bending the pins around the board seemed to be a good way to avoid permanently modifying (or damaging) the backplane. A liberal squidge of glue front and back helps to keep the LED's stable as long as they aren't knocked. I picked up a Dell cable assembly to get the 8pin plug to fit the card and discarded the rest (the plug is available but I didn't have the correct crimp tool to make a cable up). I ran this to a small rectangle of veroboard with stake headers on it for each of the eight connections. 

I soldered a 2 pin fly lead from each LED which connected to the veroboard. A short flylead was also connected from the veroboard to ground on the backplane. Sadly when I assembled this all back together bays 0,1 and 3 all lit up, either my soldering was naff or theres some signalling on the 8 wire connection.

I abandoned this, and refocussed on what I wanted the box to do.

Take 2

The original plan was to have a vertical riser in the case so I could use full height cards. These turned out to be rare as rocking horse poo and had to be ordered from the US. I also stumbled across the BMC/IPMI card for the board, which sadly uses up one of those 3 ports on the riser. It also fouls the top slot where I'd planned to fit the eSATA brackets. Since I only had two SATA drives in the bays I discarded those until I one day decide to do a Take 3. As you can see, the CERC card at the bottom takes the PCI-X 133 slot, the Fibre card takes a 100 slot (that way my I/O on the card never exceeds what the disk subsystem can manage) and the top slot contains the IPMI card.

I also took the opportunity to fit right angled SATA cables which stopped everything being so tight.

Take 3

I dumped the ARC2-656-C9 riser as it proved to be unreliable, once again proving that if you buy shite you buy twice. A proper Tyan M2044 is in place now, with a HP SAS card.

So a stack of SATA cables later, and about £100 over budget its all done - all to fix this.

I've also finally got 6x WD3000JS hard disks in, not great disks but big enough and good enough to do whats needed.

..and now its ready to go, I've gone for OpenIndiana, a Solaris fork which is ideally suited to serving up ZFS filesystems over iSCSI or Fibre. Its clocking in 95MiB/s over iSCSI/Gigabit ethernet and ~200MiB/s over 2GB fibre, not bad for a thrown together SAN box.