PCI-X risers, OpenIndiana, and grumpy Emulex cards

I've recently been having problems with Emulex LP10000DC cards on my Tyan S2882 based SAN/NAS box that I documented here where its been core-dumping under light load over fibre connections. The first dual port card in the box was faulty, so I swapped it out for a known good card which still showed the same symptoms.

For the last few months I've been getting by just using my SAN/NAS box via iSCSI only (more on this later as it proved exceptionally easy to use with AIX) but frustration with not being able to use the fibre connection with the T2000 and VMWare systems meant it was a constant nag to sort.

Well, I found the issue, and its a geekily-interesting one. You can view the S2882 board here if you have the same issue on the same board or if you merely want to follow this closely. The S2882 has four PCI-X slots, 2 belong to PCIX-A, 2 belong to PCIX-B. The PCI-X riser I'm using in the 2U case takes one slot from PCIX-B slot 1 @ 133Mhz, and the other 2 slots come from PCIX-A slot 3 and 4 which can be configured as 100MHz (default) or as 66MHz via jumper J43.

The Emulex card is on PCIX-A, the Dell CERC RAID card is using the 133Mhz slot from PCIX-B - I figured that the NIC's + Fibre traffic being limited to under the bus bandwidth of the disk controller would prevent bottlenecks (especially as I plan to move to RAIDZ(2) at some point soon which would require some of that extra bandwidth). I originally thought to swap the cards around to rule that out, as the ARC2-656-C9 riser has a too long flexible cable that I have had to fold flat to make it fit (the correct riser in this range is the ARC2-656-C5 - the C is the length of cable between the separate PCIX bus slots).

It turns out that as part of the PCI-X specification, using the Emulex LP10000-DC which is a 133Mhz capable device in a 100MHz slot is out of spec - I'm guessing it tries to run the card at 200MHz! The fix for stable operation was to drop the PCIX-B bus down to 66Mhz. Sadly this also affects the onboard Broadcom NIC's (and will also affect the onboard SCSI if you have it).

Update: Nov 2012 (original article Jun 2012)

Turns out the riser flex cable had gone bad and continued to deteriorate further. Reducing the speed helped bring everything in spec, and it worked for a month or two until I started getting core dumps again. I've dug in my pockets again for this project and got a proper Tyan M2044 riser which has fixed the core dumping and has so far been rock solid. The PCI-X bus speed remains a point to note, as its a specification thing.

I've also ditched the CERC as quite frankly as soon as I hit 6 disks the ~90MiB/s the card manages to chew through started to be the limiting factor. I now have a HP 8port SAS card reflashed with stock 3080XT firmware and I've clocked it giving the Sun T2000 over 200MiB/s under iostat, and giving 95MiB/s over iSCSI to the iMac.