Recovering WD Passport drives

Well, this turned out to be an adventure. You know the story, a relative dropped their external USB drive and its all gone pop, along with pictures of their children from the day they were born.

These new WD (Western Digital) drives dispense of the old SATA drive with a USB converter setup in favour of simply replacing the SATA connector with a USB one. This poses a couple of problems when dealing with recovery, the first is that SMART status doesn't seem to be implemented, the second is that on some disks they choose to use encryption as well. The final problem is a standard one - the performance is just abysmal, USB just isn't designed for bulk transfer in the same way say SATA is, or even how Firewire was - yes the numbers are good, but real world use case performance doesn't match up.

In the case of this drive there must have been some minor damage to the cable and the USB socket, as well as what seems to be damage to the platter in the first 32MB. As ever, dd to the rescue:-

dd if=/dev/disk4 of=/Volumes/Recovery/deadhd.raw bs=4096 conv=noerror,sync

Most people that have been around any *nix will know the dd basics, but it offers a whole boatload of other features besides squirting blocks from here to there via the converter option. In this case skip any errors but keep writes in sync - i.e. insert nulls for any data that can't be read.

Hows the recovery going? Well 20 hours later its done 300GB out of its 1TB size, two areas of platter damage detected. Using strings on the raw image I can see email addresses and other data around photos - so fingers crossed I'll have at least some recovered data to hand back.

So it turns out I'm lucky with this one, scanning on the internet shows others have had to resort to soldering a SATA connection onto the board or replacing the USB controller board with a compatible SATA one and transplanting some of the chips.

PCI-X risers, OpenIndiana, and grumpy Emulex cards

I've recently been having problems with Emulex LP10000DC cards on my Tyan S2882 based SAN/NAS box that I documented here where its been core-dumping under light load over fibre connections. The first dual port card in the box was faulty, so I swapped it out for a known good card which still showed the same symptoms.

For the last few months I've been getting by just using my SAN/NAS box via iSCSI only (more on this later as it proved exceptionally easy to use with AIX) but frustration with not being able to use the fibre connection with the T2000 and VMWare systems meant it was a constant nag to sort.

Well, I found the issue, and its a geekily-interesting one. You can view the S2882 board here if you have the same issue on the same board or if you merely want to follow this closely. The S2882 has four PCI-X slots, 2 belong to PCIX-A, 2 belong to PCIX-B. The PCI-X riser I'm using in the 2U case takes one slot from PCIX-B slot 1 @ 133Mhz, and the other 2 slots come from PCIX-A slot 3 and 4 which can be configured as 100MHz (default) or as 66MHz via jumper J43.

The Emulex card is on PCIX-A, the Dell CERC RAID card is using the 133Mhz slot from PCIX-B - I figured that the NIC's + Fibre traffic being limited to under the bus bandwidth of the disk controller would prevent bottlenecks (especially as I plan to move to RAIDZ(2) at some point soon which would require some of that extra bandwidth). I originally thought to swap the cards around to rule that out, as the ARC2-656-C9 riser has a too long flexible cable that I have had to fold flat to make it fit (the correct riser in this range is the ARC2-656-C5 - the C is the length of cable between the separate PCIX bus slots).

It turns out that as part of the PCI-X specification, using the Emulex LP10000-DC which is a 133Mhz capable device in a 100MHz slot is out of spec - I'm guessing it tries to run the card at 200MHz! The fix for stable operation was to drop the PCIX-B bus down to 66Mhz. Sadly this also affects the onboard Broadcom NIC's (and will also affect the onboard SCSI if you have it).

Update: Nov 2012 (original article Jun 2012)

Turns out the riser flex cable had gone bad and continued to deteriorate further. Reducing the speed helped bring everything in spec, and it worked for a month or two until I started getting core dumps again. I've dug in my pockets again for this project and got a proper Tyan M2044 riser which has fixed the core dumping and has so far been rock solid. The PCI-X bus speed remains a point to note, as its a specification thing.

I've also ditched the CERC as quite frankly as soon as I hit 6 disks the ~90MiB/s the card manages to chew through started to be the limiting factor. I now have a HP 8port SAS card reflashed with stock 3080XT firmware and I've clocked it giving the Sun T2000 over 200MiB/s under iostat, and giving 95MiB/s over iSCSI to the iMac.

Disabling hardware from OBP on SPARC

If you are struggling to get a SPARC server to boot it may be possible to disable any suspect hardware using ASR.

asr disable <name of device>

To get a list of device names, use asr-list-keys

To view already disabled hardware use .asr

Building a SAN/NAS combo server...

Getting started

I've been mulling over how best to produce a NAS box for backing up the Mac's and xbox360 media, as well as how to best provide some cheap mass storage for the Sun box (only takes expensive SAS and SCA) and the VMWare box. Most NAS boxes offer iSCSI which is good, but only 4 bays, when I realistically needed something that resembled a big cheap box'o'disk with dual LAN ports for link aggregation and a fibre card chucked in.

I decided to go the roll my own route, first looking at the Chenbro ES34069 which offered a mini-itx form factor and 4 built in bays that assuming I could squeeze enough cards in would give me an eSATA port for expansion. Trouble being, after I totted up everything I needed it started to get beyond my cheapo budget for the box itself, without any disk.

Back to the drawing board and much scouring of eBay later I decided a bigger rack mount case (2-3u would do nicely and allow me to use whatever cheap parts I could find). Turns out Chenbro cases are really hard to find in the UK, although xcase hold a good stock.

I stumbled on a RM21406 plus Tyan S2882 mobo going dirt cheap on eBay that got me halfway there. The only disappointing feature of the bundle was that the mobo had 4 onboard SATA ports, while the case had 6 bays. So I'd need a RAID card. Dell CERC's which seem to be some kind of Intel/Adaptec/SiliconImage mashup offered 6 ports at a reasonable price, probably as per the reviews its not a great performer. At the price, and with 64mb cache it would be good enough.

Luckily I already had a Emulex E10000, a 20GB PATA drive, a bag of ECC DDR and a pair of Dual Core Opterons kicking around from an earlier upgrade to the Fujitsu RX220 which filled in the rest. Next stop cooling. A pair of 1u coolers were sourced and fitted after a light dose of thermal paste. Why 1u coolers in a 2u case? Well this board may get retired into a 1u case and replaced with something newer one day as it runs ESX5 which the RX220 doesn't due to a buggy ACPI implementation. After Intel coolers, this were a pig to fit, but certainly felt solid once they were on.

Take 1

With a single 1u PCI-X bracket I fitted the Dell CERC card into the slot, and ran cables from the mobo to the rear brackets to some eSATA ports. The whole lot was a tight fit and I did wonder if the strain on the SATA cables and sockets would cause damage. After a quick test firing up I was really disappointed to find that this Chenbro had no activity lights on the SATA side. A little googling on the CERC suggests that the 8 pin connector on the card holds discrete LED activity lines. So I endeavoured to knock up a cable with a soldering iron, a bag of LED's and a glue gun.

Bending the pins around the board seemed to be a good way to avoid permanently modifying (or damaging) the backplane. A liberal squidge of glue front and back helps to keep the LED's stable as long as they aren't knocked. I picked up a Dell cable assembly to get the 8pin plug to fit the card and discarded the rest (the plug is available but I didn't have the correct crimp tool to make a cable up). I ran this to a small rectangle of veroboard with stake headers on it for each of the eight connections. 

I soldered a 2 pin fly lead from each LED which connected to the veroboard. A short flylead was also connected from the veroboard to ground on the backplane. Sadly when I assembled this all back together bays 0,1 and 3 all lit up, either my soldering was naff or theres some signalling on the 8 wire connection.

I abandoned this, and refocussed on what I wanted the box to do.

Take 2

The original plan was to have a vertical riser in the case so I could use full height cards. These turned out to be rare as rocking horse poo and had to be ordered from the US. I also stumbled across the BMC/IPMI card for the board, which sadly uses up one of those 3 ports on the riser. It also fouls the top slot where I'd planned to fit the eSATA brackets. Since I only had two SATA drives in the bays I discarded those until I one day decide to do a Take 3. As you can see, the CERC card at the bottom takes the PCI-X 133 slot, the Fibre card takes a 100 slot (that way my I/O on the card never exceeds what the disk subsystem can manage) and the top slot contains the IPMI card.

I also took the opportunity to fit right angled SATA cables which stopped everything being so tight.

Take 3

I dumped the ARC2-656-C9 riser as it proved to be unreliable, once again proving that if you buy shite you buy twice. A proper Tyan M2044 is in place now, with a HP SAS card.

So a stack of SATA cables later, and about £100 over budget its all done - all to fix this.

I've also finally got 6x WD3000JS hard disks in, not great disks but big enough and good enough to do whats needed.

..and now its ready to go, I've gone for OpenIndiana, a Solaris fork which is ideally suited to serving up ZFS filesystems over iSCSI or Fibre. Its clocking in 95MiB/s over iSCSI/Gigabit ethernet and ~200MiB/s over 2GB fibre, not bad for a thrown together SAN box.