Recovering WD Passport drives

Well, this turned out to be an adventure. You know the story, a relative dropped their external USB drive and its all gone pop, along with pictures of their children from the day they were born.

These new WD (Western Digital) drives dispense of the old SATA drive with a USB converter setup in favour of simply replacing the SATA connector with a USB one. This poses a couple of problems when dealing with recovery, the first is that SMART status doesn't seem to be implemented, the second is that on some disks they choose to use encryption as well. The final problem is a standard one - the performance is just abysmal, USB just isn't designed for bulk transfer in the same way say SATA is, or even how Firewire was - yes the numbers are good, but real world use case performance doesn't match up.

In the case of this drive there must have been some minor damage to the cable and the USB socket, as well as what seems to be damage to the platter in the first 32MB. As ever, dd to the rescue:-

dd if=/dev/disk4 of=/Volumes/Recovery/deadhd.raw bs=4096 conv=noerror,sync

Most people that have been around any *nix will know the dd basics, but it offers a whole boatload of other features besides squirting blocks from here to there via the converter option. In this case skip any errors but keep writes in sync - i.e. insert nulls for any data that can't be read.

Hows the recovery going? Well 20 hours later its done 300GB out of its 1TB size, two areas of platter damage detected. Using strings on the raw image I can see email addresses and other data around photos - so fingers crossed I'll have at least some recovered data to hand back.

So it turns out I'm lucky with this one, scanning on the internet shows others have had to resort to soldering a SATA connection onto the board or replacing the USB controller board with a compatible SATA one and transplanting some of the chips.

Disk Mirroring in Linux - a call to arms...

Seriously...

When is mirrored root disks on Linux not going to be a total disaster.

Solaris 8/9 had a shitty way with SVM until ZFS happened in 10, AIX has it pretty good since 4.x, geriatric HP-UX can even do it...

If you are insisting md is the way to go then you need to be able to clone your disk partitions and metadb equivalent easily across to diskN and encapsulate (to use the nasty Veritas term). Its Solaris 8 era half-arsed stupidity with an extra helping of utter stupidity. HP and IBM offered the LVM stuff, drag them back in to finish the job so you can mirror and use a volume manager.

If btrfs is the solution, then please, make it at least as functional as ZFS circa 2005. My experiences here suggest this is a highly immature option that given its gestation period already will never hit usable status.

Finally... distro builders, spend a little more time ensuring mirrored root installs aren't a 'sacrifice a goat' affair as to whether the bastard thing will boot or not. Ubuntu gets this mostly right once you get the shitty partitioner under control, RHEL/Centos gets it wrong, although at least the partitioner is less retarded.

A return to adventures in OpenStack-land

So... OpenStack....

I'd dabbled with it some time ago, using both https://www.rdoproject.org/install/quickstart/ and Solaris 11, and i'd pretty much written it off as meh I can do that with zones and zfs *much* easier in my sleep. As ever in life, better engineered quality loses out to popular and cheap every time and its undeniable fact the future is linux and OpenStack for cloud anywhere that you aren't using AWS (wish they'd release that!).

I briefly investigated alternatives to OpenStack that were more AWS like such as Eucalyptus (which is even less simple to setup than OpenStack!) but frankly, the designed by committee mess has won already.

Here's a recount of my findings and personal opinions.

• Ubuntu 16_04,MaaS and JuJu - this appears to be a total no brainer. Ubuntu, MaaS (Metal As A Service) which is awesome, and Juju as an orchestration tool. Coupled with lxc/lxd and zfs this just screamed pick me to a Solaris refuge as all the goodness I love is here. Shame it just doesn't work. Until they fix it the simple advice is avoid like the plague - when they do, this is the only sane choice.
• RedHats RDO initiative - https://www.rdoproject.org/install/quickstart/ - this is the earliest effort I know of to get an easily configured and working OpenStack install, its not perfect, its not fast, its not really great, but it does work. If you need a play around this is the only usable option on linux.

I *really* had to stop myself throwing Solaris on the box (Oracle's licensing and patch stance since the demise of Sun being the number 1 reason not to) and using its OpenStack install that just works and gives native ZFS and Zones.

Shame when linux software runs better elsewhere!

Managing VMWare ESX without VMWare's Windows Client Part 1

For a number of years since switching to a Mac the only need for Windows has been for the VMWare client for my home lab. This has worked reasonably well through the XP years, and survived a migration to Windows 7 even, but recently whatever Windows 10 ‘preload’ does has broken my Windows 7 VM’s while leaving them unable to upgrade to Windows 10 and finally the moment has arrived where I need to break free however painful.

VMWare Fusion Pro looks like a good way forward, but given that my next mobile device to replace my aging Macbook Pro is going to be an iPad and keyboard that doesn’t work as a way forward either. I needed something lightweight and command line.

Step forward the libvirt project and its tools - especially virsh. I do remember its inception many years ago as a way to unify management of the then iffy KVM, the likely at the time winner XEN, plus the other smaller players like lxc.

Installing virt-manager

# yum install virt-manager

Connecting to the ESX server… if like myself you don’t have a full SSL certificate authority setup you’ll need the ‘?no_verify=1’. Omit it in a production environment.

$ virsh -c esx://192.168.1.73/?no_verify=1
Enter username for 192.168.1.73 [root]: 
Enter root's password for 192.168.1.73: 
Welcome to virsh, the virtualisation interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh #

Connecting to the ESX server… if like myself you don’t have a full SSL certificate authority setup you’ll need the ‘?no_verify=1’. Omit it in a production environment. First thing you’re likely to need is to see what VM’s you have. In virsh parlance - ‘domains’.

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 1     switch1                        running
 2     hopper                         running
 3     dns1                           running
 -     dns2                           shut off
 -     ldap1                          shut off
- - - - - - - - - - - -8< - - - - - - - - - - - - - -
 -     solweb2                        shut off
 -     solwlserv1                     shut off
 -     solwlserv2                     shut off
 -     switch2                        shut off

Next thing is you’ll need to start some of them...

virsh # start linc7web1
Domain linc7web1 started

...and of course, stop some of them.

virsh # shutdown linc7lb1
error: Failed to shutdown domain linc7lb1
error: internal error: HTTP response code 500 for call to 'ShutdownGuest'. Fault: ServerFaultCode - Cannot complete operation because VMware Tools is not running in this virtual machine.

If you see the above on a VM, install the vmware tools (e.g. yum install -y open-vm-tools).

virsh # shutdown linc7lb1
Domain linc7lb1 is being shutdown

Finding JVM arguments easily from the shell

You know the score, you're asked 'oh just check what the Xmx is on that JVM'... easy huh...

$ pgrep -fl java
17506 /bin/bash /Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/nbexec --userdir /Users/someuser/Library/Application Support/NetBeans/8.0.2 --cachedir /Users/someuser/Library/Caches/NetBeans/8.0.2 --jdkhome -J-Dcom.apple.mrj.application.apple.menu.about.name=NetBeans -J-Xdock:name=NetBeans -J-Xdock:icon=/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/nb/netbeans.icns --branding nb --clusters /Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/nb:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/ergonomics:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/ide:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/extide:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/java:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/apisupport:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/webcommon:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/websvccommon:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/enterprise:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/mobility:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/profiler:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/python:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/php:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/identity:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/harness:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/cnd:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/dlight:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/groovy:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/extra:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/javacard:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/javafx: -J-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade -J-Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense -L-XX:MaxPermSize=384m -J-Xmx1024m -J-client -J-Xss2m -J-Xms32m -J-XX:PermSize=32m -J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true -J-Dsun.java2d.noddraw=true -J-Dsun.java2d.dpiaware=true -J-Dsun.zip.disableMemoryMapping=true SHELL=/bin/bash
17758 /Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/bin/java -Djdk.home=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home -classpath /Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/boot.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/org-openide-modules.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/org-openide-util-lookup.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/org-openide-util.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/boot_ja.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/boot_pt_BR.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/boot_ru.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/boot_zh_CN.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-modules_ja.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-modules_pt_BR.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-modules_ru.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-modules_zh_CN.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util-lookup_ja.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util-lookup_pt_BR.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util-lookup_ru.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util-lookup_zh_CN.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util_ja.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util_pt_BR.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util_ru.jar:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform/lib/locale/org-openide-util_zh_CN.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home/lib/tools.jar -Dnetbeans.default_userdir_root=/Users/someuser/Library/Application Support/NetBeans -Dnetbeans.dirs=/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/nb:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/ergonomics:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/ide:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/extide:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/java:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/apisupport:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/webcommon:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/websvccommon:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/enterprise:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/mobility:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/profiler:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/python:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/php:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/identity:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/harness:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/cnd:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/dlight:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/groovy:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/extra:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/javacard:/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/javafx: -Dnetbeans.home=/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/platform -Dcom.apple.mrj.application.apple.menu.about.name=NetBeans -Xdock:name=NetBeans -Xdock:icon=/Applications/NetBeans/NetBeans 8.0.2.app/Contents/Resources/NetBeans/nb/netbeans.icns -Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade -Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense -Xmx1024m -client -Xss2m -Xms32m -XX:PermSize=32m -Dapple.laf.useScreenMenuBar=true -Dapple.awt.graphics.UseQuartz=true -Dsun.java2d.noddraw=true -Dsun.java2d.dpiaware=true -Dsun.zip.disableMemoryMapping=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/someuser/Library/Application Support/NetBeans/8.0.2/var/log/heapdump.hprof org.netbeans.Main --cachedir /Users/someuser/Library/Caches/NetBeans/8.0.2 --userdir /Users/someuser/Library/Application Support/NetBeans/8.0.2 --branding nb

Did you see it?

It then sinks in, its all on one line! You can't grep Xmx (well you can I guess if you have grep with colour highlighting - believe me, not everyone gets that luxury in the workplace!). Its still gonna be fun to find in that lot!

Help is at hand...

$ pgrep -fl java | tr -s " " "\n" | grep "Xmx"
-J-Xmx1024m
-Xmx1024m

Or perhaps you want all the -arguments...?<

$ pgrep -fl java | tr -s " " "\n" | grep "^-"

--userdir

--cachedir

--jdkhome

-J-Dcom.apple.mrj.application.apple.menu.about.name=NetBeans

-J-Xdock:name=NetBeans

-J-Xdock:icon=/Applications/NetBeans/NetBeans

--branding

--clusters

-J-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade

-J-Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense

-L-XX:MaxPermSize=384m

-J-Xmx1024m

-J-client

-J-Xss2m

-J-Xms32m

-J-XX:PermSize=32m

-J-Dapple.laf.useScreenMenuBar=true

-J-Dapple.awt.graphics.UseQuartz=true

-J-Dsun.java2d.noddraw=true

-J-Dsun.java2d.dpiaware=true

-J-Dsun.zip.disableMemoryMapping=true

-Djdk.home=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home

-classpath

-Dnetbeans.default_userdir_root=/Users/someuser/Library/Application

-Dnetbeans.dirs=/Applications/NetBeans/NetBeans

-Dnetbeans.home=/Applications/NetBeans/NetBeans

-Dcom.apple.mrj.application.apple.menu.about.name=NetBeans

-Xdock:name=NetBeans

-Xdock:icon=/Applications/NetBeans/NetBeans

-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade

-Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense

-Xmx1024m

-client

-Xss2m

-Xms32m

-XX:PermSize=32m

-Dapple.laf.useScreenMenuBar=true

-Dapple.awt.graphics.UseQuartz=true

-Dsun.java2d.noddraw=true

-Dsun.java2d.dpiaware=true

-Dsun.zip.disableMemoryMapping=true

-XX:+HeapDumpOnOutOfMemoryError

-XX:HeapDumpPath=/Users/someuser/Library/Application

--cachedir

--userdir

--branding

Using vi as a hex editor

I recently had an issue where a license key was unable to be transferred, and attempts to fix using vi/tr/sed (don’t laugh!) didn’t get the file into the correct shape. As it was clearly using a binary forma, I really needed a proper hex editor.

Luckily I was able to list the files with od -ax on two machines, so I knew what I wanted!

It turns out you can use vi as a hex editor - its a commonly known thing out on the internet as ever (my source was http://www.kevssite.com/2009/04/21/using-vi-as-a-hex-editor/).

:%!xxd turns your file into a hex editor style representation

:%!xxd -r turns your edited file back into a binary form which you can then save.

Also handy for editing firmware, binaries, libraries - although be careful, it does seem to add a newline that you’ll need to remove using sed/tr/etc.

Encypting and decrypting files with OpenSSL

While this is posted elsewhere in abundance here is the full roundtrip process. This was required due to my workplace deciding that all communications where to be encrypted when sent over email or http so that they could detect information leaks (if its encrypted don't ask me how they know!). To avoid falling foul I've been encrypting diags using the below:-

Create a file with your secret contents

$ cat > confidential.txt
Here is my secret!
<ctrl-d>

Encrypt it, and supply an encryption password

$ openssl enc -aes-256-cbc -salt -in ./confidential.txt -out ./confidential.txt.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:

You just can't read the file now... its safe to send over an unencrypted connection.

$ cat ./confidential.txt.enc
Salted__??(?*???q^s????_ۂꠤ?~|~#?`+@

To decrypt, you'll need to agree on the encryption password with your third party, or somehow have sent it securely already.

$ openssl enc -d -aes-256-cbc -in ./confidential.txt.enc
enter aes-256-cbc decryption password:
Here is my secret!