Pogoplug SAN using AoE

The Pogoplug E02 is a Linux-capable embedded computer with gigabit Ethernet and USB connectivity. It can be found quite cheap (US$35 or so) on the secondary market. As it was originally sold as a storage appliance for home users, it ships with somewhat inflexible factory firmware. However, it is reasonably easy to replace the bootloader with one which can boot an arbitrary Linux distribution.

Once I got my Pogoplug running Debian, it was surprisingly easy to set it up as network storage using ATA-over-Ethernet. Details after the jump.

This article assumes you’re reasonably comfortable with the Linux command line, are familiar with basic networking and UNIX concepts, and that you have a desktop Linux machine and root access to same.

Note: Throughout the rest of this page, when I say “Pogoplug”, I’m referring specifically to the pink Pogoplug model E02. There are other models which are pink and look almost identical (B01, B02, B03, B04, P21 and maybe others I don’t know about), but that use radically different hardware under the hood. Always check the model number!

Also note: These are instructions for re-purposing your Pogoplug hardware. If you follow them, your Pogoplug will no longer work with the pogoplug.com network service. It’s an either/or thing.

Command-Line Access

Out of the box, the Pogoplug device is set up to get an IP address via DHCP, and to run an SSH server. Connect your Pogoplug to a network with a DHCP server, make sure no USB devices are plugged in, apply power and let it boot.

Determine what IP address your DHCP server has given your Pogoplug. The MAC address is on a printed label on the bottom of the Pogoplug if you need it. (Mine is 00:25:31:xx:yy:zz, a “Cloud Engines, Inc.” OUI.)

Make sure you can ping the Pogoplug.

SSH in. The default username is root, password ceadmin. (If you have registered your Pogoplug device, you may need to use the password you chose during registration instead.)

This should get you to a shell prompt on the Pogoplug.

Updating the Pogoplug Bootloader

Warning: The process in this section carries the risk of bricking your Pogoplug. Even if you follow all the steps perfectly, you might end up with a non-functional Pogoplug if the power fails or the network hiccups at the wrong time. If this happens, you will need a JTAG programmer, a 3.3V serial terminal and access to an internal header on the Pogoplug board to fix it. Don’t hack hardware you can’t afford to destroy, and don’t blame me if you break your toys.

The factory bootloader on the Pogoplug is locked down and will only run software signed by the vendor. To their credit, that vendor does not do anything to obstruct the owner of the device replacing the bootloader with one that’s a bit more flexible: uBoot.

If your Pogoplug has Internet connectivity, by far the easiest way to replace the bootloader is to use Jeff Doozan’s install script. From the Pogoplug command line:

cd /tmp
wget http://projects.doozan.com/uboot/install_uboot_mtd0.sh
chmod a+x install_uboot_mtd0.sh
./install_uboot_mtd0.sh

Note that at the time of this writing, Jeff’s installer gives you the updated uBoot with davygravy’s patch for the L2 Cache bug. (That’s a good thing. You want that.)

If you’d prefer to do things manually (or just want to better understand what’s going on behind the scenes), there’s a post on the doozan.com forum with detailed instructions. (Said forum is also an excellent place to search if you have problems.)

Once the bootloader is updated, but before you reboot, consider setting up netconsole (see next section).

Netconsole

The uBoot bootloader offers a network console capability, as an more convenient alternative to connecting a 3.3V serial terminal to the debug header on the Pogoplug mainboard. To take advantage of this capability, use the fw_setenv utility from the Pogoplug shell prompt. (The fw_setenv and fw_printenv utilities are installed by the install_uboot_mtd0.sh script. They function much like the familiar setenv and printenv command, except that they manipulate a special firmware environment from which uBoot reads its configuration.)

Setting up netconsole is totally optional, but it can save you having to open up your Pogoplug and fiddle with serial terminals if something goes wrong.

To enable netconsole, issue the following commands at the Pogoplug shell prompt (from the doozan.com forum):

fw_setenv serverip 192.168.1.2
fw_setenv ipaddr 192.168.1.100
fw_setenv if_netconsole 'ping $serverip'
fw_setenv start_netconsole 'setenv ncip $serverip; setenv bootdelay 10; setenv stdin nc; setenv stdout nc; setenv stderr nc; version;'
fw_setenv preboot 'run if_netconsole start_netconsole'

Replace the server ip value (192.168.1.2 in the example) with the numeric IP (not the DNS name) of the computer you wish to use as a network console for the Pogoplug. Replace the ipaddr value (192.168.1.100 in the example) with a numeric, static IP the Pogoplug can use during the boot process. (Once the OS is booted, it can use a different IP, including one assigned via DHCP.) Do not set ipaddr to an IP address which is already in use, or which is within the DHCP pool!

This configuration tells uBoot that at the beginning of the boot process (“preboot”) it should set the Pogoplug IP to ipaddr and try to ping serverip. If the ping succeeds, it runs some additional commands (“start_netconsole”). These commands map uBoot’s stdio to the network, display the uBoot version number and force a ten-second delay before the boot process begins to allow for user input.

The netconsole uses single-cast UDP datagrams (for both input and output). The uBoot output is sent to serverip, port 6666. Input is accepted on the Pogoplug’s port 6666.

To view the output, you can use netcat. (Note for Ubuntu users: The default seems to be BSD-style netcat; my examples — and those elsewhere on the ‘net — assume a traditional netcat. To deal with this, sudo apt-get install netcat-traditional then use the nc.traditional command everywhere the examples use nc.)

On the server (identified by serverip), run:

nc -l -u -p 6666

then reboot the Pogoplug. You should see the uBoot output. If you want both input and output, you will need to run the above command in the background, then a second nc command to send your keyboard input to the Pogoplug:

nc -l -u -p 6666 &
nc -u 192.168.1.100 6666

(Remember to kill the background nc when you are done.)

Note that netconsole has no security or authentication of any sort. Carefully consider the security implications of leaving it enabled long-term, especially in a context where an attacker might be able to force the Pogoplug to reboot, and/or be able to inject/intercept UDP packets.

Installing a Temporary USB Root FS

Jeff has a tempting script for installing Debian, with instructions on his website. I ran into some trouble with this, as the factory-default NAND image I was using had a 2.x kernel, and the debootstrap command used in the script really wants something more recent.

So, my (roundabout) solution was:

1. Update the bootloader as discussed above.
2. Use a canned root filesystem image to create a bootable USB flash drive with a Linux system based on a recent kernel.
3. Boot into the system from (2).
4. From that system, replace the contents of the NAND flash with a modern rescue system (also based on a recent Linux kernel).
5. Boot into the system from (4).
6. Use Jeff’s instructions to perform a debootstrap-based install to a flash drive.

This section contains detailed instructions for step (2). Details for subsequent steps are in later sections.

First, get a canned filesystem image. I used the one davygravy prepared for the Zyxel NSA320 (per his forum post). (In this particular case, even though it’s named after different hardware, it’ll work on the Pogoplug E02.) On your desktop Linux box (not the Pogoplug), do the following:

mkdir pogo
cd pogo
wget 'http://dl.dropbox.com/u/1015928/Kirkwood/rootfs_images/NSA320quickstart.tar.gz'
tar xzf NSA320quickstart.tar.gz
cd NSA320quickstart
sudo losetup --show -f kirkwood-debian_rootfs.img
mkdir mnt
sudo mount /dev/loopX mnt # replace X with the number output by losetup above
cd mnt
sudo tar czfp ../../debian-rootfs.tar.gz .
cd ..
sudo umount mnt
sudo losetup -d /dev/loopX # replace X with the number again
cd ..
rm -rf NSA320quickstart
sudo chown 'id -u'.'id -g' debian-rootfs.tar.gz

As a (hopefully simpler) alternative to the above, I have prepared a Debian Wheezy image, which you can download directly (84MB bzip-ed tar archive).

Once you have your root filesystem image (either the NSA320quickstart one or my Debian Wheezy), find a USB flash drive of at least 1GB capacity containing nothing you care about. Connect it to your desktop Linux box, and partition it. (I used fdisk, but any reasonable partitioning tool will work.) The goal is:

• Primary partition 1: All but about 200MB of space, type 83 (Linux), active.
• Primary partition 2: All remaining space, type 82 (Linux swap).
• No other partitions.

Be very sure you have the right device! You will (almost-)irrevocably lose data if you get this wrong.

Note that the partition numbering shown above is not optional (unless you edit various Debian config files, in which case GLHF).

Make a filesystem on the first partition using: mkfs.ext2 -j /dev/your/device/here

Be even more sure you have the right device; you will (utterly and completely) irrevocably lose data if you get this part wrong.

Mount the new filesystem, and (as root) extract the debian-rootfs.tar.gz (or pogo-wheezy-20130305.tar.bz2) tarball at the top level. The output of ls should look something like:

bin/   dev/  home/  lost+found/  mnt/  proc/  sbin/     srv/  tmp/  var/
boot/  etc/  lib/   media/       opt/  root/  selinux/  sys/  usr/

Move your working directory out of the USB flash filesystem, unmount it, and remove the drive.

Power off your Pogoplug, connect the USB flash drive (to any port; I used the one on the front) and boot it up. Give it a few minutes. If you can’t ping it, double-check your DHCP leases; mine got a different IP this time around.

Since you’re booting a different system, the SSH certs will have changed. Use ssh-keygen -R ip-address (where ip-address is the numeric IP of your Pogoplug) to convince your SSH client that it’s not teh hax0r trying to pull a MITM attack.

SSH in as user root, password root. You should see a Debian system running off the USB drive with a 3.X kernel. To check:

cat /etc/issue
uname -a

Rescue System in NAND Flash

Now that you have the Pogoplug running from a (temporary) Linux of recent vintage from an external USB storage device, we can update the contents of NAND flash to something more recent as well.

Serious warning: This is another “point of no return”. Updating the bootloader was the most risky part (in terms of bricking your Pogoplug), but if you update the OS in NAND flash, you will no longer be able to use the Pogoplug in the way the vendor intended.

The rescue system I chose is davygravy’s “Doozan Forum Kirkwood/*Plug* Recovery System V2.8.2 : Oct 29 2012”. Please see the original forum post for full details and instructions.

The first thing to observe is that it requires the “Newer U-Boot w/ L2 cache bug-fix”. If you followed the instructions above regarding involving install_uboot_mtd0.sh then you already have this.

Also, this was written in early March of 2013. If you’re reading this a significantly later, then it is probably worth your while to check if there is a more recent version available.

On your Pogoplug (booted from external USB storage, not the internal NAND flash!), download https://dl.dropbox.com/u/1015928/Kirkwood/rescue/RescueV2.8.2-10292012.tar.gz .

Then follow the manual installation procedure outlined in the forum post:

tar zxvf RescueV2.8.1-05282012.tar.gz
flash_eraseall /dev/mtd1
nandwrite /dev/mtd1 uImage-mtd1.img
flash_eraseall /dev/mtd2
ubiformat /dev/mtd2 -s 512 -f rootfs-mtd2.img -y

fw_setenv set_bootargs_rescue 'setenv bootargs console=$console ubi.mtd=2 root=ubi0:rootfs ro rootfstype=ubifs $mtdparts'
fw_setenv bootcmd_rescue 'run set_bootargs_rescue; nand read.e 0x800000 0x100000 0x400000; bootm 0x800000'
fw_setenv bootcmd_pogo 'run bootcmd_rescue'
fw_setenv rescue_installed 1

Once you have done this, sync and halt your Pogoplug. Disconnect the power and all external USB devices. Power back up (optionally, watching the netconsole if you set that up). The Pogoplug should get an address via DHCP, and you should be able to SSH in as user root, password root. (Once again, you may have to use ssh-keygen -R as the host key will have changed.)

You should see a rescue system running Linux 3.x. Note that the root filesystem is mounted read-only, with the handful of things that need to change in a RAM-based tmpfs. If you need to modify something, remount the root fs read-write, make the change, then (very important!) put it back to read-only. For example, to change your root password (which is an incredibly good idea):

remount -o remount,rw /
passwd
remount -o remount,ro /

Installing Debian

All the above was just to get to the point where we have the Pogoplug running a Linux-3.x-based system from NAND flash. Now that we do, we can install Debian to an external USB device.

Start with the Pogoplug running the internal rescue system, with no USB devices connected. Once it is up and running and you have logged in, connect the flash device from which you want your Pogoplug to boot. (I used a USB flash drive. It is OK to re-use the same device you used for the temporary root FS, above.)

Follow Jeff Doozan’s instructions for installing Debian Wheezy using debootstrap:

cd /tmp
wget http://projects.doozan.com/debian/kirkwood.debian-wheezy.sh
chmod +x kirkwood.debian-wheezy.sh
export PATH=$PATH:/usr/sbin:/sbin
./kirkwood.debian-wheezy.sh

This will install Debian Wheezy to your USB device, using the latest packages from the repositories. It takes a while. (Exactly how long depends on your ‘net connection and the speed of your USB storage device, but 20-30 minutes would not be unusual.)

Once it has finished, you can reboot the Pogoplug. It should come back up running from your USB device. SSH in, user root, password root, ssh-keygen -R probably required. All your favorite, familiar Debian command should work as expected (or if not, be available after an appropriate use of apt-get install).

Setting up ATA-over-Ethernet

Concepts

Huge volumes of text have been written about ATA-over-Ethernet (aka AoE or aoe), including a Wikipedia article, a comparison of AoE with other storage-related network protocols such as iSCSI, and a Debian How-To. These are useful and interesting, but it isn’t necessary to fully understand them for the purposes of this post. There are a few core concepts that are essential, though:

• In AoE, the system connected to the physical storage device is called the target.
• The system accessing the target over the network using AoE is called the initiator.
• AoE uses raw Ethernet frames (type 0x88A2), not IP.  You can’t route AoE packets. The target and initiator have to be on the same network segment (to be pedantic: the same broadcast domain) for AoE to work. (In other words, AoE can work across bridges, hubs and switches but not across routers.) This is in contrast to iSCSI, which is routable.
• AoE has no built-in security (other than MAC filtering — which is no security at all — and its innate non-routability). You absolutely must put your AoE traffic on its own isolated network (or VLAN) if you’re using it to store data you care about.
• Multiple initiators can use a single AoE storage device at the same time, but they must provide their own mechanism for handling concurrency issues. (Mounting a filesystem like ext4 from two computers at the same time will make a mess. You can do that using special shared disk filesystems, though.)

Software Installation

We are going to set up the Pogoplug as an AoE target. Installing the software to do this is very easy. We just need to make sure the aoe module is loaded, and to install a couple of packages:

sudo modprobe aoe
sudo sh -c 'echo aoe >>/etc/modules'
sudo apt-get install vblade vblade-persist

The aoe module provides kernel support for the protocol. The vblade package is the user-space daemon which manages AoE traffic for a particular device. The vblade-persist package is a command-line front end which make it more convenient to configure and manage vblade. (You don’t have to install vblade-persist, but it makes things much easier.)

If your Pogoplug doesn’t have Internet connectivity (and you don’t want to provide it temporarily to facilitate the install), you’ll need to download the packages out-of-band and install them manually. That is, as they say, beyond the scope of this document.

Shared Storage Device

Having a SAN is really no fun unless you have at least one storage device to which you want to provide access over the network. You can share any of the following:

• a whole-disk block device (e.g. /dev/sdb)
• a partition block device (e.g. /dev/sdb1)
• a logical volume (e.g. /dev/myvolgrp/somevol)
• a plain file in the target’s local filesystem

For the purposes of this example, I’m going to assume that you want to share a USB hard disk that appears as /dev/sdb (and that you want to share the whole disk and whatever partition(s) it contains).

Connect your storage device, and check the dmesg output to make sure it is recognized and that the device name is what you think.

Target Software Setup

The easiest way to set up the vblade software is using the vblade-persist utility. To share a device:

sudo vblade-persist setup 0 0 eth0 /dev/sdb
sudo vblade-persist auto all
sudo vblade-persist start all

The “setup” command sets up the configuration for a new shared device (but does not start the vblade process for that device, nor make it so that device starts on boot).

The “0 0” are the shelf and slot of the device. The shelf, slot pair is an identifier for this particular shared device, and must be unique on the local network (not just on this particular AoE target).

The “eth0” is the name of the local interface over which you wish to share the AoE device, and over which all AoE traffic for that device will pass.

Finally, “/dev/sdb” is the name of the device (or partition, logical device or file) you wish to share.

The “vblade-persist auto all” command sets things up so all shares configured through vblade-persist will be started when the system is next booted.

“vblade-persist start all” starts the share right now.

You can query the status of vblade-persist managed shared with:

sudo vblade-persist ls

Initiator Software Setup

On your initiator system, we must load the same aoe module as is used on the target, and install the aoetools package:

sudo sh -c 'echo aoe >>/etc/modules'
sudo apt-get install aoetools

Then, discover the AoE targets visible to the initiator:

sudo aoe-interfaces eth0
sudo aoe-discover
sudo aoe-stat

The “aoe-interfaces” command (which is strictly optional) restricts AoE traffic to one or more specified network interfaces. The default is to use all available interfaces.

“aoe-discover” finds targets on the local network and creates corresponding device nodes under /dev/etherd/.

The “aoe-stat” command shows a list of the available targets. If everything went well, you should see output that looks something like:

e0.0       320.069GB eth0 up

The numbers in “e0.0” corrspond to the shelf and slot numbers chosen on the target. (If you do not see any output from aoe-stat, troubleshoot before going on.)

You should now have a block device special file in /dev/etherd with a name corresponding to the first part of the aoe-stat output:

brw-rw---- 1 root disk 152, 0 Mar  5 19:35 /dev/etherd/e0.0

If the shared device is a whole-disk device that has partitions, you should also see block device special files for each partition:

brw-rw---- 1 root disk 152, 0 Mar  8 12:16 e0.0
brw-rw---- 1 root disk 152, 1 Mar  8 12:16 e0.0p1

(Note that the partitions do not show up in aoe-stat output.)

Typical dmesg output following discovery of an AoE device with a partition looks like:

 aoe: AoE v47q initialised.
 aoe: 00253100425c e0.0 v4014 has 625142448 sectors
  etherd/e0.0: p1

You can now use the block devices as you would any other: Make filesystems on them, mount them, use them as part of logical volumes or with LUKS.

Refinements

Traffic Segregation

Since AoE has no built-in security mechanism, it’s a very good idea to keep AoE traffic where it can be seen only by the initiator(s) and the target(s). Obviously, one way to accomplish this is by using a dedicated physical network (including a dedicated NIC on each machine involved). This has the benefits of being easy to understand, and may have bandwidth benefits as well. However, it may be impractical in terms of wiring if the targets and initiators aren’t physically close together.

An alternative is to put the AoE traffic on its own VLAN. This requires a VLAN-capable switching infrastructure, but if you already have that, you don’t need any new NICs or cabling. (Remember that a Linux box can participate in an arbitrary number of VLANs — as well as sending and receiving untagged packets — using a single physical NIC.)

I have not tried this, but it should be possible to pass AoE traffic through a bridge-style VPN tunnel. (By “bridge-style” I mean a tunnel that passes L2 traffic through the tunnel, where the endpoints are in the same broadcast domain. It should use a “tap” device, not a “tun” device.)

Device Naming

The examples above glossed over a rather important point: the simple block device names for USB storage devices (like /dev/sdb) depend on the order in which corresponding hardware was plugged into the bus (or became ready, or was discovered). It’s actually a really bad idea to use the simple block device name for an AoE share that gets set up automatically at boot time. (If someone plugs in another random device, you may end up sharing — and overwriting — that instead of the one you want.)

Fortunately, the udev device manager gives us some better alterantives. Under /dev/disk/ are a number of subdirectories containing symlinks to the simple block device names. These directories are:

• by-id/ — symlink names correspond to the (hopefully, unique) ID reported by the device hardware
• by-path/ — symlink names correspond to how (by what bus, and where on that bus) the device is connected
• by-uuid/ — symlink names are the unique ID assigned to the device

You can use one of these symlinks instead of the name of the simple block device with vblade-persist to ensure that you are always sharing the device you mean to.

For most uses, you’d probably want to use by-uuid/ so you always refer to a specific device no matter how or where it’s connected. Alternatively, you could use by-path/ if you wanted to always share whatever storage device happened to be plugged in to a particular USB port. Example command:

sudo vblade-persist setup 0 0 eth0 /dev/disk/by-uuid/0c01c370-0329-45d8-97cb-ebff27514e36

Performance Tuning

When you’re using the Pogoplug E02 as your AoE target, the chief performance bottleneck is very likely to be the USB 2.0 interface, and there’s not a whole lot you can do about that without going to different hardware entirely.

However, there are still some things you can do that will help:

• ensure that the Ethernet path between target and initiator is gigabit, full-duplex
• avoid saturating the AoE link with other unrelated traffic
• remember that not every USB HDD enclosure is created equal (nor is every HDD)
• enable jumbo frames if possible

The last item deserves some additional discussion. Some gigabit Ethernet devices support frame sizes in excess of the 802.3-mandated 1500-byte payload. (Note that in order for this to work, every device along the path must support jumbo frames. This includes the NICs at each end as well as any intervening switches.)

It’s a bit of a hassle, as jumbo frame support is quite dependent on hardware and driver specifics. If you try to use a packet size bigger than your link will handle, you’ll very likely have intermittent connectivity problems. (I’d suggest using scp to copy a big file as a test. If it stalls after the first few seconds, lower your MTU on both sides.) Also remember that even if your switch hardware supports jumbo frames, you may have to set a configuration option to enable this support.

And, after all that, the performance gains are not that great (because: USB 2.0). But keep things in perspective: It’s still an excellent SAN given that the price is US$35 plus some junk you probably already have in your stash.

MAC Filtering

It’s possible to set up an AoE target to accept traffic only from one or more specific initiator MAC addresses. It’s probably a mistake to regard this as a security feature. But it doesn’t hurt anything, and it may help save you or others from honest human error.

Say for the sake of example that I have a single initiator and its MAC (on the interface it uses for AoE) is 8c:89:a5:11:22:33. Then, I could add MAC filtering to my existing shelf 0 slot 0 target by running the following command (on the target system):

sudo vblade-persist mac 0 0 add 8c:89:a5:11:22:33

You can see the allowed initiator MAC addresses for shelf 0 slot 0 using this command:

sudo vblade-persist mac 0 0 ls

Update 01 June 2014 DGH: Fixed missing newline in Debian install commands.