TedPage: Development in LXD

Most of my development is done in LXD containers. I love this for a few reasons. It takes all of my development dependencies and makes it so that they’re not installed on my host system, reducing the attack surface there. It means that I can do development on any Linux that I want (or several). But it also means that I can migrate my development environment from my laptop to my desktop depending on whether I need more CPU or whether I want it to be closer to where I’m working (usually when travelling).

When I’m traveling I use my Pagekite SSH setup on a Raspberry Pi as the SSH gateway. So when I’m at home I want to connect to the desktop directly, but when away connect through the gateway. To handle this I set up SSH to connect into the container no matter where it is. For each container I have an entry in my .ssh/config like this:

Host container-name
	User user
	IdentityFile ~/.ssh/id_container-name
	CheckHostIP no
	ProxyCommand ~/.ssh/if-home.sh desktop-local desktop.pagekite.me %h

You’ll notice that I use a different SSH key for each container. They’re easy to generate and it is worth not reusing them, this is a good practice. Then for the ProxyCommand I have a shell script that’ll setup a connection depending on where the container is running, and what network my laptop is on.

#!/bin/bash

set -e

CONTAINER_NAME=$3

SSH_HOME_HOST=$1
SSH_OUT_HOST=$2

ROUTER_IP=$( ip route get to 8.8.8.8 | sed -n -e "s/.*via \(.*\) dev.*/\\1/p" )
ROUTER_MAC=$( arp -n ${ROUTER_IP} | tail -1 | awk '{print $3}' )

HOME_ROUTER_MAC="▒▒:▒▒:▒▒:▒▒:▒▒:▒▒"

IP_COMMAND="lxc list --format csv --columns 6 ^${CONTAINER_NAME}\$ | head --lines=1 | cut -d ' ' -f 1"
NC_COMMAND="nc -6 -q0"

IP=$( bash -c "${IP_COMMAND}" )
if [ "${IP}" != "" ] ; then
	# Local
	exec ${NC_COMMAND} ${IP} 22
fi

SSH_HOST=${SSH_OUT_HOST}
if [ "${HOME_ROUTER_MAC}" == "${ROUTER_MAC}" ] ; then
	SSH_HOST=${SSH_HOME_HOST}
fi

IP=$( echo ${IP_COMMAND} | ssh ${SSH_HOST} bash -l -s )

exec ssh ${SSH_HOST} -- bash -l -c "\"${NC_COMMAND} ${IP} 22\"" 

What this script does it that it first tries to see if the container is running locally by trying to find its IP:

IP_COMMAND="lxc list --format csv --columns 6 ^${CONTAINER_NAME}\$ | head --lines=1 | cut -d ' ' -f 1"

If it can find that IP, then it just sets up nc command to connect to the SSH port on that IP directly. If not, we need to see if we’re on my home network or out and about. To do that I check to see if the MAC address of the default router matches the one on my home network. This is a good way to check because it doesn’t require sending additional packets onto the network or otherwise connecting to other services. To get the router’s IP we look at which router is used to get to an address on the Internet:

ROUTER_IP=$( ip route get to 8.8.8.8 | sed -n -e "s/.*via \(.*\) dev.*/\\1/p" )

We can then find out the MAC address for that router using the ARP table:

ROUTER_MAC=$( arp -n ${ROUTER_IP} | tail -1 | awk '{print $3}' )

If that MAC address matches a predefined value (redacted in this post) I know that it’s my home router, else I’m on the Internet somewhere. Depending on which case I know whether I need to go through the proxy or whether I can connect directly. Once we can connect to the desktop machine, we can then look for the IP address of the container off of there using the same IP command running on the desktop. Lastly, we setup an nc to connect to the SSH daemon using the desktop as a proxy.

exec ssh ${SSH_HOST} -- bash -l -c "\"${NC_COMMAND} ${IP} 22\"" 

What all this means so that I just type ssh contianer-name anywhere and it just works. I can move my containers wherever, my laptop wherever, and connect to my development containers as needed.