Development in LXD

Most of my development is done in LXD containers. I love this for a few reasons. It takes all of my development dependencies and makes it so that they're not installed on my host system, reducing the attack surface there. It means that I can do development on any Linux that I want (or several). But it also means that I can migrate my development environment from my laptop to my desktop depending on whether I need more CPU or whether I want it to be closer to where I'm working (usually when travelling).

When I'm traveling I use my Pagekite SSH setup on a Raspberry Pi as the SSH gateway. So when I'm at home I want to connect to the desktop directly, but when away connect through the gateway. To handle this I set up SSH to connect into the container no matter where it is. For each container I have an entry in my .ssh/config like this:

Host container-name
    User user
    IdentityFile ~/.ssh/id_container-name
    CheckHostIP no
    ProxyCommand ~/.ssh/if-home.sh desktop-local desktop.pagekite.me %h

You'll notice that I use a different SSH key for each container. They're easy to generate and it is worth not reusing them, this is a good practice. Then for the ProxyCommand I have a shell script that'll setup a connection depending on where the container is running, and what network my laptop is on.

#!/bin/bash

set -e

CONTAINER_NAME=$3

SSH_HOME_HOST=$1
SSH_OUT_HOST=$2

ROUTER_IP=$( ip route get to 8.8.8.8 | sed -n -e "s/.*via \(.*\) dev.*/\\1/p" )
ROUTER_MAC=$( arp -n ${ROUTER_IP} | tail -1 | awk '{print $3}' )

HOME_ROUTER_MAC="▒▒:▒▒:▒▒:▒▒:▒▒:▒▒"

IP_COMMAND="lxc list --format csv --columns 6 ^${CONTAINER_NAME}\$ | head --lines=1 | cut -d ' ' -f 1"
NC_COMMAND="nc -6 -q0"

IP=$( bash -c "${IP_COMMAND}" )
if [ "${IP}" != "" ] ; then
    # Local
    exec ${NC_COMMAND} ${IP} 22
fi

SSH_HOST=${SSH_OUT_HOST}
if [ "${HOME_ROUTER_MAC}" == "${ROUTER_MAC}" ] ; then
    SSH_HOST=${SSH_HOME_HOST}
fi

IP=$( echo ${IP_COMMAND} | ssh ${SSH_HOST} bash -l -s )

exec ssh ${SSH_HOST} -- bash -l -c "\"${NC_COMMAND} ${IP} 22\"" 

What this script does it that it first tries to see if the container is running locally by trying to find its IP:

IP_COMMAND="lxc list --format csv --columns 6 ^${CONTAINER_NAME}\$ | head --lines=1 | cut -d ' ' -f 1"

If it can find that IP, then it just sets up nc command to connect to the SSH port on that IP directly. If not, we need to see if we're on my home network or out and about. To do that I check to see if the MAC address of the default router matches the one on my home network. This is a good way to check because it doesn't require sending additional packets onto the network or otherwise connecting to other services. To get the router's IP we look at which router is used to get to an address on the Internet:

ROUTER_IP=$( ip route get to 8.8.8.8 | sed -n -e "s/.*via \(.*\) dev.*/\\1/p" )

We can then find out the MAC address for that router using the ARP table:

ROUTER_MAC=$( arp -n ${ROUTER_IP} | tail -1 | awk '{print $3}' )

If that MAC address matches a predefined value (redacted in this post) I know that it's my home router, else I'm on the Internet somewhere. Depending on which case I know whether I need to go through the proxy or whether I can connect directly. Once we can connect to the desktop machine, we can then look for the IP address of the container off of there using the same IP command running on the desktop. Lastly, we setup an nc to connect to the SSH daemon using the desktop as a proxy.

exec ssh ${SSH_HOST} -- bash -l -c "\"${NC_COMMAND} ${IP} 22\"" 

What all this means so that I just type ssh contianer-name anywhere and it just works. I can move my containers wherever, my laptop wherever, and connect to my development containers as needed.


posted Jun 14, 2019 | permanent link

OAuth2 in the Shell

For some scripts at work I need to log into our Gitlab instance and use its API. To do that you need an OAuth2 token, and I wasn't able to find any examples that I could crib from, so I'm posting what I made. Hopefully this'll help you do the same for your scripts. I should mention that I'm using this with Gitlab as per their instructions, it might be slightly different for other OAuth implementors, but should be roughly the same.

First let's just put the whole script out there before we break it down:

OAUTH_CLIENT="123456789abcdef"
OAUTH_SECRET="123456789abcdef"
PORT=5000

xdg-open "https://gitlab.example.com/oauth/authorize?client_id=${OAUTH_CLIENT}&redirect_uri=http://localhost:${PORT}/&response_type=code" &> /dev/null

OAUTH_CODE=$( echo -e "HTTP/1.1 200 OK\n\n<HTML><body><blink>Thank you</blink></body></HTML>" | nc -l -p ${PORT} | sed -n "s/^GET.*code=\([a-fA-F0-9]*\).*/\1/p" ) 

if [ "${OAUTH_CODE}" == "" ] ; then
    echo "Unable to get OAUTH code"
    exit 1
fi

OAUTH_TOKEN=$(curl -X POST -F "client_id=${OAUTH_CLIENT}" -F "client_secret=${OAUTH_SECRET}" -F "code=${OAUTH_CODE}" -F "grant_type=authorization_code" -F "redirect_uri=http://localhost:5000/" https://gitlab.example.com/oauth/token | jq --raw-output ."access_token" )

if [ "${OAUTH_TOKEN}" == "" ] ; then
    echo "Unable to get OAUTH token"
    exit 1
fi

When you want to use an OAuth2 client with Gitlab the first thing you need to do register as a client, getting the OAUTH_CLIENT and OAUTH_SECRET strings. You'll need to use the first one in the call to open the user's browser.

xdg-open "https://gitlab.example.com/oauth/authorize?client_id=${OAUTH_CLIENT}&redirect_uri=http://localhost:${PORT}/&response_type=code" &> /dev/null

The thing to notice in this call is that we're using localhost for the redirect URL. That means that after (assuming they do) they authenticate the script it will redirect the browser back this host with the code needed to get the token. We then need a webserver running on this machine to get that code.

OAUTH_CODE=$( echo -e "HTTP/1.1 200 OK\n\n<HTML><body><blink>Thank you</blink></body></HTML>" | nc -l -p ${PORT} | sed -n "s/^GET.*code=\([a-fA-F0-9]*\).*/\1/p" ) 

For our webserver we're using the trusty netcat to open a port and give us the data sent there. We go ahead and give the browser a nice webpage to say thanks (you know it's a serious Thank You when you use the <blink> tag). The output you get from netcat is something like this:

GET /?code=123456789abcdef HTTP/1.1
Host: localhost:5000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

So we use sed to pull out the code field by replacing the line with just the code and printing it. This gives us the code that we can then turn into a token.

OAUTH_TOKEN=$(curl -X POST -F "client_id=${OAUTH_CLIENT}" -F "client_secret=${OAUTH_SECRET}" -F "code=${OAUTH_CODE}" -F "grant_type=authorization_code" -F "redirect_uri=http://localhost:5000/" https://gitlab.example.com/oauth/token | jq --raw-output ."access_token" )

We set up a rather long curl call with several parameters that results in a JSON object that looks something like:

{
 "access_token": "de6780bc506a0446309bd9362820ba8aed28aa506c71eedbe1c5c4f9dd350e54",
 "token_type": "bearer",
 "expires_in": 7200,
 "refresh_token": "8257e65c97202ed1726cf9571600918f3bffb2544b26e00a61df9897668c33a1"
}

Which we then use jq to select the access_token and we're good to go. Now we can use that token to access the Gitlab API as we need it.


posted May 8, 2019 | permanent link

Knot Boards

Each year Cub Scouts has a birthday party for Scouting in February, which is called the Blue & Gold banquet. We have a tradition that at the banquet were we thank all of our volunteers who help make the Cub Scout Pack run. For the Den Leaders, who are so critical to the program, I like to do something special that helps them to run a better program for the scouts. For 2018 (notice I'm a little behind) I decided to make all of the Den Leaders for our Pack knot boards.

SVG file for the knotboard design

When I was a Scout I remember my mom making knot boards. Back then we had a piece of paper with the various knots that was varnished onto a piece of plywood, which had a rope attached to it. High technology for the time, but today I'm a member of TheLab.ms makerspace and have access to a laser cutter. While these knot boards are the same in spirit, we can do some very cool things with big toys.

Laser actively cutting boards with a cool sparkle

First step is to pull out Inkscape and design the graphics. I grabbed a rope border from Open Clipart and grabbed some knot graphics from a Scouting PDF (which I can't find a link to). I put those together to create the basic design along with labes for the knots. I also added a place for each Scout to sign their name as a Thank you to the Den Leader. I then make some small circles for the laser cutter to cut out holes for the ropes. I made a long oblong region on the right so the board would have a handle and a post to tie the hitches around. Then lastly I added the outline to cut out the board.

To get the design into the laser cutter I exported it from Inkscape in two graphics. I exported the cut lines as a DXF and I exported the etching as a 300 DPI PNG. The cut lines were simpler and the laser cutter software was able to handle those and create simple controls for the cutter. The knots on the other hand were more complex vector objects and the laser cutter software couldn't handle them. Inkscape could, so I had it do the rendering to a bitmap. The laser cutter can then setup scans that use the bitmap data which worked very well.

For the boards I used ¼th" Lauan Plywood which I was able to get in 2'x2' sheets at Lowe's. Those sheets have a nice grain on both sides. I also liked being able to get sheets that were exactly the size I needed to fit into the laser cutter. Saved me a step. I'm certain the knot boards would be great in many other woods and other materials.


Cut knot boards sitting the bed of the laser cutter

After cutting out the knot boards I needed short lengths of rope to be able to insert into the holes. I couldn't find anywhere that would sell me short pieces of rope. I felt like I needed a Monty Python sketch. To make short lengths of the paracord I looped it in a circle with the circumference as the length I needed. Then I took a blowtorch and cut the circle. This also sealed the ends of the paracord.

Final knot boards with ropes in the holes


posted Feb 2, 2019 | permanent link

Texas Linux Fest 2019 in Dallas

Texas Linux Fest 2019 will be in Dallas! It will be held May 31st and June 1st at the Irving Convention Center and the Call For Papers is open right now.

A few years ago I started to suggest to TXLF staff that coming to Dallas was a good idea. I wanted there to be more tech conferences in Dallas, and I love the community organized nature of TXLF and similarly SCALE. Plus, it was Texas Linux Fest, it can't always be in Austin! This year I was able to convince them to take the risk and try a year in Dallas. It is a huge risk, as it is likely that many sponsors and regular attendees might not be interested in traveling up I-35 to attend. Being in Dallas also opens up huge opportunity to reach new audiences and new sponsors. Now to prove that.

I'm excited that we were able to secure the conference center in Irving. It is on a light rail line that goes to both airports, and surrounded by restaurants if you don't want to take the train. We'll be securing blocks of rooms at local hotels if you want to attend from afar. It should be an easy adventure and put you in a location that you can enjoy the best of Dallas.

Now is your time to help out by submitting a talk to make the conference great. If you know of a sponsor we'd love to hear from you as well. And we'll be having registrations open as we announce the speakers.

Hope to see y'all in Dallas!


posted Jan 10, 2019 | permanent link

Letting Information Go

There's a lot of information stored all over the Internet about me, about you, about everyone. At best, most of it can just go away because it's useless, at worst it is potentially harmful. A humorous take on this by Molly Lewis:

The place that this is the most obvious is social media. I really liked this post on old tweets by Vicki Lai which talks about the why and how of deleting Tweets. It applies to all social media. But this all got me thinking about my blog.

Blog posts tend to be more thought out (or at least I try) and seem to me to be part of the larger web. So just deleting them after a matter of time doesn't feel the same as tweets. If someone was writing about the Unity HUD I would hope they'd reference my HUD 2.0 post, as I love the direction it was going. I have other posts that are... less significant. The ones that are the most interesting are the ones that are linked to by other people, so what I'm going to do is stop linking to old blog posts. That way posts that aren't linked to by other people will stop being indexed by search engines and effectively disappear from the Internet. I have no idea if this will actually work.

The policy that I settled on was to have the latest five posts on my blog page, and then having the archives point to posts of the last two years. This means I need to write five posts every two years (easy right!) to keep it consistent. Turned out implementing it in Jekyll was a little tricky, but this post on Jekyll date filtering helped me put it together.

I think that my attitudes to data are generational difference. For my generation the idea that we could have hard drives big enough to keep historical data is exciting. Talking to younger people I think they understand it is a liability. Perhaps fixing my blog is just me trying to be young.


posted Oct 1, 2018 | permanent link

All the older posts...