Extract keys using JQ

This is probably something very specific to my use case, but I was working with some JSON that looked something like this:

  "something": {
    "Identifying Key": [
        "foo": "a.b.c",
        "bar": "First Three"
        "foo": "a.b.d",
        "bar": "Second Three"
    "Another Key": [
        "foo": "z.b.c",
        "bar": "First Three, Take Two"
        "foo": "z.b.d",
        "bar": "Second Three, Take Two"

From this, I wanted to extract everything before the first period in every instance of the foo key. Like I said, very specific use case.

To do it, I built the following jq expression.

cat data.json | jq -r '.something | map(.[].foo | split(".")[0]) | unique | join("\n")'

Breaking that down:

# Select the top level namespace
cat data.json | jq -r '.something'

# Map over everything as key => value
cat data.json | jq -r '.something | map(.)'

# From each of those elements, step in to each array
cat data.json | jq -r '.something | map(.[])'

# And extract the foo key
cat data.json | jq -r '.something | map(.[].foo)'

# Inside the map, use "split" to split the string on a "."
cat data.json | jq -r '.something | map(.[].foo | split("."))'

# And only select the first element in the returned arrays
cat data.json | jq -r '.something | map(.[].foo | split(".")[0])'

# We only want to know about unique keys
cat data.json | jq -r '.something | map(.[].foo | split(".")[0]) | unique'

# And we want it as a string, not an array
cat data.json | jq -r '.something | map(.[].foo | split(".")[0]) | unique | join("\n")'

And with that, I have a unique list of values extracted from a specific key in an object. As I mentioned, it’s very specific to my use case but it might help someone in the future

fatal error: xxhash.h: No such file or directory

When trying to compile a project, I started to receive the error fatal error: xxhash.h: No such file or directory. I tried to install liblz4-dev, but that didn’t seem to fix the error (looking at the repo, it looks as though xxhash was removed in revision 123).

Eventually, I decided to just download the relevant files and place them in /usr/include which fixed the issue,

cd /usr/include
sudo wget http://lz4.googlecode.com/svn-history/r122/trunk/programs/xxhash.c
sudo wget http://lz4.googlecode.com/svn-history/r122/trunk/programs/xxhash.h

Then, when I tried to compile the project it worked fine

golang: How to `go get` private repos

Recently, I’ve been working on some golang projects at DataSift that depend on some private modules. This means that when I run go get to fetch the module, Github asks me for my username and password. Unfortunately, as I use 2FA I need to generate a new, temporary password for applications that want my password. As go get doesn’t cache credentials, I’d have to do this every time I needed a private repo. Instead, I wanted to force go get to use my SSH key.

Fortunately, as go get uses git, you can change your .gitconfig to force it to use SSH.

$ git config --global url."git@github.com:".insteadOf "https://github.com/"
$ cat ~/.gitconfig
[url "git@github.com:"]
    insteadOf = https://github.com/

Here, we say use git@github.com any time you’d use https://github.com. This works for everything, not just go get. It just has the nice side effect of using your SSH key any time you run go get too.