michaelheap.com

Make an S3 bucket public

2022-07-08T16:01:00Z

I recently needed to make all objects in a bucket public, but not allow people to list the contents of the bucket. Finding the correct policy took me a little while, so I'm reproducing here for posterity:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
        }
    ]
}

jamf stuck on “Locating hardware information (macOS 11.6.5)”

2022-04-10T13:53:44Z

We use jamf at work, and I periodically run jamf recon and jamf policy to ensure that things are all up to date.

I noticed that the process was consistently hanging with the message Stuck on “Locating hardware information (macOS 11.6.5)”. From my googling it looked as though this was software update related, so I uninstalled jamf, applied the update and then reinstalled. Problem solved!

That is, until I tried to run jamf recon again and it hung at the same point. I opened up software update and it just hung, with the gear spinning forever.

More googling ensued, and I eventually found the magic command that I needed to get things working again:

bash
sudo launchctl kickstart -k system/com.apple.softwareupdated

Once softwareupdated had restarted, I could run jamf recon without any issues

Remove Docker containers by tag

2022-03-24T13:40:47Z

I always forget to pass the --rm flag when running command line tools with Docker. This means that I end up with a large history of exited containers that I no longer need.

There's plenty of information out there about how to clean up all stopped containers, or all unused images. I wanted to delete all stopped containers with a specific tag. Here's how to do it:

bash
docker rm $(docker container ls -a -q --filter ancestor=image-name:tag --filter status=exited)

This works thanks to the filtering capabilities when combined with the -q option to output only container IDs. These are then fed in to docker rm to remove all images with that tag that are in the exited status.

Update a single item in a list with `vuex`

2022-02-07T18:30:21Z

I recently needed to find an item in a list of objects by ID and update one of it’s properties whilst working with Vue 3 and VueX. It took me longer than I'd like to admit to figure out, so I'm writing it down here.

Due to how objects are passed by reference in JS, you can reference the item using Object.assign to update a single property.

Here’s the store, including a mutation to update the value:

js
import { createStore } from "vuex";
const store = createStore({
  state() {
    return {
      items: [
        { id: 1, value: "One", color: "red" },
        { id: 2, value: "Two", color: "yellow" },
        { id: 3, value: "Three", color: "skyblue" },
      ],
    };
  },
  mutations: {
    setColor(state, { id, color }) {
      console.log(`Setting block ${id} to ${color}`);
      const item = state.items.find((i) => i.id == id);
      if (item) {
        Object.assign(item, { ...item, color });
      }
    },
  },
});
export default store;

Here’s how I called the mutation from the component:

js
setColor(color) {
  store.commit("setColor", { id: this.id, color });
  return false;
}

The GitHub repo in this post renders a sample application where you can change the colour of some blocks.

Using `openvpn-client` with Docker

2022-01-28T15:06:56Z

I recently worked out the correct incantation to get a set of containers to connect to the internet via a VPN using docker-compose. I run it on a QNAP NAS, but it should work on any Linux-like system (I couldn’t get it working on MacOS).

The following docker-compose.yml will create two containers - one to run the VPN client and a second that runs curl. I use this to run curl ipv4.canhazip.com to check that the container has connectivity and the VPN IP address is returned.

yaml
---
version: "3"
services:
  vpn:
    container_name: vpn
    image: dperson/openvpn-client:latest
    cap_add:
      - net_admin
    restart: unless-stopped
    volumes:
      - /dev/net/tun:/dev/net/tun
      - ./vpn-config:/vpn # You'll need to provide this
    security_opt:
      - label:disable
    ports:
      - 8001:8001
      - 8002:8002
      - 8003:8003
    networks:
      - bridge_vpn
    entrypoint: ["/sbin/tini", "--", "/usr/bin/openvpn.sh", "-d"]
  curl:
    image: alpine/curl
    container_name: curl
    restart: unless-stopped
    network_mode: service:vpn
networks:
  bridge_vpn:

This compose file will expose ports 8001, 8002 and 8003 from any containers using network_mode: service:vpn and make them accessible via a bridge network. This is useful when running a service that connects to the internet using a VPN. (There are no exposed ports in this demo, but I wanted to make a note here as in my actual deployment some of the other services expose ports.)

The biggest change to this configuration compared to a lot of guides out there is the entrypoint line for the vpn service. The openvpn-client image supports a -d flag that adds some DNS related pre/post scripts. I found that these are required to make connectivity work via the VPN.

You may have noticed the vpn-config folder being mounted. This is where you’ll provide your VPN configuration and authentication files. I tested this with Private Internet Access. If you’re a PIA customer, you’ll need to create a file named vpn.auth with your PIA username on the first line, and password on the second line, plus a vpn.conf file with the following contents:

apache
client
dev tun
proto udp
remote sweden.privacy.network 1198
resolv-retry infinite
nobind
persist-key
# persist-tun # disable to completely reset vpn connection on failure
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass /vpn/vpn.auth # to be reachable inside the container
comp-lzo
verb 1
reneg-sec 0
crl-verify /vpn/crl.rsa.2048.pem # to be reachable inside the container
ca /vpn/ca.rsa.2048.crt # to be reachable inside the container
disable-occ
keepalive 10 30 # send a ping every 10 sec and reconnect after 30 sec of unsuccessfull pings
pull-filter ignore "auth-token" # fix PIA reconnection auth error that may occur every 8 hours

This configuration connects to Sweden, but you can switch to any endpoint that you like. You’ll also need to download crl.rsa.2048.pem and ca.rsa.2048.pem from the PIA site.

Once you have everything configured, it’s time to create the containers by running docker-compose up -d. Once that’s complete, you can test your connection by logging in to the vpn service and making a HTTP call:

bash
docker-compose exec vpn bash -c "curl ipv4.canhazip.com"

If the above command returns an IP address successfully, you can also test it using the curl container which is configured to use the VPN for all network connectivity:

bash
docker-compose run curl ipv4.canhazip.com

At this point, you have a docker-compose setup that connects all of the containers configured via an OpenVPN connection. Replace the curl service with any other service you may want to run behind a VPN and enjoy as your traffic is safe from snooping eyes.

The GitHub wiki is an anti-pattern

2022-01-23T18:08:26Z

The “should I use the wiki or a docs folder on GitHub?” discussion comes up every 6 months or so, and following Shawn Wang’s three strikes rule I thought it was about time I wrote something down about it.

The initial version of this post opened with “You can use the wiki or a docs folder for your GitHub project, both are valid choices” but as I wrote more, I realised that there is a single reason to use a wiki, and many more reasons not to use the wiki. So many in fact, that I consider using the wiki on GitHub is an anti-pattern.

Let’s start with the benefits of using a wiki:

You can get to the wiki contents in a single click from anywhere in the repo
There is no 2.

Really, the only benefit that I’ve been able to find to using the wiki is that it’s always there.

How about the reasons not to use the wiki?

Documentation is versioned alongside your code when using the /docs folder. If you need to use an old version, the docs are easy to find
The documentation isn’t available locally when someone clones your repo (you can clone the wiki separately, but this is a hidden feature)
Documentation edits get the same treatment as code. They get a full peer review through the pull request process
You can use GitHub Actions to lint your docs using tools such as Vale
People can work with tooling that they already know (e.g. vscode with spellcheck)
Wikis provide limited branding opportunities. They all look pretty much the same
The wiki doesn’t support image uploads, so you have to put images somewhere else anyway

Now that you’re sold on the idea of keeping docs alongside your code, how do you make it easy for people to view them?

Add your docs to your repository in the /docs folder. Do not use the gh-pages branch as this prevents docs being versioned alongside code
Set up a GitHub pages build to publish the docs
- If you’re just getting started, I recommend using the just-the-docs theme and letting GitHub build and publish your docs
- If you prefer to build your own workflow (e.g. using Hugo), you can use this GitHub Action to publish the docs
Add a single wiki page directing people to the hosted documentation

Using the /docs folder is the highest effort-to-reward ratio option whilst you’re building out a new product. At some point your docs will outgrow a single folder, and then all bets are off. You’ll want a separate repo with its own build process, pull request review guidelines and a whole host of other things. At that point people are already used to working with docs in a repository, and the migration from /docs to its own repo should be seamless for your contributors.

Whether you agree or disagree, I’d love to hear your thoughts on Twitter

VSCode only showing one open tab

2021-11-22T12:53:51Z

My VSCode instance has been refusing to show more than a single tab for a while, even with preview disabled and when double clicking on a filename to open.

Most blog posts talking about how to fix this have a similar resolution to this Stack Overflow thread. That is:

json
"workbench.editor.showTabs": false

This didn't work for me, nor did setting "workbench.editor.enablePreview": false

What worked for me is to lock a tab as opened by pressing Ctrl+K+Enter whilst the tab is focused. This updated the UI to show a tab bar and when I double clicked on another file it opened as expected.

Convert a PDF to a stacked jpg

2021-11-11T13:36:24Z

I built some UI mockups in Google Slides today and wanted to share the output as a single jpg file. The way to achieve this is to export the deck as PDF, use imagemagick to convert the PDF to images, then combine the images.

Once you have your PDF, save it as input.pdf in a new folder, then run the following commands:

bash
convert -density 150 input.pdf -quality 90 output.jpg
convert -append output-* final.jpg

You'll now have a file named final.jpg in your folder that contains all of the pages from the PDF, stacked vertically.

If you'd prefer to stack the images horizontally rather than vertically, you can use +append rather than -append.

Test experimental versions with GitHub Actions

2021-10-20T12:25:54Z

If you’re a library maintainer you may want to test your code against every available version of a language, including pre-release versions. After all, you want to know if your code won’t work on the latest and greatest version of a language before your users do.

Using a matrix

GitHub Actions allows you to specify a matrix of values and will spawn one job per combination.

The following workflow will trigger two jobs, build (7.4) and build (8.0):

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]

Our libraries aren’t just what we write though, it’s a combination of the code we author and the libraries we depend on. Let’s add another parameter that checks with the lowest version of our dependencies and the highest available version of our dependencies.

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        composer-version: ["lowest", "highest"]

This workflow will generate four jobs:

build (7.4, lowest)
build (7.4, highest)
build (8.0, lowest)
build (8.0, lowest)

As you can see, the number of jobs can grow exponentially as you add new dimensions to the matrix.

Testing a prerelease version

PHP 8.1 has been released and we’d like to test our code against this version, but we don’t want our build to fail if the tests don’t pass as we don’t officially support PHP 8.1 yet.

GitHub Actions supports a continue-on-error parameter for jobs that if set to true will record a failure for the job, but won’t fail the whole workflow run:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: true

This has the unwanted side effect that our workflow will never fail, even if 7.4 or 8.0 don’t pass. Fortunately, we can set continue-on-error conditionally:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: ${{ matrix.php == '8.1' }}

In this example continue-on-error only evaluates to true for 8.1, which achieves the behaviour that we were looking for.

Testing multiple versions

Checking the value directly in the continue-on-error field works, but it gets hard to read if you’re running multiple experimental versions. Imagine that in addition to the new version (8.4), we want to test on PHP 7.3 too. We don’t officially support it any more, but our customers are still heavy users of it and it’d be good to know if all the tests pass.

We start with a matrix definition that contains all of our non-experimental versions. This will run two jobs as there is only a single entry in the experimental row.

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]

Matrix definitions allow an include parameter to be provided to add new entries to the matrix. Anything added with include adds a single entry, not a new permutation.

Let’s add an experimental 8.1 job to our workflow:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true

Which would generate the following set of jobs:

json
[
  { "php": "7.4", "experimental": false },
  { "php": "8.0", "experimental": false },
  { "php": "8.1", "experimental": true }
]

In code, the algorithm would look like the following:

js
const jobs = [];
for (let version of matrix.php) {
  for (let experiment of matrix.experimental) {
    jobs.push({ php: version, experimental: experiment });
  }
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false }
// ]
for (let j of matrix.include) {
  jobs.push(j);
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false },
//   { php: '8.1', experimental: true }
// ]

We can now use the experimental flag to decide if we want to continue-on-error:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

Adding a new experimental version is easy at this point. We can add an additional include entry for any versions that we want to test:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
          - php: "7.3"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

This workflow will run four jobs, and allow 7.3 and 8.1 to fail without failing the entire workflow run.

Dynamic matrix generation with GitHub Actions

2021-10-18T08:24:23Z

Using a build matrix with GitHub Actions allows us to run tests across multiple combinations of operating systems, platforms and languages. You can set up huge matrices (a matrix with three parameters, each of which has three values will run 27 jobs!), but most workflows that you see will have a single matrix entry.

In this example, we’re going to run our tests on the currently supported versions of Node using the following workflow:

yaml
on: push
jobs:
  ci:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: [12, 14, 16]
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.version }}
      - run: npm ci
      - run: npm test

The list of supported Node.js versions is accurate today, but what happens in 6 months? How about 12 months? We’d have to come back and update this list each time the list of supported versions changes.

Let’s update this workflow to build the list of supported versions dynamically so that it’s always up to date.

Create a matrix from an output

In the workflow above we set matrix.version manually, but it doesn’t have to be hardcoded. We can populate it using the output values a previous job. Let’s update the workflow to use an output as the matrix.version value and add a new job that returns a list of currently supported node versions:

yaml
on: push
jobs:
  build-matrix:
    runs-on: ubuntu-latest
    steps:
      - id: set-matrix
        run: echo '::set-output name=version_matrix::["12","14","16"]'
    outputs:
      version_matrix: ${{ steps.set-matrix.outputs.version_matrix }}
  ci:
    needs: build-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: ${{ fromJson(needs.build-matrix.outputs.version_matrix) }}
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.version }}
      - run: npm ci
      - run: npm test

The list of versions is still hardcoded in the set-matrix job, but we’re one step closer to dynamically populating the list of active versions.

Fetch supported Node.js versions

The next step is to replace the hardcoded string (["12","14","16"]) with a data source that dynamically updates. For this post I’m using endoflife.date to fetch the list of active versions.

The API returns a list of objects that look like the following:

json
{
  "cycle": "16",
  "release": "2021-04-20",
  "lts": false,
  "support": "2022-10-18",
  "eol": "2024-04-30",
  "latest": "16.10.0"
}

Out action doesn’t need all of that data - all we need is the cycle value, and to ensure that the eol date is before today. Fortunately the Actions runners have jq installed, which we can use to manipulate the data.

Here’s the command that we use to return the list of active versions:

bash
curl https://endoflife.date/api/nodejs.json | jq -c '[.[] | select(.eol > (now | strftime("%Y-%m-%d"))) | .cycle]'
# ["16","14","12"]

We need to use a sub-shell to run the command in our workflow, which means wrapping the command in $().

Putting it all together, this is what the workflow looks like:

yaml
on: push
jobs:
  build-matrix:
    runs-on: ubuntu-latest
    steps:
      - id: set-matrix
        run: echo "::set-output name=version_matrix::$(curl https://endoflife.date/api/nodejs.json | jq -c '[.[] | select(.eol > (now | strftime("%Y-%m-%d"))) | .cycle]')"
    outputs:
      version_matrix: ${{ steps.set-matrix.outputs.version_matrix }}
  ci:
    needs: build-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        version: ${{ fromJson(needs.build-matrix.outputs.version_matrix) }}
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.version }}
      - run: npm ci
      - run: npm test

When this workflow runs, it fetches the list of active Node.js versions in the first job then triggers one instance of the ci job per version returned.

Try adding it to one of your own projects to see it work 🥳

Conclusion

The example shown above is simple, but super-useful! As soon as there’s a new version of node available, our code will start being tested against it. No more surprises where the community find out that your code doesn’t work before you do.

I’ve seen a couple of interesting applications of this, but none more interesting than RectorPHP testing their refactoring tool against the top 50 packages in the PHP ecosystem. The top 50 list is populated dynamically to ensure that as new packages are released and picked up, they get early feedback on if their tool is compatible.