michaelheap.com

How I work: Pocket

2022-09-02T15:26:50Z

There are plenty of tools (like Pocket) that sell us on the dream of saving things for later if we don't have chance to read or watch it there and then. Unfortunately, these tools usually end up being a graveyard of forgotten links that we never look at again and we'd have been better off just closing the tab.

I saved my first link to Pocket in October 2012 (I was an Instapaper user before then), and have read 15,433 items (with 124 unread) since then.

How do I do it? Let me introduce you to pocket-tagger-cli

Categorizing content

Pocket Tagger is a CLI tool that I built back in 2018 to help automatically categorise saved items in Pocket using tags based on predefined rules. It allows you to define regular expressions and define tags based on those expressions. You can express things like:

Tag it with length-medium if it's > 1500 and < 5000 characters
Tag t with has-slides if the URL contains slideshare.net or speakerdeck.com
Tag it with github if the page mentions github but not github-actions (this is a separate tag)

The rules can be executed against the item URL, text content or raw HTML (I usually use URL and text content).

Here's a sample configuration that will execure the rules above. If multiple entries are provided (e.g. has-slides) then they are evaluated as an OR. If you nest them in another array (e.g. github), then they're evaluated as an AND.

json
{
  "regexes": {
    "length-medium": "^[\\s\\S]{1501,5000}$",
    "slideshare-url": "^https://www.slideshare.net",
    "speakerdeck-url": "^https://www.speakerdeck.com",
    "github": "github.com",
    "github-actions": "github actions"
  },
  "tags": {
    "url": {
      "has-slides": ["slideshare-url", "speakerdeck-url"],
      "github": [["github", "!github-actions"]]
    },
    "content": {
      "length-medium": ["length-medium"]
    },
    "html": {}
  }
}

This is a very small configuration that should get you started. If you want to see my full configuration (which is still evolving) scroll to the end of this post.

Running Pocket Tagger

Pocket tagger isn't a particularly user-friendly tool. It involves registering for your own Pocket developer app and obtaining your own authentication credentials. I've built some tooling to help, but it still involves having Node.js installed and running some CLI commands.

If you want to run Pocket tagger yourself, here are the instructions:

Create an application in the Pocket Apps section
Run npx pocket-auth-cli <consumer_key>to obtain an access_token
Install pocket-tagger-cli
Generate a ruleset with pocket-tagger --generate ~/.pocket/tagger.json
Run DEBUG="pocket-tagger*" pocket-tagger and watch as it fetches all your Pocket entries and categorises them (this may take a little while)

Michael's configuration

I mentioned above that my configuration has been developed over time and is considerably larger than the sample set of rules provided. I try to segment things based on context and batch process things when reading. My rough split of content is:

Sites such as Goodreads (probably a book to buy), Hacker News (yes, I read the comments), reddit (easy-reading) or BoardGameGeek (games to buy)
Leadership related reading (New York Times, Harvard Business Review, Fast Company, Inc, Forbes etc)
Cloud related reading (AWS, GCP, Azure)
Tooling related content (Ansible, Chef, Prometheus, Jenkins, VSCode)
GitHub related content (I'm a member of the Stars programme)
Work related content (Kong, DevRel, DevX, Documentation, API Gateway, OpenAPI)
Media to check out (YouTube, Spotify)
Fitness related things (Fitness, Gym)

I then choose which tags to read based on what I'm doing. I'll check my media tag during my lunch break, or read work related content between calls. Leadership related content is my go-to when I'm in the queue at the supermarket, and GitHub related content gets read on a Monday morning ahead of a weekly Stars call. Finally, I tend to save the length-essay items for when I'm travelling by train or by plane. Having a long piece of content to read is a great way to get absorbed and time passes very quickly.

Here's my current configuration, in case you want to take any inspiration from it:

json
{
  "regexes": {
    "length-too-short": "^[\\s\\S]{0,5}$",
    "length-short": "^[\\s\\S]{6,1500}$",
    "length-medium": "^[\\s\\S]{1501,5000}$",
    "length-long": "^[\\s\\S]{5001,15000}$",
    "length-essay": "^[\\s\\S]{15001,100000}$",
    "length-too-long": "^[\\s\\S]{100001,}$",
    "code-php": "(\\sphp\\s|<\\?php|composer)",
    "code-javascript": "\\sjavascript\\s",
    "code-node": "(node\\.js|nodejs)",
    "code-golang": "golang",
    "tool-composer": "composer",
    "tool-ansible": "ansible",
    "tool-chef": "\\schef\\s",
    "tool-prometheus": "prometheus",
    "tool-jenkins": "jenkins",
    "tool-docker": "docker",
    "tool-kubernetes": "kubernetes",
    "call-for-proposals-url": "(https://cfp|http://cfp)",
    "call-for-proposals": "(call for proposals|call for papers)",
    "openapi": "openapi",
    "documentation": "(documentation|docs|portal)",
    "management": "(manager|1:1)",
    "product-management": "product manage",
    "api-management": "api manage",
    "api": "api",
    "devrel": "(devrel|dev rel|developer relations)",
    "devx": "(devx|dx|developer experience|dev experience)",
    "kong": "\\s(kong|kuma|mesh|gateway)\\s",
    "fitness": "(fitness|workout)",
    "gym": "\\s(gym|bench|weights)\\s",
    "twitch": "twitch",
    "slideshare-url": "slideshare.net",
    "speakerdeck-url": "speakerdeck.com",
    "slideshare-embed": "https://www.slideshare.net/slideshow/embed_code",
    "speakerdeck-embed": "speakerdeck-embed",
    "is-pdf": ".pdf$",
    "hacker-news": "ycombinator",
    "hukd": "hotukdeals.com",
    "github": "github.com",
    "github-actions": "github actions",
    "npm": "npmjs.com",
    "twitter": "twitter.com",
    "reddit": "reddit.com",
    "medium": "medium.com",
    "devto": "dev.to",
    "hackernoon": "hackernoon.com",
    "lifehacker": "lifehacker.com",
    "theverge": "theverge.com",
    "youtube": "(youtube.com|youtu.be)",
    "nytimes": "nytimes.com",
    "hbr": "hbr.org",
    "fastcompany": "fastcompany.com",
    "inc": "inc.com",
    "qz": "qz.com",
    "entrepreneur": "entrepreneur.com",
    "forbes": "forbes.com",
    "bloomberg": "bloomberg.com",
    "wsj": "wsj.com",
    "nordicapis": "nordicapis.com",
    "spotify": "spotify.com",
    "amazon": "amazon.com",
    "papercall": "papercall.io",
    "goodreads": "goodreads.com",
    "boardgamegeek": "(boardgamegeek.com|bgg)",
    "api-gateway": "api gateway",
    "aws": "(amazon web service|aws)",
    "google-cloud": "(google cloud platform|gcp)",
    "azure": "azure",
    "work-api": "(rest|api)",
    "vscode-marketplace": "marketplace.visualstudio.com",
    "vscode": "vscode"
  },
  "tags": {
    "url": {
      "openapi": ["openapi"],
      "cloud-aws": ["aws"],
      "cloud-gcp": ["google-cloud"],
      "cloud-azure": ["azure"],
      "tool-vscode": ["vscode-marketplace"],
      "is-cfp": ["papercall", "call-for-proposals-url"],
      "is-pdf": ["is-pdf"],
      "has-slides": ["slideshare-url", "speakerdeck-url"],
      "site-goodreads": ["goodreads"],
      "site-hn": ["hacker-news"],
      "site-hukd": ["hukd"],
      "site-mediumesque": ["medium", "devto", "hackernoon"],
      "site-lifehacker": ["lifehacker"],
      "site-twitter": ["twitter"],
      "site-reddit": ["reddit"],
      "site-bgg": ["boardgamegeek"],
      "site-nordicapis": ["nordicapis"],
      "motivational": [
        "nytimes",
        "hbr",
        "fastcompany",
        "qz",
        "inc",
        "entrepreneur",
        "forbes",
        "bloomberg",
        "wsj"
      ],
      "fun-read": ["theverge"],
      "github": [["github", "!github-actions", "!openapi"]],
      "github-actions": ["github-actions"],
      "npm": ["npm"],
      "amazon": ["amazon"],
      "management": ["management"],
      "media-youtube": ["youtube"],
      "media-spotify": ["spotify"]
    },
    "content": {
      "boardgames": ["boardgamegeek"],
      "ops-kubernetes": ["tool-kubernetes"],
      "ops-docker": ["tool-docker"],
      "cloud-aws": ["aws"],
      "cloud-gcp": ["google-cloud"],
      "cloud-azure": ["azure"],
      "management": [["management", "!product-management", "!api-management"]],
      "product-management": ["product-management"],
      "api-management": ["api-management"],
      "github-actions": ["github-actions"],
      "is-cfp": ["papercall", "call-for-proposals"],
      "length-error": ["length-too-short", "length-too-long"],
      "length-short": ["length-short"],
      "length-medium": ["length-medium"],
      "length-long": ["length-long"],
      "length-essay": ["length-essay"],
      "tool-composer": ["tool-composer"],
      "tool-ansible": ["tool-ansible"],
      "tool-chef": ["tool-chef"],
      "tool-prometheus": ["tool-prometheus"],
      "tool-jenkins": ["tool-jenkins"],
      "tool-vscode": ["vscode-marketplace", "vscode"],
      "work-kong": ["kong"],
      "work-devx": ["devx"],
      "work-devrel": ["devrel"],
      "work-documentation": ["documentation"],
      "work-api-gateway": ["api-gateway"],
      "work-api": [["api", "!openapi"]],
      "work-openapi": ["openapi"],
      "personal-fitness": ["fitness"],
      "personal-gym": ["gym"],
      "streaming": ["twitch"]
    },
    "html": {
      "code-php": ["code-php"],
      "code-javascript": ["code-javascript"],
      "code-node": ["code-node"],
      "code-golang": ["code-golang"],
      "has-slides": ["slideshare-embed", "speakerdeck-embed"]
    }
  },
  "cache": {
    "engine": "file",
    "tmpDir": "/tmp/url-tagger",
    "ttl": 86400
  }
}

Conclusion

Pocket Tagger can't magically create more time for you to read the things you save to Pocket, but it can help you find the right things to read based on your current context. I never want to watch YouTube videos on a train, and I don't want to start a long article if I only have 10 minutes before my next call.

Categorising content allows you to break your backlog down into smaller, accessible chunks of content that you can start to read one by one. Focus on clearing a specific tag, and when that tag is empty you'll get the dopamine kick from finishing something, even if you still have 500 more articles to read later.

PS: I use Reeder on MacOS and iOS as my Pocket client, if you're looking for a client too

Make an S3 bucket public

2022-07-08T16:01:00Z

I recently needed to make all objects in a bucket public, but not allow people to list the contents of the bucket. Finding the correct policy took me a little while, so I'm reproducing here for posterity:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
        }
    ]
}

jamf stuck on “Locating hardware information (macOS 11.6.5)”

2022-04-10T13:53:44Z

We use jamf at work, and I periodically run jamf recon and jamf policy to ensure that things are all up to date.

I noticed that the process was consistently hanging with the message Stuck on “Locating hardware information (macOS 11.6.5)”. From my googling it looked as though this was software update related, so I uninstalled jamf, applied the update and then reinstalled. Problem solved!

That is, until I tried to run jamf recon again and it hung at the same point. I opened up software update and it just hung, with the gear spinning forever.

More googling ensued, and I eventually found the magic command that I needed to get things working again:

bash
sudo launchctl kickstart -k system/com.apple.softwareupdated

Once softwareupdated had restarted, I could run jamf recon without any issues

Remove Docker containers by tag

2022-03-24T13:40:47Z

I always forget to pass the --rm flag when running command line tools with Docker. This means that I end up with a large history of exited containers that I no longer need.

There's plenty of information out there about how to clean up all stopped containers, or all unused images. I wanted to delete all stopped containers with a specific tag. Here's how to do it:

bash
docker rm $(docker container ls -a -q --filter ancestor=image-name:tag --filter status=exited)

This works thanks to the filtering capabilities when combined with the -q option to output only container IDs. These are then fed in to docker rm to remove all images with that tag that are in the exited status.

Update a single item in a list with `vuex`

2022-02-07T18:30:21Z

I recently needed to find an item in a list of objects by ID and update one of it’s properties whilst working with Vue 3 and VueX. It took me longer than I'd like to admit to figure out, so I'm writing it down here.

Due to how objects are passed by reference in JS, you can reference the item using Object.assign to update a single property.

Here’s the store, including a mutation to update the value:

js
import { createStore } from "vuex";
const store = createStore({
  state() {
    return {
      items: [
        { id: 1, value: "One", color: "red" },
        { id: 2, value: "Two", color: "yellow" },
        { id: 3, value: "Three", color: "skyblue" },
      ],
    };
  },
  mutations: {
    setColor(state, { id, color }) {
      console.log(`Setting block ${id} to ${color}`);
      const item = state.items.find((i) => i.id == id);
      if (item) {
        Object.assign(item, { ...item, color });
      }
    },
  },
});
export default store;

Here’s how I called the mutation from the component:

js
setColor(color) {
  store.commit("setColor", { id: this.id, color });
  return false;
}

The GitHub repo in this post renders a sample application where you can change the colour of some blocks.

Using `openvpn-client` with Docker

2022-01-28T15:06:56Z

I recently worked out the correct incantation to get a set of containers to connect to the internet via a VPN using docker-compose. I run it on a QNAP NAS, but it should work on any Linux-like system (I couldn’t get it working on MacOS).

The following docker-compose.yml will create two containers - one to run the VPN client and a second that runs curl. I use this to run curl ipv4.canhazip.com to check that the container has connectivity and the VPN IP address is returned.

yaml
---
version: "3"
services:
  vpn:
    container_name: vpn
    image: dperson/openvpn-client:latest
    cap_add:
      - net_admin
    restart: unless-stopped
    volumes:
      - /dev/net/tun:/dev/net/tun
      - ./vpn-config:/vpn # You'll need to provide this
    security_opt:
      - label:disable
    ports:
      - 8001:8001
      - 8002:8002
      - 8003:8003
    networks:
      - bridge_vpn
    entrypoint: ["/sbin/tini", "--", "/usr/bin/openvpn.sh", "-d"]
  curl:
    image: alpine/curl
    container_name: curl
    restart: unless-stopped
    network_mode: service:vpn
networks:
  bridge_vpn:

This compose file will expose ports 8001, 8002 and 8003 from any containers using network_mode: service:vpn and make them accessible via a bridge network. This is useful when running a service that connects to the internet using a VPN. (There are no exposed ports in this demo, but I wanted to make a note here as in my actual deployment some of the other services expose ports.)

The biggest change to this configuration compared to a lot of guides out there is the entrypoint line for the vpn service. The openvpn-client image supports a -d flag that adds some DNS related pre/post scripts. I found that these are required to make connectivity work via the VPN.

You may have noticed the vpn-config folder being mounted. This is where you’ll provide your VPN configuration and authentication files. I tested this with Private Internet Access. If you’re a PIA customer, you’ll need to create a file named vpn.auth with your PIA username on the first line, and password on the second line, plus a vpn.conf file with the following contents:

apache
client
dev tun
proto udp
remote sweden.privacy.network 1198
resolv-retry infinite
nobind
persist-key
# persist-tun # disable to completely reset vpn connection on failure
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass /vpn/vpn.auth # to be reachable inside the container
comp-lzo
verb 1
reneg-sec 0
crl-verify /vpn/crl.rsa.2048.pem # to be reachable inside the container
ca /vpn/ca.rsa.2048.crt # to be reachable inside the container
disable-occ
keepalive 10 30 # send a ping every 10 sec and reconnect after 30 sec of unsuccessfull pings
pull-filter ignore "auth-token" # fix PIA reconnection auth error that may occur every 8 hours

This configuration connects to Sweden, but you can switch to any endpoint that you like. You’ll also need to download crl.rsa.2048.pem and ca.rsa.2048.pem from the PIA site.

Once you have everything configured, it’s time to create the containers by running docker-compose up -d. Once that’s complete, you can test your connection by logging in to the vpn service and making a HTTP call:

bash
docker-compose exec vpn bash -c "curl ipv4.canhazip.com"

If the above command returns an IP address successfully, you can also test it using the curl container which is configured to use the VPN for all network connectivity:

bash
docker-compose run curl ipv4.canhazip.com

At this point, you have a docker-compose setup that connects all of the containers configured via an OpenVPN connection. Replace the curl service with any other service you may want to run behind a VPN and enjoy as your traffic is safe from snooping eyes.

The GitHub wiki is an anti-pattern

2022-01-23T18:08:26Z

The “should I use the wiki or a docs folder on GitHub?” discussion comes up every 6 months or so, and following Shawn Wang’s three strikes rule I thought it was about time I wrote something down about it.

The initial version of this post opened with “You can use the wiki or a docs folder for your GitHub project, both are valid choices” but as I wrote more, I realised that there is a single reason to use a wiki, and many more reasons not to use the wiki. So many in fact, that I consider using the wiki on GitHub is an anti-pattern.

Let’s start with the benefits of using a wiki:

You can get to the wiki contents in a single click from anywhere in the repo
There is no 2.

Really, the only benefit that I’ve been able to find to using the wiki is that it’s always there.

How about the reasons not to use the wiki?

Documentation is versioned alongside your code when using the /docs folder. If you need to use an old version, the docs are easy to find
The documentation isn’t available locally when someone clones your repo (you can clone the wiki separately, but this is a hidden feature)
Documentation edits get the same treatment as code. They get a full peer review through the pull request process
You can use GitHub Actions to lint your docs using tools such as Vale
People can work with tooling that they already know (e.g. vscode with spellcheck)
Wikis provide limited branding opportunities. They all look pretty much the same
The wiki doesn’t support image uploads, so you have to put images somewhere else anyway

Now that you’re sold on the idea of keeping docs alongside your code, how do you make it easy for people to view them?

Add your docs to your repository in the /docs folder. Do not use the gh-pages branch as this prevents docs being versioned alongside code
Set up a GitHub pages build to publish the docs
- If you’re just getting started, I recommend using the just-the-docs theme and letting GitHub build and publish your docs
- If you prefer to build your own workflow (e.g. using Hugo), you can use this GitHub Action to publish the docs
Add a single wiki page directing people to the hosted documentation

Using the /docs folder is the highest effort-to-reward ratio option whilst you’re building out a new product. At some point your docs will outgrow a single folder, and then all bets are off. You’ll want a separate repo with its own build process, pull request review guidelines and a whole host of other things. At that point people are already used to working with docs in a repository, and the migration from /docs to its own repo should be seamless for your contributors.

Whether you agree or disagree, I’d love to hear your thoughts on Twitter

VSCode only showing one open tab

2021-11-22T12:53:51Z

My VSCode instance has been refusing to show more than a single tab for a while, even with preview disabled and when double clicking on a filename to open.

Most blog posts talking about how to fix this have a similar resolution to this Stack Overflow thread. That is:

json
"workbench.editor.showTabs": false

This didn't work for me, nor did setting "workbench.editor.enablePreview": false

What worked for me is to lock a tab as opened by pressing Ctrl+K+Enter whilst the tab is focused. This updated the UI to show a tab bar and when I double clicked on another file it opened as expected.

Convert a PDF to a stacked jpg

2021-11-11T13:36:24Z

I built some UI mockups in Google Slides today and wanted to share the output as a single jpg file. The way to achieve this is to export the deck as PDF, use imagemagick to convert the PDF to images, then combine the images.

Once you have your PDF, save it as input.pdf in a new folder, then run the following commands:

bash
convert -density 150 input.pdf -quality 90 output.jpg
convert -append output-* final.jpg

You'll now have a file named final.jpg in your folder that contains all of the pages from the PDF, stacked vertically.

If you'd prefer to stack the images horizontally rather than vertically, you can use +append rather than -append.

Test experimental versions with GitHub Actions

2021-10-20T12:25:54Z

If you’re a library maintainer you may want to test your code against every available version of a language, including pre-release versions. After all, you want to know if your code won’t work on the latest and greatest version of a language before your users do.

Using a matrix

GitHub Actions allows you to specify a matrix of values and will spawn one job per combination.

The following workflow will trigger two jobs, build (7.4) and build (8.0):

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]

Our libraries aren’t just what we write though, it’s a combination of the code we author and the libraries we depend on. Let’s add another parameter that checks with the lowest version of our dependencies and the highest available version of our dependencies.

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        composer-version: ["lowest", "highest"]

This workflow will generate four jobs:

build (7.4, lowest)
build (7.4, highest)
build (8.0, lowest)
build (8.0, lowest)

As you can see, the number of jobs can grow exponentially as you add new dimensions to the matrix.

Testing a prerelease version

PHP 8.1 has been released and we’d like to test our code against this version, but we don’t want our build to fail if the tests don’t pass as we don’t officially support PHP 8.1 yet.

GitHub Actions supports a continue-on-error parameter for jobs that if set to true will record a failure for the job, but won’t fail the whole workflow run:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: true

This has the unwanted side effect that our workflow will never fail, even if 7.4 or 8.0 don’t pass. Fortunately, we can set continue-on-error conditionally:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: ${{ matrix.php == '8.1' }}

In this example continue-on-error only evaluates to true for 8.1, which achieves the behaviour that we were looking for.

Testing multiple versions

Checking the value directly in the continue-on-error field works, but it gets hard to read if you’re running multiple experimental versions. Imagine that in addition to the new version (8.4), we want to test on PHP 7.3 too. We don’t officially support it any more, but our customers are still heavy users of it and it’d be good to know if all the tests pass.

We start with a matrix definition that contains all of our non-experimental versions. This will run two jobs as there is only a single entry in the experimental row.

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]

Matrix definitions allow an include parameter to be provided to add new entries to the matrix. Anything added with include adds a single entry, not a new permutation.

Let’s add an experimental 8.1 job to our workflow:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true

Which would generate the following set of jobs:

json
[
  { "php": "7.4", "experimental": false },
  { "php": "8.0", "experimental": false },
  { "php": "8.1", "experimental": true }
]

In code, the algorithm would look like the following:

js
const jobs = [];
for (let version of matrix.php) {
  for (let experiment of matrix.experimental) {
    jobs.push({ php: version, experimental: experiment });
  }
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false }
// ]
for (let j of matrix.include) {
  jobs.push(j);
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false },
//   { php: '8.1', experimental: true }
// ]

We can now use the experimental flag to decide if we want to continue-on-error:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

Adding a new experimental version is easy at this point. We can add an additional include entry for any versions that we want to test:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
          - php: "7.3"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

This workflow will run four jobs, and allow 7.3 and 8.1 to fail without failing the entire workflow run.