michaelheap.com

Filter Google Analytics report

2022-11-29T13:58:08Z

I'm trying out Radar to keep track of analytics at a glance. It's Google Analytics collection works great, but the way that we've set up Google Analytics the data for multiple sites ends up in the same property.

I wanted to filter the report to only traffic to a specific subdomain. It took me too long to work out that I needed a dimensionFilterClauses entry with the ga:pagePath (via the schema).

Here's how to filter your reports in case you (or I) need to do it in the future:

json
{
  "reportRequests": [
    {
      "viewId": "YOUR_VIEW_ID",
      "dateRanges": [
        {
          "startDate": "today",
          "endDate": "today"
        }
      ],
      "dimensions": [
        {
          "name": "ga:date"
        }
      ],
      "metrics": [
        {
          "expression": "ga:pageviews"
        }
      ],
      "dimensionFilterClauses": [
        {
          "filters": [
            {
              "dimensionName": "ga:pagePath",
              "operator": "PARTIAL",
              "expressions": ["docs.konghq.com"]
            }
          ]
        }
      ]
    }
  ]
}

Other things that may be useful:

Endpoint: https://analyticsreporting.googleapis.com/v4/reports:batchGet
Method: POST
Auth: Bearer [token]

Then transform the results using the following JS snippet:

javascript
const pageCount = Number(
  data.reports[0].data.totals[0].values[0]
).toLocaleString();

Upgrade from Winston 2 to 3 using colors

2022-10-23T11:39:59Z

I updated a project from Winston 2 to Winston 3 this morning and needed to make a few changes to how it was configured.

I previously created a logger instance, then switched to using syslog levels with a custom colour scheme. Here's how I did it in Winston 2:

javascript
var winston = require("winston");
var logger = new winston.Logger({
  transports: [
    new winston.transports.Console({
      level: "warning",
      colorize: true,
    }),
  ],
});
syslogColors = {
  debug: "rainbow",
  info: "cyan",
  notice: "white",
  warning: "yellow",
  error: "bold red",
  crit: "inverse yellow",
  alert: "bold inverse red",
  emerg: "bold inverse magenta",
};
winston.addColors(syslogColors);
logger.setLevels(winston.config.syslog.levels);

In Winston 3, the new winston.Logger call has been replaced with winston.createLogger, levels must be provided in the constructor, and formatting is handled by individual formatters using the logform library.

Here's how to achieve the same thing as above in Winston 3:

javascript
const winston = require("winston");
const syslogColors = {
  debug: "rainbow",
  info: "cyan",
  notice: "white",
  warning: "yellow",
  error: "bold red",
  crit: "inverse yellow",
  alert: "bold inverse red",
  emerg: "bold inverse magenta",
};
let format = winston.format.combine(
  winston.format.colorize({
    all: true,
    colors: syslogColors,
  }),
  winston.format.simple()
);
var logger = winston.createLogger({
  transports: [
    new winston.transports.Console({
      level: "warning",
      format,
    }),
  ],
  levels: winston.config.syslog.levels,
});

Running `gickup` on a QNAP NAS

2022-10-15T18:00:01Z

I learned about Gickup whilst reading Jeff Geerling’s backup plan and thought it would be a useful thing to run for my own repos.

I initially ran it on my local machine periodically, then realised that I could run it on my QNAP NAS every night. Here’s how I did it:

Create a shared folder named gickup to contain the backups. I’ve got a volume named Backups that contains this, but any volume will do
Create a directory to store your backups by running mkdir -p /share/gickup/repos
Edit /etc/config/crontab and add the following line to the end: 0 3 * * * docker run --rm -v /share/gickup:/hostvol buddyspencer/gickup:0.9 /gickup/app /hostvol/config.yml > /share/gickup/last-run.log 2>&1. This will run gickup every day at 3am
Reload cron to pick up that change by running crontab /etc/config/crontab && /etc/init.d/crond.sh restart

You may have noticed that I reference /hostvol/config.yml. I mount the /share/gickup folder a /hostvol, and config.yml is my gickup configuration file. Here’s what it looks like:

yaml
# config.yml in /share/gickup
source:
  github:
    - token: CHANGE_ME_TO_YOUR_GITHUB_PAT
      excludeorgs: # this excludes repos from the organizations "foo" and "bar"
        - foo
        - bar
destination:
  local:
    - path: /hostvol/repos
      structured: true

Create your own config.yml, making sure you change the token value. If you’ve configured all of the above, all the repositories you have access to will be backed up at 3am each day.

Transcribe audio locally on MacOS

2022-10-14T12:01:20Z

I've been using Just Press Record on my watch to capture things that are on my mind whilst working outdoors. I'm usually pretty good at processing the recordings regularly, but recently found a folder of recordings that I may or may not have processed.

Rather than listening to all of the recordings, I wanted to batch transcribe them and read the text. There are plenty of online tools available, but I wanted a local CLI tool which lead me to hear.

The 0.1 release of hear is from March and is missing some important bugfixes. I cloned the repo and ran make to build the tool locally and it works great.

hear accepts an input file and a language, and writes the output to stdout. Here's an example that processes a file against the en-GB model and writes out transcription.txt:

bash
hear -i ./2022-01-29/10-17-46.m4a -l en-GB > transcription.txt

Important: You may see the word abort when running hear. This is due to hear needing to request microphone permissions. If you get an error, try running hear in Terminal.app rather than any other application

If you have a whole folder of recordings to transcribe, you might find this snippet useful. It produces a .txt file next to each recording with the contents of the transcription:

bash
for i in `find . -name '*.m4a'`; do hear -i $i -l en-GB > ${i/.m4a/}.txt; done

Here's what it looks like for me after running that command (with some folders removed for brevity):

❯ ls -R
2021-12-27	2022-01-29

./2021-12-27:
14-17-39.m4a	14-19-34.m4a	14-21-18.m4a	14-21-54.m4a	14-49-09.m4a	14-53-17.m4a
14-17-39.txt	14-19-34.txt	14-21-18.txt	14-21-54.txt	14-49-09.txt	14-53-17.txt

./2022-01-29:
10-17-46.m4a	10-17-46.txt	10-56-59.m4a	10-56-59.txt

Mock `process.env` using `mocked-env`

2022-10-07T10:18:07Z

When testing values in process.env, they persist for all further tests in your test suite. This can have unwanted side effects, making tests pass or fail unintentionally.

Using mocked-env, you can reset the environment to the pre-test state after every test runs. It works by taking a copy of process.env when you call mockEnv, then restores it once the test has run (afterEach).

Here's an example that shows how environment variables set earlier in the test suite don't persist in to later tests:

javascript
const mockEnv = require("mocked-env");
// Variables to store references to what we need to reset
let restore;
let restoreTest;
beforeEach(() => {
  restore = mockEnv({
    VALUE_ONE: "Hello",
  });
  restoreTest = () => {};
});
afterEach(() => {
  restore();
  restoreTest();
});
test("environment concatenation (world)", async () => {
  restoreTest = mockEnv({
    VALUE_TWO: "World",
  });
  expect(concatenateValuesFromEnv("VALUE_ONE", "VALUE_TWO")).toBe(
    "Hello World"
  );
});
test("environment concatenation (michael)", async () => {
  restoreTest = mockEnv({
    VALUE_TWO: "Michael",
  });
  expect(concatenateValuesFromEnv("VALUE_ONE", "VALUE_TWO")).toBe(
    "Hello Michael"
  );
});
test("environment concatenation (missing value)", async () => {
  expect(concatenateValuesFromEnv("VALUE_ONE", "VALUE_TWO")).toBe("Hello ");
});
function concantenateValuesFromEnv(a, b) {
  return `${process.env[a]} ${process.env[b]}`;
}

Without using mocked-env, the last test would fail as the returned value would be Hello Michael as VALUE_TWO is set in the previous test, and would not be cleaned up without mocked-env.

Using `GITHUB_TOKEN` as a default `input` value

2022-10-07T10:10:27Z

If you need to authenticate against the GitHub API in your GitHub Action, you might prompt the user to provide their GITHUB_TOKEN secret through an input. This isn't necessarily needed thanks to the github.token context and default input values.

You can gain access to the GITHUB_TOKEN secret by default by configuring your action.yml file to access github.token:

yaml
name: Action Name
description: My awesome description goes here
runs:
  using: node12
  main: dist/index.js
inputs:
  token:
    description: The GitHub token to use when calling the API
    default: ${{ github.token }}
    required: false

How I work: Pocket

2022-09-02T15:26:50Z

There are plenty of tools (like Pocket) that sell us on the dream of saving things for later if we don't have chance to read or watch it there and then. Unfortunately, these tools usually end up being a graveyard of forgotten links that we never look at again and we'd have been better off just closing the tab.

I saved my first link to Pocket in October 2012 (I was an Instapaper user before then), and have read 15,433 items (with 124 unread) since then.

How do I do it? Let me introduce you to pocket-tagger-cli

Categorizing content

Pocket Tagger is a CLI tool that I built back in 2018 to help automatically categorise saved items in Pocket using tags based on predefined rules. It allows you to define regular expressions and define tags based on those expressions. You can express things like:

Tag it with length-medium if it's > 1500 and < 5000 characters
Tag t with has-slides if the URL contains slideshare.net or speakerdeck.com
Tag it with github if the page mentions github but not github-actions (this is a separate tag)

The rules can be executed against the item URL, text content or raw HTML (I usually use URL and text content).

Here's a sample configuration that will execure the rules above. If multiple entries are provided (e.g. has-slides) then they are evaluated as an OR. If you nest them in another array (e.g. github), then they're evaluated as an AND.

json
{
  "regexes": {
    "length-medium": "^[\\s\\S]{1501,5000}$",
    "slideshare-url": "^https://www.slideshare.net",
    "speakerdeck-url": "^https://www.speakerdeck.com",
    "github": "github.com",
    "github-actions": "github actions"
  },
  "tags": {
    "url": {
      "has-slides": ["slideshare-url", "speakerdeck-url"],
      "github": [["github", "!github-actions"]]
    },
    "content": {
      "length-medium": ["length-medium"]
    },
    "html": {}
  }
}

This is a very small configuration that should get you started. If you want to see my full configuration (which is still evolving) scroll to the end of this post.

Running Pocket Tagger

Pocket tagger isn't a particularly user-friendly tool. It involves registering for your own Pocket developer app and obtaining your own authentication credentials. I've built some tooling to help, but it still involves having Node.js installed and running some CLI commands.

If you want to run Pocket tagger yourself, here are the instructions:

Create an application in the Pocket Apps section
Run npx pocket-auth-cli <consumer_key>to obtain an access_token
Install pocket-tagger-cli
Generate a ruleset with pocket-tagger --generate ~/.pocket/tagger.json
Run DEBUG="pocket-tagger*" pocket-tagger and watch as it fetches all your Pocket entries and categorises them (this may take a little while)

Michael's configuration

I mentioned above that my configuration has been developed over time and is considerably larger than the sample set of rules provided. I try to segment things based on context and batch process things when reading. My rough split of content is:

Sites such as Goodreads (probably a book to buy), Hacker News (yes, I read the comments), reddit (easy-reading) or BoardGameGeek (games to buy)
Leadership related reading (New York Times, Harvard Business Review, Fast Company, Inc, Forbes etc)
Cloud related reading (AWS, GCP, Azure)
Tooling related content (Ansible, Chef, Prometheus, Jenkins, VSCode)
GitHub related content (I'm a member of the Stars programme)
Work related content (Kong, DevRel, DevX, Documentation, API Gateway, OpenAPI)
Media to check out (YouTube, Spotify)
Fitness related things (Fitness, Gym)

I then choose which tags to read based on what I'm doing. I'll check my media tag during my lunch break, or read work related content between calls. Leadership related content is my go-to when I'm in the queue at the supermarket, and GitHub related content gets read on a Monday morning ahead of a weekly Stars call. Finally, I tend to save the length-essay items for when I'm travelling by train or by plane. Having a long piece of content to read is a great way to get absorbed and time passes very quickly.

Here's my current configuration, in case you want to take any inspiration from it:

json
{
  "regexes": {
    "length-too-short": "^[\\s\\S]{0,5}$",
    "length-short": "^[\\s\\S]{6,1500}$",
    "length-medium": "^[\\s\\S]{1501,5000}$",
    "length-long": "^[\\s\\S]{5001,15000}$",
    "length-essay": "^[\\s\\S]{15001,100000}$",
    "length-too-long": "^[\\s\\S]{100001,}$",
    "code-php": "(\\sphp\\s|<\\?php|composer)",
    "code-javascript": "\\sjavascript\\s",
    "code-node": "(node\\.js|nodejs)",
    "code-golang": "golang",
    "tool-composer": "composer",
    "tool-ansible": "ansible",
    "tool-chef": "\\schef\\s",
    "tool-prometheus": "prometheus",
    "tool-jenkins": "jenkins",
    "tool-docker": "docker",
    "tool-kubernetes": "kubernetes",
    "call-for-proposals-url": "(https://cfp|http://cfp)",
    "call-for-proposals": "(call for proposals|call for papers)",
    "openapi": "openapi",
    "documentation": "(documentation|docs|portal)",
    "management": "(manager|1:1)",
    "product-management": "product manage",
    "api-management": "api manage",
    "api": "api",
    "devrel": "(devrel|dev rel|developer relations)",
    "devx": "(devx|dx|developer experience|dev experience)",
    "kong": "\\s(kong|kuma|mesh|gateway)\\s",
    "fitness": "(fitness|workout)",
    "gym": "\\s(gym|bench|weights)\\s",
    "twitch": "twitch",
    "slideshare-url": "slideshare.net",
    "speakerdeck-url": "speakerdeck.com",
    "slideshare-embed": "https://www.slideshare.net/slideshow/embed_code",
    "speakerdeck-embed": "speakerdeck-embed",
    "is-pdf": ".pdf$",
    "hacker-news": "ycombinator",
    "hukd": "hotukdeals.com",
    "github": "github.com",
    "github-actions": "github actions",
    "npm": "npmjs.com",
    "twitter": "twitter.com",
    "reddit": "reddit.com",
    "medium": "medium.com",
    "devto": "dev.to",
    "hackernoon": "hackernoon.com",
    "lifehacker": "lifehacker.com",
    "theverge": "theverge.com",
    "youtube": "(youtube.com|youtu.be)",
    "nytimes": "nytimes.com",
    "hbr": "hbr.org",
    "fastcompany": "fastcompany.com",
    "inc": "inc.com",
    "qz": "qz.com",
    "entrepreneur": "entrepreneur.com",
    "forbes": "forbes.com",
    "bloomberg": "bloomberg.com",
    "wsj": "wsj.com",
    "nordicapis": "nordicapis.com",
    "spotify": "spotify.com",
    "amazon": "amazon.com",
    "papercall": "papercall.io",
    "goodreads": "goodreads.com",
    "boardgamegeek": "(boardgamegeek.com|bgg)",
    "api-gateway": "api gateway",
    "aws": "(amazon web service|aws)",
    "google-cloud": "(google cloud platform|gcp)",
    "azure": "azure",
    "work-api": "(rest|api)",
    "vscode-marketplace": "marketplace.visualstudio.com",
    "vscode": "vscode"
  },
  "tags": {
    "url": {
      "openapi": ["openapi"],
      "cloud-aws": ["aws"],
      "cloud-gcp": ["google-cloud"],
      "cloud-azure": ["azure"],
      "tool-vscode": ["vscode-marketplace"],
      "is-cfp": ["papercall", "call-for-proposals-url"],
      "is-pdf": ["is-pdf"],
      "has-slides": ["slideshare-url", "speakerdeck-url"],
      "site-goodreads": ["goodreads"],
      "site-hn": ["hacker-news"],
      "site-hukd": ["hukd"],
      "site-mediumesque": ["medium", "devto", "hackernoon"],
      "site-lifehacker": ["lifehacker"],
      "site-twitter": ["twitter"],
      "site-reddit": ["reddit"],
      "site-bgg": ["boardgamegeek"],
      "site-nordicapis": ["nordicapis"],
      "motivational": [
        "nytimes",
        "hbr",
        "fastcompany",
        "qz",
        "inc",
        "entrepreneur",
        "forbes",
        "bloomberg",
        "wsj"
      ],
      "fun-read": ["theverge"],
      "github": [["github", "!github-actions", "!openapi"]],
      "github-actions": ["github-actions"],
      "npm": ["npm"],
      "amazon": ["amazon"],
      "management": ["management"],
      "media-youtube": ["youtube"],
      "media-spotify": ["spotify"]
    },
    "content": {
      "boardgames": ["boardgamegeek"],
      "ops-kubernetes": ["tool-kubernetes"],
      "ops-docker": ["tool-docker"],
      "cloud-aws": ["aws"],
      "cloud-gcp": ["google-cloud"],
      "cloud-azure": ["azure"],
      "management": [["management", "!product-management", "!api-management"]],
      "product-management": ["product-management"],
      "api-management": ["api-management"],
      "github-actions": ["github-actions"],
      "is-cfp": ["papercall", "call-for-proposals"],
      "length-error": ["length-too-short", "length-too-long"],
      "length-short": ["length-short"],
      "length-medium": ["length-medium"],
      "length-long": ["length-long"],
      "length-essay": ["length-essay"],
      "tool-composer": ["tool-composer"],
      "tool-ansible": ["tool-ansible"],
      "tool-chef": ["tool-chef"],
      "tool-prometheus": ["tool-prometheus"],
      "tool-jenkins": ["tool-jenkins"],
      "tool-vscode": ["vscode-marketplace", "vscode"],
      "work-kong": ["kong"],
      "work-devx": ["devx"],
      "work-devrel": ["devrel"],
      "work-documentation": ["documentation"],
      "work-api-gateway": ["api-gateway"],
      "work-api": [["api", "!openapi"]],
      "work-openapi": ["openapi"],
      "personal-fitness": ["fitness"],
      "personal-gym": ["gym"],
      "streaming": ["twitch"]
    },
    "html": {
      "code-php": ["code-php"],
      "code-javascript": ["code-javascript"],
      "code-node": ["code-node"],
      "code-golang": ["code-golang"],
      "has-slides": ["slideshare-embed", "speakerdeck-embed"]
    }
  },
  "cache": {
    "engine": "file",
    "tmpDir": "/tmp/url-tagger",
    "ttl": 86400
  }
}

Conclusion

Pocket Tagger can't magically create more time for you to read the things you save to Pocket, but it can help you find the right things to read based on your current context. I never want to watch YouTube videos on a train, and I don't want to start a long article if I only have 10 minutes before my next call.

Categorising content allows you to break your backlog down into smaller, accessible chunks of content that you can start to read one by one. Focus on clearing a specific tag, and when that tag is empty you'll get the dopamine kick from finishing something, even if you still have 500 more articles to read later.

PS: I use Reeder on MacOS and iOS as my Pocket client, if you're looking for a client too

Make an S3 bucket public

2022-07-08T16:01:00Z

I recently needed to make all objects in a bucket public, but not allow people to list the contents of the bucket. Finding the correct policy took me a little while, so I'm reproducing here for posterity:

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME_HERE/*"
        }
    ]
}

jamf stuck on “Locating hardware information (macOS 11.6.5)”

2022-04-10T13:53:44Z

We use jamf at work, and I periodically run jamf recon and jamf policy to ensure that things are all up to date.

I noticed that the process was consistently hanging with the message Stuck on “Locating hardware information (macOS 11.6.5)”. From my googling it looked as though this was software update related, so I uninstalled jamf, applied the update and then reinstalled. Problem solved!

That is, until I tried to run jamf recon again and it hung at the same point. I opened up software update and it just hung, with the gear spinning forever.

More googling ensued, and I eventually found the magic command that I needed to get things working again:

bash
sudo launchctl kickstart -k system/com.apple.softwareupdated

Once softwareupdated had restarted, I could run jamf recon without any issues

Remove Docker containers by tag

2022-03-24T13:40:47Z

I always forget to pass the --rm flag when running command line tools with Docker. This means that I end up with a large history of exited containers that I no longer need.

There's plenty of information out there about how to clean up all stopped containers, or all unused images. I wanted to delete all stopped containers with a specific tag. Here's how to do it:

bash
docker rm $(docker container ls -a -q --filter ancestor=image-name:tag --filter status=exited)

This works thanks to the filtering capabilities when combined with the -q option to output only container IDs. These are then fed in to docker rm to remove all images with that tag that are in the exited status.