michaelheap.com

Flatten nested foreach loops with Terraform

2024-09-12T09:26:22Z

Given a list of identifiers and an associated list of values, here's how to flatten them all down to a single list of unique values.

hcl
# Define your input data. We'll convert this in to a list of ${name}-${version} values
variable "ai_providers" {
  type = list(object({
    name     = string
    versions = list(string)
  }))
  default = [
    {
      name     = "OpenAI"
      versions = ["gpt-3.5-turbo", "gpt-4o"]
    },
    {
      name     = "Anthropic"
      versions = ["Opus", "Sonnet", "Haiku"]
    },
    {
      name     = "Mistral"
      versions = ["7B", "8x7B", "8x22B"]
    },
    {
      name     = "llama3"
      versions = ["8B", "70B"]
    },
  ]
}
## Flatmap using nested loops and flatten()
locals {
  ai_providers_flattened = flatten([
    for provider in var.ai_providers : [
      for version in provider.versions : {
        version  = version
        provider = provider
        key      = "${provider.name}-${version}"
      }
    ]
  ])
}
# Iterate over the flattened list to create resources
resource "demo_foo" "my_resource" {
  for_each = { for api in local.ai_providers_flattened : api.key => api }
  name     = each.value.provider
  version  = each.value.version
}

The for_each = { for api in local.ai_providers_flattened : api.key => api } line confused me for a while. This line is used to identify the resource later as my_resource is now a map of objects rather than a single object. Let's walk through it step by step

demo_foo.my_resource is a map of objects
demo_foo.my_resource[api.key] points to a single resource
api.key is defined as "${provider.name}-${version}"
Details for the demo_foo resource for OpenAI gpt-4o is available at demo_foo.my_resource["OpenAI-gpt-4o"]

Bind a Hyper/Meh key with Keychron Launcher

2024-09-05T07:55:33Z

I've been using a Keychron Q1 HE this week instead of my trusty Ergodox EZ as I've been travelling.

One of the biggest issues I had is that none of my keyboard shortcuts were working as I didn't have a hyper key bound. I could have used something like Karabiner to rebind keys, but I knew I could do it at the keyboard level.

It took me far too long to figure out, so I'm writing it down for future me.

Configuring Hyper/Meh keys

First of all, VIA App doesn't work for the Q1 HE. You have to use the Keychron configuration UI at https://www.launcher.keychron.com/.

I learned about the Any key definition from this useful Reddit comment. On the keymap screen, select Custom then Any. It will show a text input where you should put the keys to send.

Be aware - the key that you're binding will change as you type on the keyboard. After entering the keys, click on the key that you want to bind before pressing Enter.

To create a Hyper (Left Control + Shift + Alt + GUI) key, enter HYPR(KC_NO). To create a Meh (Left Control + Shift + Alt) key, enter MEH(KC_NO).

The GUI key is cmd on MacOS, and the Windows key on Windows.

The KC_NO parameter tells QMK not to send any additional keycodes with the modifiers. This allows you to press other keys on the keyboard.

Personally, I like to define a LCAG(KC_NO) (Left Control + Alt + GUI) key. This means I can have a single key as my keyboard trigger, and double the number of available shortcuts by additionally pressing Shift.

Learn more about QMK

The full list of modifier keys is available in the QMK docs.

I've used prebuilt modifiers such as HYPR and LCAG, but you can also nest them to build composite modifiers. e.g. Send ctrl+alt with LCTL(LALT(KC_NO)). You can change KC_NO for a key to send a full key combination, e.g. ctrl+alt+delete with LCTL(LALT(KC_DEL))

Downloading webcomics with Dosage

2024-05-20T16:25:59Z

I’ve been reading some webcomics for a few years, but never went back to the beginning to learn how each character was introduced. I liked the idea, but with 5000+ posts in some feeds, I constantly lost my position and/or had better things to do.

I recently spent some time on holiday and decided to try and copy the whole series on to my iPad to read. I intended to write a small app to download a series and convert to a .cbz file, but it turns out it already exists.

Dosage

Dosage is a Python CLI designed to download webcomics in to a folder. I always have issues installing Python apps, so I looked to see if Dosage was available as a Docker image. It wasn’t, but it wasn’t too difficult to package up.

If you want to do the same, create a Dockerfile with the following contents:

docker
FROM python:bullseye
# Install pipx
RUN python3 -m pip install --user pipx \
    && python3 -m pipx ensurepath
# Install dosage
RUN /root/.local/bin/pipx install "dosage[css,bash] @ git+https://github.com/webcomics/dosage.git"
ENTRYPOINT ["/root/.local/bin/dosage"]

Build the image (you don’t need to clone the Dosage repo - the Dockerfile will do it for you):

bash
docker build -t dosage .

Then finally, add an alias for the a dosage command. This is optional, but encouraged:

bash
alias dosage='docker run --rm -v $(pwd):/Comics dosage:latest'

Finally, run dosage to download the comic of your choice:

bash
dosage --list
dosage -a TheOrderOfTheStick

Create a CBZ

A .cbz file is a renamed zip file. To create one:

bash
zip -r oots.cbz TheOrderOfTheStick/*

Now you can copy the .cbz in to the comic app of your choice.

Presentations on an ultrawide monitor

2024-05-13T09:17:51Z

I love having an ultra wide monitor, but it does make presenting tricky at times. All of the available conferencing systems allow you to share either a single window, or all of your screen. For most people, sharing a single window which is the right size works.

For me, it doesn’t. Most of the time I’m switching between a code editor, web browser and terminal. I need a way to share a 1080p section of my screen, no matter which window it’s showing.

Here are the two options I use:

Use Zoom to share a specific area of the screen + Hammerspoon to position the windows
Use DeskPad + any other conferencing tool

Zoom and Hammerspoon

This is the approach I use 99% of the time. It fits my existing workflow with a single display and allows me to drag things in to view as needed (or rather, use the Hammerspoon shortcut below).

This approach consists of two sections. The first is to position the window where it needs to be using Hammerspoon. I have a keyboard binding for this:

lua
function setWindowPosition(key, w,h,x,y)
  hs.hotkey.bind({"cmd", "alt", "ctrl", "shift"}, key, function()
    local win = hs.window.focusedWindow()
    local cursize = win:size()
    cursize.w = w
    cursize.h = h
    local f = win:frame()
    f.x = x
    f.y = y
    win:setFrame(f)
    win:setSize(cursize)
  end)
end
setWindowPosition("P", 1920, 1080, 2420, 285) -- Right

2420, 285 places the window 2420px from the left, and 285px down from the top of my screen. This is just below where my camera is mounted, allowing me to see the window and also look at the camera.

The second trick is to use Zoom’s “Portion of screen” option and select the area of the screen that your window covers.

At this point the core problem is solved, but there’s one remaining thing that frustrated me. The app switcher shows when I cmd+tab, but some icons are cut off as I’m only sharing part of the screen.

To solve this, I bind specific windows to a keyboard shortcut with Hammerspoon and switch using the keyboard. This prevents the app switcher from showing on screen.

I use cmd+alt+ctrl+shift+y to bind the focused window to y, and cmd+alt+ctrl+y to switch to it instantly. I have six hotkeys bound, which provides plenty of windows if I need them.

lua
function windowSwitch(binder)
  local windowId = 0;
  hs.hotkey.bind({"cmd", "alt", "ctrl", "shift" }, binder, function()
    windowId = hs.window.focusedWindow():id();
  end)
  hs.hotkey.bind({"cmd", "alt", "ctrl"}, binder, function()
    hs.window.find(windowId):focus();
  end)
end
windowSwitch("y")
windowSwitch("u")
windowSwitch("i")
windowSwitch("h")
windowSwitch("j")
windowSwitch("k")

DeskPad

Not all conferencing systems allow you to share a portion of your screen like Zoom. If I’m presenting through an app that doesn’t allow you to share a specific area and I need to show multiple windows, I use DeskPad.

DeskPad adds a virtual screen to your Mac, and an app that shows what’s on that screen. This allows you to share your entire “screen” whilst still keeping the majority of your ultra wide available for other things.

I configure DeskPad’s screen to be above my usual screen to make it easy to drag windows on to it.

If you’re feeling really fancy, you can combine DeskPad and Hammerspoon to move windows to DeskPad with a keyboard shortcut:

lua
hs.hotkey.bind({"cmd", "alt", "ctrl" }, ".", function()
  hs.window.focusedWindow():moveToScreen('Deskpad'):maximize()
end)

The End

If you're using Zoom, I can highly recommend the first approach. I use it daily and it hasn't failed me yet. If you're using a conferencing platform that doesn't allow you to share a portion of your screen, DeskPad is a great tool that lets you share a virtual screen at the resolution you desire.

Kong Gateway Quickstart

2024-05-07T08:34:54Z

A $dayjob related TIL today. Here's how to quickly deploy Kong Gateway locally with the Dev Portal enabled (you'll need an enterprise license).

This is useful for testing things locally.

Run the Gateway:

bash
curl -Ls https://get.konghq.com/quickstart | PROXY_PORT=80 bash -s -- -m \
    -e KONG_PASSWORD=changeme \
    -e KONG_ADMIN_GUI_URL=http://manager.example \
    -e KONG_ADMIN_GUI_API_URL=http://admin.example \
    -e KONG_PORTAL_GUI_PROTOCOL=http \
    -e KONG_PORTAL_API_URL=http://portalapi.example \
    -e KONG_PORTAL_GUI_HOST=portal.example \
    -e KONG_PORTAL_SESSION_CONF='{"cookie_name": "portal_session", "secret": "PORTAL_SUPER_SECRET", "storage": "kong", "cookie_secure": false, "cookie_domain":".example"}'
    -e KONG_PORTAL="on" \
    -e KONG_LICENSE_DATA \
    -t 3.4

Create routes that proxy based on host name:

bash
curl -X POST localhost:8001/services -d name=admin -d url=http://localhost:8001
curl -X POST localhost:8001/services -d name=manager -d url=http://localhost:8002
curl -X POST localhost:8001/services -d name=portal -d url=http://localhost:8003
curl -X POST localhost:8001/services -d name=portalapi -d url=http://localhost:8004
curl -X POST localhost:8001/services/admin/routes -d hosts=admin.example
curl -X POST localhost:8001/services/manager/routes -d hosts=manager.example
curl -X POST localhost:8001/services/portal/routes -d hosts=portal.example
curl -X POST localhost:8001/services/portalapi/routes -d hosts=portalapi.example

Add those domains to your /etc/hosts file:

bash
echo '
127.0.0.1   proxy.example
127.0.0.1   admin.example
127.0.0.1   manager.example
127.0.0.1   portal.example
127.0.0.1   portalapi.example
' | sudo tee -a /etc/hosts

Now you can visit http://manager.example and log in with kong_admin / changeme.

Designing OpenAPI Schemas

2024-04-27T19:36:48Z

I’ve written a lot of OpenAPI schemas over the last couple of years, and have developed a pattern that helps with maintenance. You have a minimum of two logical schemas for every entity in your system, Foo and FooRequest . Foo is a union of FooRequest and any computed fields from the system.

Let’s look as a concrete example, a Pet in an adoption shelter. A pet has two user set fields, name and type, and one computed field, created_at. You can’t set the created_at field when creating or updating a pet, which means we have two schemas:

PetRequest : name, type
Pet: name, type, created_at

The following example isn't the best way to accomplish the GET/POST split! It's shown as it's a common pattern used in many specifications, but keep reading for a better solution.

When you model this with JSON schema, it looks like the following:

yaml
components:
  schemas:
    PetRequest:
      type: object
      properties:
        name:
          type: string
        type:
          type: string
    Pet:
      allOf:
        - $ref: "#/components/schemas/PetRequest"
        - type: object
          properties:
            created_at:
              type: string

Any updates to the PetRequest object will automatically be reflected in the Pet object. However, this is such a common pattern that OpenAPI has keywords to help built in.

Simplify with `readOnly: true`

If you can split your schema in to "user provided" and "computed" values cleanly, you only need a single schema thanks to the readOnly keyword.

yaml
components:
  schemas:
    Pet:
      type: object
      properties:
        name:
          type: string
        type:
          type: string
        created_at:
          type: string
          readOnly: true

The readOnly keyword causes the schema to be split in to two virtual schemas - one for GET and one for POST/PATCH/PUT. Any OpenAPI renderer (I tested with Redoc) will remove created_at from the POST schema.

Complex APIs

In an ideal world, you wouldn’t need more than one single Pet schema. However, we live in the real world and sometimes there are additional requirements. Here are some examples:

Pets have an adopted_at time, which can only be set when updating a pet, not when creating
Pets have a total_steps field which is computed from an external source that is not cached and is too expensive to show when listing multiple pets

These requirements mean that we have to split Pet in to two, Pet and CreatePetRequest. We also need a MinimalPet representation for the GET /pets endpoint. This results in three distinct schemas:

CreatePetRequest: name, type
Pet: name, type, adopted_at (readOnly), created_at (readOnly)
PetWithDetails: name, type, adopted_at (readOnly), total_steps (readOnly), created_at (readOnly)

These schemas can be composed to build the entities we need at runtime. CreatePetRequest is the base as it contains the minimum available information. Pet builds on this by adding adopted_at and created_at. Finally, PetWithDetails adds computed fields that are expensive to calculate such as total_steps.

CreatePetRequest -> Pet -> PetWithDetails.

Expressed using JSON schema, it looks like this:

yaml
components:
  schemas:
    CreatePetRequest:
      type: object
      properties:
        name:
          type: string
        type:
          type: string
    Pet:
      allOf:
        - $ref: "#/components/schemas/CreatePetRequest"
        - type: object
          properties:
            adopted_at:
              type: string
            created_at:
              type: string
              readOnly: true
    PetWithDetails:
      allOf:
        - $ref: "#/components/schemas/Pet"
        - type: object
          properties:
            total_steps:
              type: string
              readOnly: true

This gets expanded to the following schemas (courtesy of Swagger UI):

Here's a complete OpenAPI specification that uses these schemas.

An ideal world

Although the model works for APIs that have complex requirements, I consider needing separate models for create and update requests a design flaw. In this example API, you could make adopted_at a nullable field and use the same schema for both create and update requests.

I also consider needing a minimal representation of an entity a design flaw.There are cases where there is a real requirement that needs these schemas (e.g. if total_steps must always be accurate and can’t be cached) but these cases are rare.

If you find yourself using more than one schema, take a moment to reconsider your API design and see how you can simplify it.

Workspace layouts with Hammerspoon Grid

2024-01-10T15:10:39Z

Since I switched from Arch to MacOS there’s been an i3 shaped hole in my life. I’ve used apps like Divvy to approximate it, but it’s not the same.

I tried out some of the MacOS tiling window managers, but couldn’t get to grips with them. Between having to trigger things using external apps (Yabai and skhd) and having to disable system integrity protection, things just didn’t click. One of the things I loved about i3 was its simplicity.

After many years of searching, I think I’ve found a solution in Hammerspoon.

Hammerspoon?

Hammerspoon is an automation framework for MacOS that allows you to script various things using Lua.

I’ve written about Hammerspoon before, and have been using it for window management since I abandoned Divvy a couple of years ago (when I configured a dedicated hyper key on my keyboard).

I was recently browsing through Jakub’s Hammerspoon config and spotted his usage of hs.grid to configure predefined layouts.

Predefined layouts were one of my big use cases for i3, and so I went digging.

hs.grid

hs.grid splits your screen into an x by y grid. You can place windows by specifying a width/height along with an offset for the x and y values.

I have split my screen in to an 8x2 grid, with zero margin. I also have a shortcut to show the grid overlay on screen in case I want to position a single window in a non-standard layout.

lua
hs.grid.setMargins(hs.geometry.size(0,0))
hs.grid.setGrid('8x2')
hs.hotkey.bind(hyper,'g',function()
  hs.grid.show()
end)

To align a window to the grid, you call hs.grid.set and pass an instance of hs.window as the first parameter. The window is being placed on the right hand side of the screen (4 cells offset on the x-axis, 0 on the y-axis, and it covers a 4x2 space on an 8x2 grid).

lua
hs.grid.set(window, '4,0 4x2')

Automatic layouts

One of the things that I loved about Jakub’s Hammerspoon config is that he had a helper function to discover all of the windows for a certain app before reflowing windows in to the grid.

I borrowed this idea and built some convenience functions which allow you to define layouts like this:

lua
defineLayout("Writing", 1, {
	{'Bear', '0,0 2x2'},
	{'Arc', '2,0 4x2', true}
});
defineLayout("Code", 2, {
	{'Arc', '0,0 2x2'},
	{'Code - Insiders', '2,0 4x2', true},
	{'iTerm', '6,0 2x2'}
})

Now I can press hyper+1 to have Bear on the left 1/4 of the screen and Arc in the middle half. If I’m writing code, I can press hyper+2 to have VSCode in the middle taking up half of the screen, with Arc on the left and a terminal on the right.

Applications will only be positioned if they are already launched. Hammerspoon provides a hs.application.launchOrFocus() method, but I chose to implement focusIfLaunched as sometimes I don’t have all the apps open (e.g. Bear is optional when I’m writing).

Give it a go

Hammerspoon is wonderful, and I recommend giving it a go if you haven’t already tried it. My Hammerspoon config is publicly available, and whilst I’m not a power user, it saves me a ton of time every single day.

If you’ve a Hammerspoon user and have tips and tricks you want to share, I’m @mheap@hachyderm.io. I’d love to hear them!

Bonus content

I’ve recently stumbled across aerospace, which bills itself as “an i3-like tiling window manager for macOS”. I haven’t given it a go yet, but it looks promising based on the YouTube video.

Could/Should/Will/Why

2024-01-08T20:27:59Z

I had an interesting conversation with my manager this week. I’ve been working on a lot of different projects, and was a little spread thin. In one of our calls my manager asked me why I’m working on .

We sat and chatted about it, and I tried to answer the question: why am I doing these things? After hearing me out, they agreed that everything I was doing was important. Then they repeated the question, with a slightly different emphasis:

Why am I doing these things?

I didn't have a good answer. The things needed doing. No-one else was doing them. I could do them, so I took them on. But why was I the one taking them on?

A day has a finite number of hours. If I’m spending my time doing X, then Y isn’t getting done. If Y is something that only I can (or will) do, then isn’t that a better use of my time than X?

Here’s a concrete example. I had a choice to either:

Write some tutorials for one of our products using the API
Build a proof of concept Terraform provider to understand if we want to invest in the next financial year

My decision? To do both. Work on the proof of concept first, and write some tutorials between my meetings once it's done.

That’s the wrong answer.

It turns out that writing tutorials isn’t my responsibility. More than that, by writing them I’m taking away learning opportunities from others. I’m taking it away from the product manager that wants to learn to write documentation. I’m taking it away from the writer that wants to learn more about API development.

Anything I take on should meet one of the following criteria:

No-one else can do the work
It’s time sensitive, and no-one else has capacity
There’s a learning opportunity for me

Building a system

When my manager asked why I was working on these projects, I realised that I couldn’t answer. We talked about who could do the work, who should do the work, and who will do the work. This gave me a good framework for evaluating projects before starting them.

I’ve added a final column - why - to help keep a record of why the person that will take on the project is responsible.

Here’s an example with a couple of projects:

Project	Could	Should	Will	Why
Public Demos	DevRel, Sales Engineering	Sales Engineering	Sales Engineering	Demos are a core part of the SE role. If DevRel build demos, there would be duplication.
Release Demos	Product, DevRel, Product Marketing	Product Marketing	DevRel	DevRel has an engineering background to understand the features + how to get started
Splunk Tutorial	Docs, DevRel	Docs	DevRel	Docs team is at capacity with regular release documentation
VSCode Extension	DevRel, Engineering	Engineering	N/A	Project not prioritised.
Terraform Provider	DevRel, Engineering	DevRel	DevRel	This is an initial proof of concept exploration. If the idea shows potential, it's added to the Engineering roadmap.

Let’s work through the tutorials example in more detail:

A community member asked how to send logs from Kong on a VM to Splunk
DevRel couldn’t find any existing documentation to send them
DevRel the docs team if they could write a new tutorial
The docs team share that they’re already overcommitted.

At this point DevRel have three choices:

Abandon the project
Submit it in to the docs team backlog
Do it themselves

This is your classic prioritisation problem. Whichever option you choose, fill out the why column with the rationale then move on.

Has it helped?

Kind of. I still take on more projects than I should, but having an explicit will step forces me to think about why we’re taking on this specific project. It forces me to talk to other people about the project. Could their teams take on the work? Do they want to take on the work? Are there learning opportunities?

Next time you have an idea for some work that needs doing, run it through could/should/will/why to make sure that you’re the best suited person or team to complete the project.

Create an AWS RDS database

2024-01-05T15:52:09Z

I've been using RDS to provide throwaway databases for testing and wanted to work from the CLI to speed things up. The aws rds incantations were hard to find at times, so here they are for posterity.

Create a publicly accessible Postgres database:

bash
aws rds create-db-instance --db-instance-identifier demo-db --db-instance-class db.t3.micro --allocated-storage 50 --engine postgres --publicly-accessible --master-username postgres --master-user-password YOUR_PASSWORD

Show the database status and connection details:

bash
aws rds describe-db-instances | jq '.DBInstances[] | select(.DBInstanceIdentifier == "demo-db") | .DBInstanceStatus,.Endpoint'

Delete the database:

bash
aws rds delete-db-instance --db-instance-identifier demo-db --skip-final-snapshot

Create an Azure AKS Cluster

2024-01-05T15:50:50Z

I needed to test Azure AKS with the Application Gateway Ingress Controller. Thankfully, Azure makes it easy with it's az aks CLI.

Create a new Azure resource group:

bash
az group create --name mheap-test --location uksouth

Create a new AKS cluster with the Azure Application Gateway Ingress Controller enabled:

bash
az aks create --name mheap-aks-test --resource-group mheap-test --node-count 1 --network-plugin azure --enable-managed-identity --enable-addons ingress-appgw --appgw-name mheap-appgw --appgw-subnet-cidr "10.225.0.0/16" --generate-ssh-keys

Configure kubectl to connect to this cluster:

bash
az aks get-credentials --resource-group mheap-test --name mheap-aks-test

Once you've finished testing, delete the AKS cluster and resource group:

bash
az aks delete --name mheap-aks-test --resource-group mheap-test --yes
az group delete --name mheap-test --yes