<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>michaelheap.com</title>
<subtitle>Thoughts on leadership, code and how to fix odd edge cases in tools (not necessarily in that order)</subtitle>
<link href="https://michaelheap.com/rss" rel="self"/>
<link href="https://michaelheap.com"/>
<updated>2025-12-27T17:11:26Z</updated>
<id>https://michaelheap.com</id>
<author>
<name>Michael Heap</name>
<email>[email protected]</email>
</author>
<entry>
<title>Electron Node Module Version</title>
<link href="https://michaelheap.com/electron-node-module-version/"/>
<updated>2025-12-27T17:11:26Z</updated>
<id>https://michaelheap.com/electron-node-module-version/</id>
<content type="html"><p>I've been building an Electron app for the first time. It was all going well until I tried to use a dependency that involved native components. When I tried to run the app, I got an error that <code>[...] was compiled against a different Node.js version</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">Error occurred </span><span style="color: #81A1C1">in</span><span style="color: #D8DEE9FF"> handler </span><span style="color: #81A1C1">for</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">tasks:fetch</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF">: Error: The module </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">/private/tmp/electron-demo/node_modules/better-sqlite3/build/Release/better_sqlite3.node</span><span style="color: #ECEFF4">'</span></div><div class="line"><span style="color: #D8DEE9FF">was compiled against a different Node.js version using</span></div><div class="line"><span style="color: #D8DEE9FF">NODE_MODULE_VERSION 137. This version of Node.js requires</span></div><div class="line"><span style="color: #D8DEE9FF">NODE_MODULE_VERSION 140. Please try re-compiling or re-installing</span></div><div class="line"><span style="color: #D8DEE9FF">the module </span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">for instance, using </span><span style="color: #ECEFF4">`</span><span style="color: #A3BE8C">npm rebuild</span><span style="color: #ECEFF4">`</span><span style="color: #D8DEE9FF"> or </span><span style="color: #ECEFF4">`</span><span style="color: #A3BE8C">npm install</span><span style="color: #ECEFF4">`</span><span style="color: #ECEFF4">)</span><span style="color: #D8DEE9FF">.</span></div></code></div></pre>
<p>It turns out that the node.js and Electron ABIs are not compatible (see the <a href="https://github.com/nodejs/node/blob/main/doc/abi_version_registry.json">compatibility matrix</a>), and any native code needs to be compiled specifically for Electron.</p>
<p>To solve the issue above, you have to rebuild native modules for Electron using <code>@electron/rebuild</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">npm install --save-dev @electron/rebuild</span></div><div class="line"><span style="color: #D8DEE9FF">./node_modules/.bin/electron-rebuild</span></div></code></div></pre>
<p>Then <code>npm start</code> and everything works.</p>
<blockquote>
<p>⚠️ You'll need to re-run <code>electron-rebuild</code> after every <code>npm install</code>.</p>
</blockquote>
<h2 id="reproduction-case" tabindex="-1">Reproduction case</h2>
<p>If you're looking for a simple reproduction case:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">git clone https://github.com/mheap/electron-build-sample</span></div><div class="line"><span style="color: #88C0D0">cd</span><span style="color: #D8DEE9FF"> electron-build-sample</span></div><div class="line"><span style="color: #D8DEE9FF">npm install</span></div><div class="line"><span style="color: #D8DEE9FF">npm start </span><span style="color: #616E88"># You'll get an error here</span></div></code></div></pre>
<p>Rebuild the native modules and try again:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">./node_modules/.bin/electron-rebuild</span></div><div class="line"><span style="color: #D8DEE9FF">npm start</span></div></code></div></pre>
<p>You'll see a task that says &quot;Do the thing&quot;.</p>
</content>
</entry>
<entry>
<title>Toolsmiths melt snowflakes</title>
<link href="https://michaelheap.com/toolsmiths-melt-snowflakes/"/>
<updated>2025-12-17T09:30:00Z</updated>
<id>https://michaelheap.com/toolsmiths-melt-snowflakes/</id>
<content type="html"><p>Every company has snowflakes. Each one unique, delicate, and guaranteed to melt under pressure.</p>
<p>They accumulate quietly: the one-off script someone runs before every deploy, the spreadsheet with its own arcane formula logic, the onboarding flow that requires five Slack DMs to complete. Over time, those snowflake processes pile up into a blizzard of operational friction.</p>
<p>Toolsmiths are the ones who melt them.</p>
<p>They’re not the loudest people in the room. You won’t see them pitching product roadmaps or leading all-hands. But they build the quiet infrastructure that keeps a company from freezing in place. A toolsmith sees repetition and decides it shouldn’t exist. They build a small script, an automation, a workflow, and suddenly something that was fragile becomes reusable. That’s the kind of leverage that scales an organization, yet it’s often invisible.</p>
<h2 id="from-infrastructure-to-everywhere" tabindex="-1">From Infrastructure to Everywhere</h2>
<p>The idea of codifying process isn’t new. The DevOps movement was built on it. Chef, Ansible, and later Terraform turned one-off, bespoke configurations into repeatable, version-controlled artifacts. Infrastructure stopped being artisanal; it became declarative.</p>
<p>But somewhere along the way, the rest of the business got left behind. Marketing teams still handcraft campaigns in silos. Support teams dig through logs by memory. BI teams spend days reconciling slightly different versions of the truth in spreadsheets. Every team has its own snowflakes, and very few have the tools (or the permission) to melt them.</p>
<p>If your business isn’t like this, congratulations! You’ve invested in systems and they’re paying off. Sadly though, many businesses haven’t (engineering teams included) and they’re just one typo away from a meltdown.</p>
<h2 id="the-small-tools-that-matter" tabindex="-1">The Small Tools That Matter</h2>
<p>A few years ago, I built a simple CLI tool for GitHub. Its only job was to <a href="https://github.com/mheap/pin-github-action">pin every GitHub Action to a specific SHA</a>. That’s it. It took an afternoon to write.</p>
<p>Before that, every repository had workflows that pulled the latest version of an action. This was fine, until one day an upstream change broke half our builds. The “fix” was manual: open each repo, edit the workflow to pin to a specific commit hash, commit, push. Tedious, error-prone, and repeated by multiple teams.</p>
<p>So I wrote a CLI that scanned all repos in our org and pinned each action automatically. One run. Hundreds of snowflakes gone.</p>
<p>No one outside engineering noticed. But internally, it saved dozens of hours and reduced risk every time someone added a new workflow.</p>
<h2 id="toolsmiths-aren%E2%80%99t-just-developers" tabindex="-1">Toolsmiths Aren’t Just Developers</h2>
<p><strong>Toolsmithing is a mindset, not a job title.</strong> It’s what happens when someone sees a messy process and refuses to accept it as normal.</p>
<ul>
<li>In BI, it’s the analyst who turns scattered spreadsheets into a shared dashboard that updates automatically.</li>
<li>In Support, it’s the agent who builds a small search interface over logs so others don’t need to memorize file paths or grep commands.</li>
<li>In Partnerships, it’s the manager who realises that every customer onboarding ends up being unique, and builds a form or workflow that standardizes it.</li>
</ul>
<p>These people often work in isolation because organizations don’t recognize what they’re doing as engineering. But it <em>is</em> engineering - just applied to people, processes, and information ins tead of servers and code.</p>
<h2 id="start-small%2C-then-scale" tabindex="-1">Start Small, Then Scale</h2>
<p>There’s a myth that internal tools need to be fully featured products. That mindset kills most automation before it starts. A wiki page of useful commands is a great first tool. A small script that one team uses can later become an API or a workflow in Slack.</p>
<p>The question isn’t “Can we build this perfectly?” It’s “Is this pain common enough to solve once and share?”</p>
<p>Too many companies overengineer the idea of tooling. They spin up projects, assign teams, and stall under the weight of process. But the best tools start informally. They’re acts of curiosity, not mandates. And once proven useful, they can be hardened, shared, and maintained.</p>
<h2 id="build-without-overbuilding" tabindex="-1">Build Without Overbuilding</h2>
<p>Code is a liability, and not every solution needs it. Some of the best internal tools aren’t codebases at all. They’re Zapier automations, Airtable forms, or Slack workflows. Platforms like Retool and Node-RED exist precisely to help non-developers build lightweight, domain-specific tools.</p>
<p>But even there, restraint matters. Internal tools don’t need to impress anyone; they need to save time. David Tuite calls this out in his essay on “<a href="https://www.davidtuite.com/hidden-const-mediocre-internal-tools/">the hidden cost of mediocre internal tools</a>”: if a tool is 80% right and saves hours a week, that’s success. Perfection is the enemy of progress.</p>
<h2 id="the-cultural-blind-spot" tabindex="-1">The Cultural Blind Spot</h2>
<p>Here’s the uncomfortable part: most companies don’t reward toolsmiths. They celebrate heroics instead. Everyone hears about the engineer who pulls an all-nighter, the support rep who clears a backlog manually. But <a href="https://sre.google/resources/practices-and-processes/no-heroes/">we don’t need any more superheroes</a>.</p>
<p>A toolsmith’s work is quieter. It’s the absence of pain, not the spectacle of solving it. But that quiet efficiency compounds. Each script, dashboard, and workflow removes another point of friction - another snowflake. Over time, it transforms the company’s metabolism.</p>
<p>The irony is that the companies best at building products for others are often the worst at building tools for themselves.</p>
<h2 id="melting-snowflakes-at-scale" tabindex="-1">Melting Snowflakes at Scale</h2>
<p>Empowering toolsmiths doesn’t mean spinning up a new “internal tools” department. It means creating permission structures that let anyone automate what hurts. It means treating those efforts as strategic, not peripheral. Give them time to do the work, and access to the systems they need.</p>
<p>Imagine if every engineer, analyst, and operations specialist had an hour a week to improve their own workflow. Imagine if they had access to shared infrastructure. A data warehouse, a Retool instance, a place to share scripts safely. The compound effect would dwarf most process initiatives.</p>
<p>That’s how you grow fast without breaking things: you melt snowflakes before they become avalanches.</p>
<h2 id="build-systems%2C-not-empires" tabindex="-1">Build Systems, not Empires</h2>
<p>Organizations love to talk about scale, but scale doesn’t come from adding headcount. It comes from eliminating friction. Toolsmiths are the people who make that possible.</p>
<p>If you want to grow, don’t just hire more hands. Empower the ones you have to build better tools. Give them permission to tinker. Recognise their work.</p>
<p>Because every snowflake you melt makes your company a little stronger.</p>
</content>
</entry>
<entry>
<title>Parse Codex `jsonl` with `jq`</title>
<link href="https://michaelheap.com/extract-codex-conversation/"/>
<updated>2025-12-12T16:27:46Z</updated>
<id>https://michaelheap.com/extract-codex-conversation/</id>
<content type="html"><p>I recently <a href="https://michaelheap.com/gh-saved-issues/">built a GitHub extension</a> using ChatGPT Codex, and I wanted to share the prompts that I gave to Codex to build the tool.</p>
<p>I thought about copy/pasting it out of the VSCode UI, but figured that the data must be available somewhere. It turns out that Codex stores sessions as <code>.jsonl</code> files in <code>.codex/sessions</code>.</p>
<p>Looking through the session files, there's a ton of data in there around what the LLM does, but I was only interested in messages that I sent. Fortunately each user message has <code>type: user_message</code>.</p>
<p>Here's a one liner that uses <code>jq</code> to output all of the user messages for a given session:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">cat .codex/sessions/2025/12/04/rollout-2025-12-04T22-06-28-019aeb67-0620-71c1-8cbb-756bc8845c6e.jsonl </span><span style="color: #81A1C1">|</span><span style="color: #D8DEE9FF"> \</span></div><div class="line"><span style="color: #D8DEE9FF"> jq -r </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">. |</span></div><div class="line"><span style="color: #A3BE8C"> select(.payload.type == "user_message") |</span></div><div class="line"><span style="color: #A3BE8C"> .payload.message |</span></div><div class="line"><span style="color: #A3BE8C"> split("\n## My request for Codex:\n")[1] + "\n\n--- NEW COMMAND ---\n\n"</span><span style="color: #ECEFF4">'</span></div></code></div></pre>
</content>
</entry>
<entry>
<title>Building `gh-saved-issues` using ChatGPT Codex</title>
<link href="https://michaelheap.com/gh-saved-issues/"/>
<updated>2025-12-12T07:59:44Z</updated>
<id>https://michaelheap.com/gh-saved-issues/</id>
<content type="html"><p>I spend a <em>lot</em> of time on the <code>/issues</code> page on GitHub. Working across multiple repos in multiple orgs, it can be difficult to keep up to date on what's happening. <code>/issues</code> allows me to see everything on a single page.</p>
<p>The <code>/issues</code> page allows you to create saved searches and view them when needed. Here are a couple of examples that I use every day:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># My PRs that are awaiting review</span></div><div class="line"><span style="color: #D8DEE9FF">is:pr author:@me state:open org:Kong -draft:true</span></div></code></div></pre>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Issues open on my personal projects (excluding dependency updates + my blog)</span></div><div class="line"><span style="color: #D8DEE9FF">state:open archived:false sort:updated-desc user:mheap -author:app/dependabot -repo:mheap/michaelheap.com is:issue</span></div></code></div></pre>
<p>I also started out with a &quot;all PRs awaiting review from me filter&quot;:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Awaiting review from me within the Kong org</span></div><div class="line"><span style="color: #D8DEE9FF">is:pr state:open org:Kong review-requested:@me</span></div></code></div></pre>
<p>However, I found that I wasn't looking at it as there were regularly too many items to review in a single session. So instead, I started segmenting my reviews by context:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># All Terraform PRs</span></div><div class="line"><span style="color: #D8DEE9FF">is:pr state:open -author:app/renovate </span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9FF">repo:Kong/terraform-provider-konnect OR repo:Kong/terraform-provider-konnect-beta OR repo:Kong/terraform-provider-kong-gateway</span><span style="color: #ECEFF4">)</span></div></code></div></pre>
<p>This worked much better, but then I realised that I'd need to keep 90% of the search consistent, but change the repos being shown for various different contexts.</p>
<p>So I did what anyone would - I started reverse engineering the GraphQL queries needed to configure saved searches and building a configuration file with template support.</p>
<p>If you're just looking for the tool, you can find it at <a href="https://github.com/mheap/gh-saved-issues">mheap/gh-saved-issues</a>.</p>
<p>This tool is my first attempt at building something with an agent. Keep reading to see my experience.</p>
<h3 id="building-with-codex" tabindex="-1">Building with Codex</h3>
<p>I did some research before prompting Codex, including extracting <code>curl</code> commands from developer tools. I pasted these commands into my initial prompt</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>You are an expert Go developer. You have been tasked to build a GitHub CLI extension to configure saved searches on GitHub. The CLI layer should be very thin, and all of the logic for working with the API must be included in library files.</p>
<p>The tool should accept a configuration file (located at $XDG_CONFIG_HOME/.github-searches.yaml) in the following format:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">searches</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Assigned to me (Kong)</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">state:open archived:false assignee:@me sort:updated-desc org:Kong</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Assigned to Steve (All Orgs)</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">state:open archived:false assignee:steve.doe sort:updated-desc</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Work from Alice</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">recent-work</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">user</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">alice.jones</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">time</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">7d</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Work from Bob</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">recent-work</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">user</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">bob.smith</span></div><div class="line"><span style="color: #8FBCBB">templates</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">recent-work</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">org:Kong ((assignee:{{ user }} AND is:issue) OR (is:pr AND author:{{ user }})) (is:open OR updated:&gt;@today-{{ default(time, </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">30d") }}) sort:updated-desc"</span></div></code></div></pre>
<p>If the search has an <code>id</code> associated with it, update the existing rule. Otherwise create a new one then update the configuration file to set the <code>id</code> value.</p>
<p>To create a new saved issue search, make the following request (but use Go, don't shell out to curl):</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">curl </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">https://github.com/_graphql</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF"> \</span></div><div class="line"><span style="color: #D8DEE9FF"> --data-raw </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">{"query":"c06c5627e09922bd28c6d34ff91d0530","variables":{"input":{"color":"GRAY","icon":"BOOKMARK","name":"Untitled view","query":"","searchType":"ISSUES"}}}</span><span style="color: #ECEFF4">'</span></div></code></div></pre>
<p>To update a search, use this request (shortcutId is the <code>id</code> from the config):</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">curl </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">https://github.com/_graphql</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF"> \</span></div><div class="line"><span style="color: #D8DEE9FF"> --data-raw </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">{"query":"379dbe4cf68c3485e48df2f699f5ae75","variables":{"input":{"color":"GRAY","description":"Test","icon":"BOOKMARK","name":"Demo123","query":"user:mheap","scopingRepository":null,"shortcutId":"SSC_kgDOAB3rrw"}}}</span><span style="color: #ECEFF4">'</span></div></code></div></pre>
<p>If a user sets <code>remove: true</code> on a query, you can delete the search using the following request:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">curl </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">https://github.com/_graphql</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF"> \</span></div><div class="line"><span style="color: #D8DEE9FF"> --data-raw </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">{"query":"2939ea7192de2c6284da481de6737322","variables":{"input":{"shortcutId":"SSC_kgDOAB3rsg"}}}</span><span style="color: #ECEFF4">'</span></div></code></div></pre>
<p>Build the CLI, and also bootstrap test cases for all library code.</p>
</div>
</div>
<p>At this point most of the work was done, except it didn't work. This graphql endpoint is not accessible using a GitHub token, it needs a cookie session to be specified</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>In addition to <code>req.Header.Set(&quot;Authorization&quot;, &quot;Bearer &quot;+c.token)</code>, allow me to specify a cookie like:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF"> -b </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">_device_id=ID_HERE; user_session=SESSION_HERE</span><span style="color: #ECEFF4">'</span><span style="color: #D8DEE9FF"> \</span></div></code></div></pre>
</div>
</div>
<p>This made the API requests work, but I realised that updating an existing config wasn't working due to a bug in findShortcutId.</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Fix findShortcutId</p>
<p>The response looks like the following:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">data</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">createDashboardSearchShortcut</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">dashboard</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">shortcuts</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">totalCount</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #B48EAD">4</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">nodes</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:[{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">SSC_kgDOAA2otQ</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">mheap: Open Issues</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">state:open archived:false sort:updated-desc user:mheap -author:app/dependabot -repo:mheap/michaelheap.com is:issue </span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">icon</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">BOOKMARK</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">color</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">GRAY</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">description</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">""</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">scopingRepository</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #81A1C1">null</span><span style="color: #ECEFF4">},{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">SSC_kgDOABdOWA</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Kong: My Open PRs</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">is:pr author:@me state:open org:Kong -draft:true</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">icon</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">BOOKMARK</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">color</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">GRAY</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">description</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">""</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">scopingRepository</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #81A1C1">null</span><span style="color: #ECEFF4">},{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">SSC_kgDOABdOZw</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Kong: Awaiting Review from Me</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">is:pr state:open org:Kong review-requested:@me -repo:Kong/platform-api </span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">icon</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">BOOKMARK</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">color</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">GRAY</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">description</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">""</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">scopingRepository</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #81A1C1">null</span><span style="color: #ECEFF4">},{</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">SSC_kgDOABdOaA</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">mheap: Awaiting Review from Me</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">state:open archived:false sort:updated-desc user:mheap -author:app/dependabot -repo:mheap/michaelheap.com is:pr </span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">icon</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">BOOKMARK</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">color</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">GRAY</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">description</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #ECEFF4">""</span><span style="color: #ECEFF4">,</span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">scopingRepository</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #81A1C1">null</span><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9">}</span></div></code></div></pre>
<p>Search for &quot;name: Kong: My Open PRs&quot; and extact the <code>id</code> field from that value</p>
</div>
</div>
<p>This worked, but was inefficient. Codex walked every node in the tree looking for IDs.</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>You don't have to walk the tree. The data is always in data.createDashboardSearchShortcut.dashboard.shortcuts.nodes</p>
</div>
</div>
<p>Success! Now let's add some new capabilities</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Now group the searches by <code>section</code>. For each section, create an empty query with the title &quot;== $SECTION ==&quot; above the list of items</p>
</div>
</div>
<p>This worked, but didn't feel right to me.</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Actually, I changed my mind. Don't use <code>section</code> in a config. Instead, sections will be defined explicitly e.g.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">searches</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">section</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Demo</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">SSC_kgDOAB5MiA</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Test 1</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">state:open archived:false assignee:@me sort:updated-desc org:Kong</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">""</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">remove</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">SSC_kgDOAB5MiQ</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Work from Alice</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">""</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">recent-work</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">time</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">30d</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">user</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">alice.jones</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">remove</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span></div><div class="line"><span style="color: #8FBCBB">templates</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">recent-work</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">org:Kong ((assignee:{{ user }} AND is:issue) OR (is:pr AND author:{{ user }})) (is:open OR updated:&gt;@today-{{ default(time, "30d") }}) sort:updated-desc</span></div></code></div></pre>
</div>
</div>
<p>Much better! But my section configs were being given a <code>query</code> entry unexpectedly in the updated config like this:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">SSC_kgDOAB5MiA</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">section</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Demo</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">""</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">""</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">""</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">remove</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span></div></code></div></pre>
<p>I wanted it to look like this:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">SSC_kgDOAB5MiA</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">section</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Demo</span></div></code></div></pre>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Do not replace section configs with a full query object. It should contain only <code>id</code> and <code>section</code></p>
</div>
</div>
<p>Everything looked good at this point. I wanted to try recreating everything so that the order in my config file is respected (right now, the section is the last item as it's the most recently created)</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Add a --force flag to the CLI that deletes all existing configs and recreates them, even if an ID is set</p>
</div>
</div>
<p>It worked, but I realised that the flag name should really be <code>--recreate</code>.</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Rename the force flag to --recreate</p>
</div>
</div>
<p>How about use cases where I want to remove all saved searches without recreating them?</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Also add a --reset flag that removes all entries without creating anything</p>
</div>
</div>
<p>At this point, the CLI was finished. It can create saved searches, with section markers and recreate all saved searches to enforce ordering.</p>
<p>But I wasn't done. I had an idea for another template, but needed a way to provide multiple repos as a list:</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Add support for the join function in templates. Example usage:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Terraform PRs</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">template</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">repo-prs</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vars</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">repos</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">repo:Kong/terraform-provider-konnect</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">repo:Kong/terraform-provider-konnect-beta</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">repo:Kong/terraform-provider-kong-gateway</span></div></code></div></pre>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">repo-prs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">query</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">is:pr state:open ({{ join(repos, "OR") }}) -author:app/renovate draft:false {{ additional }}</span></div></code></div></pre>
</div>
</div>
<p>Uh oh! There's a bug. I used <code>{{ additional }}</code> in the template to add additional filters using a free-form string. If there are no additional parameters, I expected it to resolve to an empty string (meaning &quot;no additional filters&quot;).</p>
<p>However, if a config didn't specify an empty string we got an error. I pasted the error message into Codex to try and fix it:</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>template: query:1: function &quot;additional&quot; not defined - can the template code treat missing values as null?</p>
</div>
</div>
<p>All working as expected, but how do people know how to use it?</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Produce a <a href="http://readme.md/">README.md</a> teaching people how to use this tool</p>
</div>
</div>
<p>The README mentions a <code>--config</code> flag, but it doesn't exist. Sounds like a good idea to me</p>
<div class="ai-input">
<div class="ai-header">
📝 Prompt
</div>
<div class="ai-body">
<p>Implement the --config flag like the README indicates</p>
</div>
</div>
<p>At this point, I had something I wanted to ship 🎉</p>
<h3 id="reflection" tabindex="-1">Reflection</h3>
<p>Honestly, I had a great time building this tool. I'm not sure it was faster than writing it by hand, but I did it in the background while doing something else and it ended up saving me a ton of time. If I had to write it by hand, I probably wouldn't have finished this project.</p>
<p>I <em>did</em> do a little bit extra by hand after the session above. I added a GitHub Action to build and release the extension, which means you can run it with <code>gh extension install https://github.com/mheap/gh-saved-issues</code>.</p>
</content>
</entry>
<entry>
<title>Display mirroring with Hammerspoon</title>
<link href="https://michaelheap.com/toggle-mirroring-hammerspoon/"/>
<updated>2025-12-08T13:59:48Z</updated>
<id>https://michaelheap.com/toggle-mirroring-hammerspoon/</id>
<content type="html"><p>I need to mirror my screens periodically - enough that it's painful to go in to system settings each time and configure mirroring.</p>
<p>Here's a Hammerspoon snippet that will toggle mirroring. It works because <code>allScreens</code> returns a single value if your second screen is mirroring the first.</p>
<p><code>C34H89x</code> and <code>LC49G95T</code> are my screen names, which I looked up in system settings.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">lua</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">hs.</span><span style="color: #D8DEE9">hotkey</span><span style="color: #D8DEE9FF">.</span><span style="color: #88C0D0">bind</span><span style="color: #D8DEE9FF">({ </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">cmd</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">, </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">alt</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">, </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">ctrl</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">, </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">shift</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> }, </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">M</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">, </span><span style="color: #81A1C1">function</span><span style="color: #ECEFF4">()</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">local</span><span style="color: #D8DEE9FF"> screens </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> hs.</span><span style="color: #D8DEE9">screen</span><span style="color: #D8DEE9FF">.</span><span style="color: #88C0D0">allScreens</span><span style="color: #D8DEE9FF">()</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">#</span><span style="color: #D8DEE9FF">screens </span><span style="color: #81A1C1">==</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">then</span></div><div class="line"><span style="color: #D8DEE9FF"> screens[</span><span style="color: #B48EAD">1</span><span style="color: #D8DEE9FF">]:</span><span style="color: #88C0D0">mirrorStop</span><span style="color: #D8DEE9FF">()</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">else</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">local</span><span style="color: #D8DEE9FF"> screenToMirror </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> hs.</span><span style="color: #D8DEE9">screen</span><span style="color: #D8DEE9FF">.</span><span style="color: #88C0D0">find</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">C34H89x</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">)</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">local</span><span style="color: #D8DEE9FF"> screenToHide </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> hs.</span><span style="color: #D8DEE9">screen</span><span style="color: #D8DEE9FF">.</span><span style="color: #88C0D0">find</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">LC49G95T</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">)</span></div><div class="line"><span style="color: #D8DEE9FF"> screenToHide:</span><span style="color: #88C0D0">mirrorOf</span><span style="color: #D8DEE9FF">(screenToMirror)</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">end</span></div><div class="line"><span style="color: #81A1C1">end</span><span style="color: #D8DEE9FF">)</span></div></code></div></pre>
</content>
</entry>
<entry>
<title>Automated NPM secret rotation in GitHub Actions</title>
<link href="https://michaelheap.com/rotate-all-npm-tokens-github-actions/"/>
<updated>2025-11-16T19:49:03Z</updated>
<id>https://michaelheap.com/rotate-all-npm-tokens-github-actions/</id>
<content type="html"><p>NPM recently announced that all long-lived tokens are being revoked, and that going forwards any new tokens may be valid for a maximum of 90 days.</p>
<p>This presents a challenge for me, as I've built a system where when I tag a release on GitHub in my JavaScript projects, it builds and publishes to NPM using an access token. Rotating those tokens regularly would be a lot of toil.</p>
<h2 id="upgrading-to-oidc" tabindex="-1">Upgrading to OIDC</h2>
<p>The correct way to solve this is to adopt <a href="https://docs.npmjs.com/trusted-publishers#for-github-actions">trusted publishing (OIDC)</a>. However, I don't have time to update all of my projects right now. I'd like to keep using an access token until I have time to update each project.</p>
<h2 id="rotating-github-actions-user-secrets-at-scale" tabindex="-1">Rotating GitHub Actions user secrets at scale</h2>
<p>Fortunately, back in 2020 I was feeling the pain of rotating GitHub secrets for a user account as I couldn't use organization level secrets and built <a href="https://github.com/mheap/github-update-secret">github-update-secret</a>.</p>
<p><code>github-update-secret</code> iterates over all of your repositories and checks if the provided secret name is set. If so, it updates the value to the new value provided.</p>
<h3 id="how-it-works" tabindex="-1">How it works</h3>
<ul>
<li>Provide GitHub authentication by populating the <code>GITHUB_TOKEN</code> environment variable, or passing the <code>--pat</code> flag to the CLI</li>
<li>Fetch a list of all repos to which the authenticated user has admin access for the provided user/org</li>
<li>Fetch the list of secrets on each repository</li>
<li>For each repository that has a secret named the same as the provided <code>SECRET_NAME</code>, update the value of that secret</li>
<li>Check if the provided target is an organisation.</li>
<li>If so:
<ul>
<li>Check if there is an org secret named <code>SECRET_NAME</code></li>
<li>If there is, update the value</li>
</ul>
</li>
</ul>
<h3 id="example" tabindex="-1">Example</h3>
<p>I just rotated all of my <code>NPM_TOKEN</code> secrets with a secret that's valid for another 90 days using the following command:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">GITHUB_TOKEN=ghp_... DEBUG=github-update-secret npx github-update-secret </span><span style="color: #81A1C1">&lt;</span><span style="color: #D8DEE9FF">user/org</span><span style="color: #81A1C1">&gt;</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">&lt;</span><span style="color: #D8DEE9FF">SECRET_NAME</span><span style="color: #81A1C1">&gt;</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">&lt;</span><span style="color: #D8DEE9FF">new_value</span><span style="color: #81A1C1">&gt;</span></div></code></div></pre>
<p>Running the tool with <code>DEBUG=github-update-secret</code> allows you to see all of the repositories that are being updated:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">❯ DEBUG=github-update-secret npx github-update-secret mheap NPM_TOKEN npm_...</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Fetching repo list +0ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Processed repo list +2s</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Building list of repos using </span><span style="color: #ECEFF4">[</span><span style="color: #D8DEE9FF">npm_token</span><span style="color: #ECEFF4">]</span><span style="color: #D8DEE9FF"> +0ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Found </span><span style="color: #ECEFF4">[</span><span style="color: #D8DEE9FF">27</span><span style="color: #ECEFF4">]</span><span style="color: #D8DEE9FF"> repos with the secret </span><span style="color: #ECEFF4">[</span><span style="color: #D8DEE9FF">npm_token</span><span style="color: #ECEFF4">]</span><span style="color: #D8DEE9FF"> +2s</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updating action-guard +1ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updated action-guard +347ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updating action-router +1ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updated action-router +381ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updating action-run +1ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updated action-run +338ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Updating actions-output-wrapper +0ms</span></div><div class="line"><span style="color: #D8DEE9FF">...snip...</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret Check </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> provided user is an org +0ms</span></div><div class="line"><span style="color: #D8DEE9FF"> github-update-secret User is not an org. Skipping org secret update +142ms</span></div></code></div></pre>
<p>NPM’s move to 90-day tokens is good for security but rough on existing workflows. OIDC is the long-term fix, but until I can update each project, <code>github-update-secret</code> gives me a quick way to rotate tokens across all my repos and keep releases moving. It’s not perfect, but it provides enough breathing room until I can do the real migration.</p>
</content>
</entry>
<entry>
<title>JSON Semantic Diff</title>
<link href="https://michaelheap.com/json-semantic-diff/"/>
<updated>2025-11-16T18:19:42Z</updated>
<id>https://michaelheap.com/json-semantic-diff/</id>
<content type="html"><p><a href="https://github.com/geofffranks/spruce">Spruce</a> is a general purpose YAML &amp; JSON merging tool, and one of it's subcommands is a semantic JSON diff. It captures added and removed fields, and highlights any type changes.</p>
<h2 id="installing-spruce" tabindex="-1">Installing Spruce</h2>
<p>Spruce is written in Go, and can be installed in many different ways:</p>
<ol>
<li>
<p>With <code>mise</code> (which I highly recommend):</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">mise use -g github:geofffranks/spruce</span></div></code></div></pre>
</li>
<li>
<p>With <code>brew</code> if you're on MacOS</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">brew install starkandwayne/cf/spruce</span></div></code></div></pre>
</li>
<li>
<p>With <code>go get</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">go get github.com/geofffranks/spruce</span></div></code></div></pre>
</li>
</ol>
<h2 id="example-usage" tabindex="-1">Example usage</h2>
<p>Here are two sample JSON files. Between the first and the second, I've changed a value, changed two data types, and removed two fields:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Alice</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">age</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">30</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">pets</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Fido</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">favourite_cake</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Carrot cake</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">another</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">value</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">here</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Alice Smith</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">age</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">30</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">pets</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Fido</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">],</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">another</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{}</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>If we run <code>spruce diff</code>, this information is returned to us in a format that makes is <em>very</em> easy to see the diffs:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">diff</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">$ spruce diff first.json second.json</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF">(root level)</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> one map entry removed:</span></div><div class="line"><span style="color: #D8DEE9FF">favourite_cake: "Carrot cake"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF">age</span></div><div class="line"><span style="color: #D8DEE9FF">± type change from string to int</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> 30</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> 30</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF">another</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> one map entry removed:</span></div><div class="line"><span style="color: #D8DEE9FF">value: here</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF">name</span></div><div class="line"><span style="color: #D8DEE9FF">± value change</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> Alice</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> Alice Smith</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF">pets</span></div><div class="line"><span style="color: #D8DEE9FF">± type change from string to list</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> Fido</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> - Fido</span></div></code></div></pre>
</content>
</entry>
<entry>
<title>Say Three Things</title>
<link href="https://michaelheap.com/say-three-things/"/>
<updated>2025-11-06T21:03:56Z</updated>
<id>https://michaelheap.com/say-three-things/</id>
<content type="html"><p>We’ve all been in <em>that</em> meeting, with a dozen people sat on Zoom, not quite sure why they’re there. The more jaded of the group might have their camera off, trying to do some work while they listen in. Others might keep their camera on, trying to stay focused on what’s being said.</p>
<p>No matter if your camera is on or off, if you’re not participating in the meeting why are you there at all? <a href="https://knowyourmeme.com/photos/1753854-shiba-inu-barking-in-an-office-meeting">This meeting could have been an email</a></p>
<p>Except, it couldn’t. People <em>are</em> interacting on the call. There’s a lot of nuance in the discussion that would be missed in a summary. You need to be there to hear it. So how do you make the most of your time?</p>
<p>Remind people that you exist. Say at least three things in the meeting. It doesn’t matter if you’re meeting with your CEO or your newest, most junior hire. Speaking up reminds people that you’re there.</p>
<p>If you stay silent, they’re likely to forget you were even there. Or worse, assume that you don’t understand what’s going on around you. By speaking up, you can paint a picture of how you want to be perceived.</p>
<h2 id="what-three-things-should-i-say%3F" tabindex="-1">What Three Things Should I Say?</h2>
<p>“But I don’t have anything to add!” I hear you say. Wonderful! Neither do a lot of people that can’t stop talking in meetings (myself included). Here are a couple of my favourite ways to inject myself into the conversation.</p>
<h3 id="1.-clarify" tabindex="-1">1. Clarify</h3>
<p>“<em>Can I try to summarise to make sure that I’m following? I heard…</em>”</p>
<p>This is a great way to illustrate that you understand what’s going on. There’s always someone on the call that was distracted by a Slack message, a knock at the door, or a butterfly and they're very appreciative that you just summarised the last 5 minutes of conversation.</p>
<h3 id="2.-capture" tabindex="-1">2. Capture</h3>
<p>“<em>So the next steps are for Alice and Bob to submit a proposal for how to proceed. I’ve captured that in the meeting notes. Is there a specific date we need that by?</em>”</p>
<p>Being the note taker is a thankless task, but it also gives you the opportunity to speak. It also helps build a reputation as someone that helps drive progress.</p>
<h3 id="3.-connect" tabindex="-1">3. Connect</h3>
<p>“<em>Has anyone spoken to Charlie about this? I can see some parallels with Project Flubjam</em>”</p>
<p>This one takes some organisational context, but being the person to connect the dots between orgs is a huge differentiator</p>
<h3 id="4.-chat" tabindex="-1">4. Chat</h3>
<p>And if all else fails, join the meeting early. The few unscripted minutes before a call are often the most human ones. Ask about someone’s weekend or joke about the weather - people remember small talk more than polished summaries.</p>
<h2 id="why-it-matters" tabindex="-1">Why It Matters</h2>
<p>Saying three things in every meeting isn’t about filling airtime or proving your worth. It’s about showing up with intention.</p>
<p>Your aim isn’t to be the person that speaks for 25 minutes of a 30 minute meeting. Prioritise clarity and brevity - 15-30 seconds of talking per point is enough.</p>
<p>You’re signalling that you’re engaged, thoughtful, and invested in the outcome. Whether you’re clarifying, summarising, or connecting dots, those small contributions compound over time into credibility and visibility.</p>
<p>Silence might feel safe, but it also makes you invisible. Speaking up, even briefly, ensures that when decisions are made or opportunities arise, people remember that you were part of the conversation rather than being just another name on the invite list.</p>
</content>
</entry>
<entry>
<title>Run a container on a schedule with ECS</title>
<link href="https://michaelheap.com/ecs-scheduled-container/"/>
<updated>2025-06-02T08:00:13Z</updated>
<id>https://michaelheap.com/ecs-scheduled-container/</id>
<content type="html"><p>I've got a Docker container that I want to run periodically to fetch data and store it in a database. As this is something that needs to run persistently, I'm using Terraform to manage the infrastructure. It took me a while to figure out all the required resources and permissions, so I thought I'd share my solution here.</p>
<p>To securely run a container on a schedule in ECS, I needed to:</p>
<ul>
<li>Initialise Terraform</li>
<li>Configure any secrets that the container needs</li>
<li>Create an ECS Fargate cluster, and a VPC for the container to run in</li>
<li>Create an IAM role for the container to execute as</li>
<li>Create an IAM role for EventBridge to trigger the ECS task</li>
<li>Define an EventBridge schedule trigger</li>
<li>Define a task</li>
</ul>
<h2 id="initialise-terraform" tabindex="-1">Initialise Terraform</h2>
<p>Create <code>providers.tf</code> containing the AWS profile and region you want to use:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">provider</span><span style="color: #A3BE8C"> "aws"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">region</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"us-east-2"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">profile</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"default"</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<p>Then run <code>terraform init</code>.</p>
<h2 id="secrets" tabindex="-1">Secrets</h2>
<p>The container that I'm running needs access to some secret values as environment variables. I read the two most sensitive items from <code>tfvars</code>, and hard code the role as the Terraform repo is private anyway. The <code>test/area/role</code> secret is really just configuration rather than being a secret:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">variable</span><span style="color: #A3BE8C"> "AREA_USERNAME"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">description</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"Username for AREA access"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">type</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> string</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">sensitive</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">true</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">variable</span><span style="color: #A3BE8C"> "AREA_PASSWORD"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">description</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"Password for AREA access"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">type</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> string</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">sensitive</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">true</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">locals</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">my_secrets</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> "test/area/username" = var.AREA_USERNAME</span></div><div class="line"><span style="color: #D8DEE9FF"> "test/area/password" = var.AREA_PASSWORD</span></div><div class="line"><span style="color: #D8DEE9FF"> "test/area/role" = "some_role"</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_secretsmanager_secret" "my_secrets"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">for_each</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> local</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> each</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">key</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_secretsmanager_secret_version" "my_secret_values"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">for_each</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> local</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">secret_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_secretsmanager_secret</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets[each</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">key].id</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">secret_string</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> each</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">value</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<h2 id="ecs-cluster-and-vpc" tabindex="-1">ECS Cluster and VPC</h2>
<p>ECS allows you to scale to zero, but you still need a VPC to spin up containers in. My containers need internet access, so here's how I create an ECS cluster and VPC with internet access:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Create an ECS cluster</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_ecs_cluster" "my_cluster"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"demo-fargate-cluster"</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Create a VPC and network, allowing all egress traffic</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_vpc" "main"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">cidr_block</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"10.0.0.0/16"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">enable_dns_hostnames</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">true</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_subnet" "public"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">vpc_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_vpc</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">main</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">cidr_block</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"10.0.1.0/24"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">availability_zone</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"us-east-2a"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">map_public_ip_on_launch</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">true</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># You could also use a NAT gateway instead. We use an internet gateway</span></div><div class="line"><span style="color: #616E88"># due to speed / cost reasons in this example</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_internet_gateway" "gw"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">vpc_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_vpc</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">main</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_route_table" "public"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">vpc_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_vpc</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">main</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">route</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">cidr_block</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"0.0.0.0/0"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">gateway_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_internet_gateway</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">gw</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_route_table_association" "public_assoc"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">subnet_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_subnet</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">public</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">route_table_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_route_table</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">public</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_security_group" "ecs_tasks"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ecs-scheduled-tasks-sg"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">description</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"Allow outbound traffic"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">vpc_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_vpc</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">main</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">egress</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">from_port</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">0</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">to_port</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">0</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">protocol</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"-1"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">cidr_blocks</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> [</span><span style="color: #A3BE8C">"0.0.0.0/0"</span><span style="color: #D8DEE9FF">]</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<h2 id="execution-iam-role" tabindex="-1">Execution IAM role</h2>
<p>The container that runs does not have access to AWS Secrets Manager by default. To allow access, I create a new IAM role that the container will assume using <code>sts:AssumeRole</code> before running. This new role has a policy attached that allows access to specific secrets in AWS Secrets Manager.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Allow ECS to assume this role</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_role" "ecs_task_execution"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ecs-task-execution-role"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">assume_role_policy</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode({</span></div><div class="line"><span style="color: #D8DEE9FF"> Version = "2012-10-17",</span></div><div class="line"><span style="color: #D8DEE9FF"> Statement = [{</span></div><div class="line"><span style="color: #D8DEE9FF"> Effect = "Allow",</span></div><div class="line"><span style="color: #D8DEE9FF"> Principal = {</span></div><div class="line"><span style="color: #D8DEE9FF"> Service = "ecs-tasks.amazonaws.com"</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Action</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"sts:AssumeRole"</span></div><div class="line"><span style="color: #D8DEE9FF"> }]</span></div><div class="line"><span style="color: #D8DEE9FF"> })</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Create a policy that allows access to the secrets we defined earlier</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_policy" "ecs_execution_secrets_access"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ecs-exec-secrets-access"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">policy</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode({</span></div><div class="line"><span style="color: #D8DEE9FF"> Version = "2012-10-17",</span></div><div class="line"><span style="color: #D8DEE9FF"> Statement = [{</span></div><div class="line"><span style="color: #D8DEE9FF"> Effect = "Allow",</span></div><div class="line"><span style="color: #D8DEE9FF"> Action = [</span></div><div class="line"><span style="color: #D8DEE9FF"> "secretsmanager:GetSecretValue",</span></div><div class="line"><span style="color: #D8DEE9FF"> "secretsmanager:DescribeSecret"</span></div><div class="line"><span style="color: #D8DEE9FF"> ],</span></div><div class="line"><span style="color: #D8DEE9FF"> Resource = [</span></div><div class="line"><span style="color: #D8DEE9FF"> aws_secretsmanager_secret.my_secrets["test/area/username"].arn,</span></div><div class="line"><span style="color: #D8DEE9FF"> aws_secretsmanager_secret.my_secrets["test/area/password"].arn,</span></div><div class="line"><span style="color: #D8DEE9FF"> aws_secretsmanager_secret.my_secrets["test/area/role"].arn,</span></div><div class="line"><span style="color: #D8DEE9FF"> ]</span></div><div class="line"><span style="color: #D8DEE9FF"> }]</span></div><div class="line"><span style="color: #D8DEE9FF"> })</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Attach the above policy to the IAM role</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_role_policy_attachment" "ecs_exec_secrets_access_attach"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">role</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_task_execution</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">name</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">policy_arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_policy</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_execution_secrets_access</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Attach the ECS Task execution policy</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_role_policy_attachment" "ecs_task_execution_attach"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">role</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_task_execution</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">name</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">policy_arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div></code></div></pre>
<h2 id="trigger-iam-role" tabindex="-1">Trigger IAM role</h2>
<p>Amazon EventBridge also needs an IAM role in order to trigger the ECS task. Here's an IAM role that has permission to run the defined tasks explicitly.</p>
<p>It took me a <em>long</em> time to realise that not only did I need to provide a <code>runTask</code> permission to the tasks, I also needed to allow <code>runTask</code> to the ECS cluster that was being used.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Allow EventBridge to assume this role</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_role" "eventbridge_invoke_ecs"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"eventbridge-ecs-invoke-role"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">assume_role_policy</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode({</span></div><div class="line"><span style="color: #D8DEE9FF"> Version = "2012-10-17",</span></div><div class="line"><span style="color: #D8DEE9FF"> Statement = [{</span></div><div class="line"><span style="color: #D8DEE9FF"> Effect = "Allow",</span></div><div class="line"><span style="color: #D8DEE9FF"> Principal = {</span></div><div class="line"><span style="color: #D8DEE9FF"> Service = "events.amazonaws.com"</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Action</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"sts:AssumeRole"</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> ]</span></div><div class="line"><span style="color: #D8DEE9FF"> })</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Create a new IAM policy that allows us to run all of the defined tasks</span></div><div class="line"><span style="color: #616E88"># We need to explicitly allow ecs:RunTask for the ecs:cluster too</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_iam_role_policy" "eventbridge_ecs_policy"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"invoke-ecs"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">role</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">eventbridge_invoke_ecs</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">policy</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode({</span></div><div class="line"><span style="color: #D8DEE9FF"> Version = "2012-10-17",</span></div><div class="line"><span style="color: #D8DEE9FF"> Statement = [</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> Effect = "Allow",</span></div><div class="line"><span style="color: #D8DEE9FF"> Action = "ecs:RunTask",</span></div><div class="line"><span style="color: #D8DEE9FF"> Resource = aws_ecs_task_definition.scheduled_my_command.arn</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Effect</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"Allow"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Action</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"iam:PassRole"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Resource</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_task_execution</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Effect</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"Allow"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Action</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ecs:RunTask"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Resource</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"*"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Condition</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ArnEquals"</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">:</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> "ecs:cluster" : aws_ecs_cluster.my_cluster.arn</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> ]</span></div><div class="line"><span style="color: #D8DEE9FF"> })</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div></code></div></pre>
<h2 id="eventbridge-schedule-trigger" tabindex="-1">EventBridge schedule trigger</h2>
<p>We need to define an event rule that runs the container on a schedule. I also create a Cloudwatch log group to capture task logs.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Run the task at 23:59 every night</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_cloudwatch_event_rule" "run_at_23_59"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"run-ecs-task-schedule"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">schedule_expression</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"cron(59 23 * * ? *)"</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># And create a Cloudwatch log group to send logs to</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_cloudwatch_log_group" "my_scheduled_task"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"/ecs/my-scheduled-task-logs"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">retention_in_days</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<h2 id="task-definition" tabindex="-1">Task Definition</h2>
<p>Finally, we need to define the task to run. You'll need to upload the docker image to ECR before defining a task. You'll also need to specify all of the secrets that the container needs access to.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Define the container and command to run, plus CPU/Memory usage</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_ecs_task_definition" "scheduled_my_command"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">family</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"scheduled-my-command"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">requires_compatibilities</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> [</span><span style="color: #A3BE8C">"FARGATE"</span><span style="color: #D8DEE9FF">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">network_mode</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"awsvpc"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">cpu</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"256"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">memory</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"512"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">execution_role_arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_task_execution</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">container_definitions</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode([</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> name = </span><span style="color: #A3BE8C">"demo"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> image = </span><span style="color: #A3BE8C">"hello-world"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> essential = </span><span style="color: #B48EAD">true</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> secrets = [</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> name = </span><span style="color: #A3BE8C">"AREA_USER"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> valueFrom = aws_secretsmanager_secret</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets[</span><span style="color: #A3BE8C">"test/area/username"</span><span style="color: #D8DEE9FF">].arn</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"AREA_PASSWORD"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">valueFrom</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_secretsmanager_secret</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets[</span><span style="color: #A3BE8C">"test/area/password"</span><span style="color: #D8DEE9FF">].arn</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"AREA_ROLE"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">valueFrom</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_secretsmanager_secret</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_secrets[</span><span style="color: #A3BE8C">"test/area/role"</span><span style="color: #D8DEE9FF">].arn</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> ],</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">logConfiguration</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> logDriver = "awslogs",</span></div><div class="line"><span style="color: #D8DEE9FF"> options = {</span></div><div class="line"><span style="color: #D8DEE9FF"> awslogs-group = "/ecs/my-scheduled-task-logs"</span></div><div class="line"><span style="color: #D8DEE9FF"> awslogs-region = "us-east-2"</span></div><div class="line"><span style="color: #D8DEE9FF"> awslogs-stream-prefix = "demo"</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> ])</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Trigger this task using the scheduled event</span></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_cloudwatch_event_target" "ecs_my_command_target"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">rule</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_cloudwatch_event_rule</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">run_at_23_59</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">name</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">role_arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_iam_role</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">eventbridge_invoke_ecs</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">target_id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"ecs-task-my-command"</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_ecs_cluster</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">my_cluster</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">ecs_target</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">task_definition_arn</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_ecs_task_definition</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">scheduled_my_command</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">launch_type</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"FARGATE"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">network_configuration</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">subnets</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> [aws_subnet</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">public</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">security_groups</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> [aws_security_group</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">ecs_tasks</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">assign_public_ip</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">true</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># This is only needed for debugging. See the final section</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># dead_letter_config {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># arn = aws_sqs_queue.failed_invocations.arn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># }</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<h2 id="help%2C-it's-not-working!" tabindex="-1">Help, it's not working!</h2>
<p>Last, but not least, debugging! If your task isn't triggering as expected, you can configure a DeadLetterQueue (DLQ) to receieve the error messages from AWS. If you choose to do this, uncomment the <code>dead_letter_config</code> section of the task definition above.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">hcl</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_sqs_queue" "failed_invocations"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">name</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"eventbridge-ecs-dlq"</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">resource</span><span style="color: #A3BE8C"> "aws_sqs_queue_policy" "allow_eventbridge"</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">queue_url</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_sqs_queue</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">failed_invocations</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">id</span></div><div class="line"></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">policy</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> jsonencode({</span></div><div class="line"><span style="color: #D8DEE9FF"> Version = "2012-10-17",</span></div><div class="line"><span style="color: #D8DEE9FF"> Statement = [</span></div><div class="line"><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> Sid = "AllowEventBridgeToSendMessages",</span></div><div class="line"><span style="color: #D8DEE9FF"> Effect = "Allow",</span></div><div class="line"><span style="color: #D8DEE9FF"> Principal = {</span></div><div class="line"><span style="color: #D8DEE9FF"> Service = "events.amazonaws.com"</span></div><div class="line"><span style="color: #D8DEE9FF"> },</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Action</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">"sqs:SendMessage"</span><span style="color: #D8DEE9FF">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Resource</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> aws_sqs_queue</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">failed_invocations</span><span style="color: #B48EAD">.</span><span style="color: #D8DEE9FF">arn,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">Condition</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> {</span></div><div class="line"><span style="color: #D8DEE9FF"> ArnEquals = {</span></div><div class="line"><span style="color: #D8DEE9FF"> "aws:SourceArn" = aws_cloudwatch_event_rule.run_at_23_59.arn</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> }</span></div><div class="line"><span style="color: #D8DEE9FF"> ]</span></div><div class="line"><span style="color: #D8DEE9FF"> })</span></div><div class="line"><span style="color: #D8DEE9FF">}</span></div></code></div></pre>
<p>To read messages from the dead letter queue, use the <code>aws</code> CLI tool:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #616E88"># Fetch the queue URL</span></div><div class="line"><span style="color: #D8DEE9FF">aws sqs get-queue-url --queue-name eventbridge-ecs-dlq</span></div><div class="line"></div><div class="line"><span style="color: #616E88"># Read messages - change the URL for your ID</span></div><div class="line"><span style="color: #D8DEE9FF">aws sqs receive-message \</span></div><div class="line"><span style="color: #D8DEE9FF"> --queue-url https://sqs.us-east-2.amazonaws.com/111111111111/eventbridge-ecs-dlq \</span></div><div class="line"><span style="color: #D8DEE9FF"> --max-number-of-messages 10 \</span></div><div class="line"><span style="color: #D8DEE9FF"> --wait-time-seconds 5 \</span></div><div class="line"><span style="color: #D8DEE9FF"> --message-attribute-names All \</span></div><div class="line"><span style="color: #D8DEE9FF"> --attribute-names All</span></div></code></div></pre>
<h2 id="conclusion" tabindex="-1">Conclusion</h2>
<p>The parts of this that gave me the most trouble were figuring out how to debug using <code>dead_letter_queue</code> and that the IAM policy for <code>runTask</code> needed access to the cluster too.</p>
<p>Hopefully this has helped you (or will help me again in the future) to deploy scheduled tasks on ECS.</p>
</content>
</entry>
<entry>
<title>Pin your GitHub Actions</title>
<link href="https://michaelheap.com/pin-your-github-actions/"/>
<updated>2025-03-15T20:19:27Z</updated>
<id>https://michaelheap.com/pin-your-github-actions/</id>
<content type="html"><p>Way back in 2019, Julien Renaux published <a href="https://julienrenaux.fr/2019/12/20/github-actions-security-risk/">Use GitHub Actions at your own risk</a>. While the title is a little sensational, it correctly pointed out that any maintainer can update a branch or tag to point at new code without you knowing. This means that if any action is compromised, you'll start leaking secrets without knowing it.</p>
<p>Today, <code>tj-actions/changed-files</code>, a widely used GitHub Action <a href="https://www.stepsecurity.io/blog/harden-runner-detection-tj-actions-changed-files-action-is-compromised">was compromised</a> and started leaking secrets.</p>
<p>Hopefully this was the wakeup call the industry needed to start paying attention to supply chain security.</p>
<h2 id="solving-the-problem" tabindex="-1">Solving the problem</h2>
<p>Security is always a trade-off. You can solve the supply chain problem by specifying a full length commit SHA, but fetching that SHA for every action is a painful process. Then, whenever you want to upgrade your action you have to do it all again.</p>
<p>Thankfully, there are tools and automations available to solve this problem. You can be secure <em>and</em> feel minimal pain thanks to these projects:</p>
<ol>
<li><a href="https://github.com/mheap/pin-github-action">pin-github-action</a> - This is one of my projects. It takes a directory of workflows and uses the GitHub API to convert tag and branch references to a full length commit SHA.</li>
<li><a href="https://github.com/zgosalvez/github-actions-ensure-sha-pinned-actions">github-actions-ensure-sha-pinned-actions</a> - A community action that will fail the build if it detects any unpinned actions being used in the current repository.</li>
<li><a href="https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot">Dependabot</a> / <a href="https://docs.renovatebot.com/modules/manager/github-actions/">Renovate</a> - Dependency management systems that submits PRs to upgrade GitHub Actions. They both natively understand the <code># &lt;ref&gt;</code> comments in workflow files.</li>
</ol>
<h3 id="pin-to-a-sha" tabindex="-1">Pin to a SHA</h3>
<p>The first thing to do is update all of your existing workflows to use the long commit SHA using <code>pin-github-action</code>.</p>
<p>You can run it with <code>npx</code> or <code>docker</code>. If you're not sure which to use, use <code>docker</code> through the following alias:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #88C0D0">alias</span><span style="color: #D8DEE9FF"> pin-github-action=</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">docker run --rm -v </span><span style="color: #ECEFF4">$(</span><span style="color: #A3BE8C">pwd</span><span style="color: #ECEFF4">)</span><span style="color: #A3BE8C">:/workflows -e GITHUB_TOKEN mheap/pin-github-action</span><span style="color: #ECEFF4">"</span></div></code></div></pre>
<p>If you're working with a large number of workflows, or any private actions you'll need to set the <code>GITHUB_TOKEN</code> environment variable to prevent rate limiting or provide valid access credentials to a repository:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">export</span><span style="color: #D8DEE9FF"> GITHUB_TOKEN=</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">ghp_YOUR_TOKEN_HERE</span><span style="color: #ECEFF4">"</span></div></code></div></pre>
<p>Finally, change directory to your repository and run <code>pin-github-action</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #88C0D0">cd</span><span style="color: #D8DEE9FF"> my-repo</span></div><div class="line"><span style="color: #D8DEE9FF">pin-github-action .github/workflows</span></div></code></div></pre>
<p>If you run <code>git diff .github/workflows</code> you'll see that all of your actions have been updated to the long commit SHA:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">diff</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF"> steps:</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> - uses: actions/checkout@v4</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> - uses: MyOrg/some-action@v1</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> - uses: MyOrg/some-action@25ed13d0628a1601b4b44048e63cc4328ed03633 # v1</span></div></code></div></pre>
<p>I recommend pinning all actions to a SHA, but this may not be feasible for some companies that use internal actions. If you want to trust internal actions, you can pass the <code>--allow</code> flag to <code>pin-github-action</code> to add a specific prefix to an allowlist:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">pin-github-action --allow </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">MyOrg/*</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> .github/workflows</span></div></code></div></pre>
<p>This will ignore any actions with the prefix <code>MyOrg</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">diff</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF"> steps:</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> - uses: actions/checkout@v4</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4</span></div><div class="line"><span style="color: #D8DEE9FF"> - uses: MyOrg/some-action@v1</span></div></code></div></pre>
<h3 id="prevent-regressions" tabindex="-1">Prevent regressions</h3>
<p>Pinning all actions to a specific SHA solves the problem today, but it doesn't guarantee that a new action won't be added in the future without using a SHA. To prevent that happening, <a href="https://github.com/zgosalvez/github-actions-ensure-sha-pinned-actions">github-actions-ensure-sha-pinned-actions</a> can be used to fail the build when any unpinned actions are detected.</p>
<p>The README for the action contains examples, but if you're using <code>pin-github-action</code> you can automatically add a new workflow using the <code>--enforce</code> flag. The <code>--enforce</code> flag writes a workflow containing <code>github-actions-ensure-sha-pinned-actions</code> to the path provided, including adding any actions passed in <code>--allow</code> to the <code>allowlist</code> input for the action.</p>
<p>The following command will create a workflow at <code>.github/workflows/security.yaml</code> that ensures all actions are using the long commit SHA, <em>unless</em> they are actions from <code>MyOrg</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">pin-github-action --allow </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">MyOrg/*</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> --enforce .github/workflows/security.yaml .github/workflows </span></div></code></div></pre>
<p>The created workflow looks like this:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">push</span></div><div class="line"></div><div class="line"><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Security</span></div><div class="line"></div><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">ensure-pinned-actions</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Checkout code</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683</span><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># v4</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">Ensure SHA pinned actions</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">zgosalvez/github-actions-ensure-sha-pinned-actions@25ed13d0628a1601b4b44048e63cc4328ed03633</span><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># v3</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">with</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">allowlist</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">|</span></div><div class="line"><span style="color: #A3BE8C"> MyOrg/</span></div></code></div></pre>
<h3 id="automating-updates" tabindex="-1">Automating updates</h3>
<p>Finally, we need to keep our dependencies up to date. You have two options here:</p>
<ol>
<li>Run <code>pin-github-action</code> again</li>
<li>Use Dependabot or Renovate</li>
</ol>
<p><strong>I recommend using Dependabot or Renovate.</strong></p>
<p>Running <code>pin-github-action</code> on a repository with pinned SHAs and will extract the target version from the comment and update the SHA like so:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">diff</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF"> steps:</span></div><div class="line"><span style="color: #ECEFF4">-</span><span style="color: #BF616A"> - uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4</span></div><div class="line"><span style="color: #ECEFF4">+</span><span style="color: #A3BE8C"> - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4</span></div></code></div></pre>
<p>This is great for updating actions in bulk, but it doesn't improve your security posture that much. You're still blindly upgrading to the latest version.</p>
<p>I recommend using Dependabot or Renovate to update your actions. Both tools create Pull Requests with updated SHAs and provide a diff for review. Each pull request contains a link to the <code>compare</code> view on GitHub that you can use to audit the changes since your last update and ensure that the action has not been compromised.</p>
<h3 id="stay-safe%2C-pin-your-dependencies" tabindex="-1">Stay safe, pin your dependencies</h3>
<p>The compromise of <code>tj-actions/changed-files</code> serves as a crucial reminder of the security risks in GitHub Actions. While pinning actions to commit SHAs adds an extra step, it significantly reduces the risk of supply chain attacks.</p>
<p>By leveraging tools like <code>pin-github-action</code>, enforcing SHA pinning with <code>github-actions-ensure-sha-pinned-actions</code>, and automating updates with Dependabot or Renovate, teams can secure their workflows without unnecessary overhead.</p>
<p>Supply chain security is an ongoing effort, but with these practices in place, you can proactively protect your repositories and secrets from potential threats.</p>
</content>
</entry>
</feed>