michaelheap.comThoughts on leadership, code and how to fix odd edge cases in tools (not necessarily in that order)2022-04-10T13:53:44Zhttps://michaelheap.comMichael Heapm@michaelheap.comjamf stuck on “Locating hardware information (macOS 11.6.5)”2022-04-10T13:53:44Zhttps://michaelheap.com/jamf-locating-hardware-information/<p>We use <code>jamf</code> at work, and I periodically run <code>jamf recon</code> and <code>jamf policy</code> to ensure that things are all up to date.</p>
<p>I noticed that the process was consistently hanging with the message <code>Stuck on “Locating hardware information (macOS 11.6.5)”</code>. From my googling it looked as though this was software update related, so I uninstalled <code>jamf</code>, applied the update and then reinstalled. Problem solved!</p>
<p>That is, until I tried to run <code>jamf recon</code> again and it hung at the same point. I opened up software update and it just hung, with the gear spinning forever.</p>
<p>More googling ensued, and I eventually found the magic command that I needed to get things working again:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">sudo launchctl kickstart -k system/com.apple.softwareupdated</span></div></code></div></pre>
<p>Once <code>softwareupdated</code> had restarted, I could run <code>jamf recon</code> without any issues</p>
Remove Docker containers by tag2022-03-24T13:40:47Zhttps://michaelheap.com/remove-docker-containers-by-tag/<p>I <em>always</em> forget to pass the <code>--rm</code> flag when running command line tools with Docker. This means that I end up with a large history of exited containers that I no longer need.</p>
<p>There's plenty of information out there about how to clean up <strong>all</strong> stopped containers, or all unused images. I wanted to delete all stopped containers with a specific tag. Here's how to do it:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">docker rm </span><span style="color: #ECEFF4">$(</span><span style="color: #A3BE8C">docker container ls -a -q --filter ancestor=image-name:tag --filter status=exited</span><span style="color: #ECEFF4">)</span></div></code></div></pre>
<p>This works thanks to the <a href="https://docs.docker.com/engine/reference/commandline/ps/#filtering">filtering capabilities</a> when combined with the <code>-q</code> option to output only container IDs. These are then fed in to <code>docker rm</code> to remove all images with that tag that are in the <code>exited</code> status.</p>
Update a single item in a list with `vuex`2022-02-07T18:30:21Zhttps://michaelheap.com/vue3-vuex-update-list-item/<p>I recently needed to find an item in a list of objects by ID and update one of it’s properties whilst working with Vue 3 and VueX. It took me longer than I'd like to admit to figure out, so I'm writing it down here.</p>
<p>Due to how objects are passed by reference in JS, you can reference the <code>item</code> using <code>Object.assign</code> to update a single property.</p>
<p><a href="https://github.com/mheap/vue3-vuex4-sample/blob/main/src/store.js">Here’s</a> the <code>store</code>, including a mutation to update the value:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">js</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">import</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">createStore</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">from</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">vuex</span><span style="color: #ECEFF4">"</span><span style="color: #81A1C1">;</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">const</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">store</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">createStore</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">state</span><span style="color: #ECEFF4">()</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">return</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">items</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> [</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">value</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">One</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">color</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">red</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">value</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Two</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">color</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">yellow</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">value</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Three</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">color</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">skyblue</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> ]</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">mutations</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">setColor</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">state</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">id</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">color</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">})</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">console</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">log</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">`</span><span style="color: #A3BE8C">Setting block </span><span style="color: #81A1C1">${</span><span style="color: #D8DEE9">id</span><span style="color: #81A1C1">}</span><span style="color: #A3BE8C"> to </span><span style="color: #81A1C1">${</span><span style="color: #D8DEE9">color</span><span style="color: #81A1C1">}</span><span style="color: #ECEFF4">`</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">const</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">item</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">state</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">items</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">find</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">(</span><span style="color: #D8DEE9">i</span><span style="color: #ECEFF4">)</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=></span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">i</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">id</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">==</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">id</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">if</span><span style="color: #D8DEE9FF"> (</span><span style="color: #D8DEE9">item</span><span style="color: #D8DEE9FF">) </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">Object</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">assign</span><span style="color: #D8DEE9FF">(</span><span style="color: #D8DEE9">item</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">...</span><span style="color: #D8DEE9">item</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">color</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">export</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">default</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">store</span><span style="color: #81A1C1">;</span></div></code></div></pre>
<p>Here’s how I <a href="https://github.com/mheap/vue3-vuex4-sample/blob/main/src/components/Block.vue#L12-L17">called the mutation</a> from the component:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">js</div><div class="code-container"><code><div class="line"><span style="color: #88C0D0">setColor</span><span style="color: #D8DEE9FF">(</span><span style="color: #D8DEE9">color</span><span style="color: #D8DEE9FF">) </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">store</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">commit</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">setColor</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">this</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">id</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">color</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">return</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false;</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>The GitHub repo in this post renders a <a href="https://vue3-vuex4-color-blocks.netlify.app/">sample application</a> where you can change the colour of some blocks.</p>
Using `openvpn-client` with Docker2022-01-28T15:06:56Zhttps://michaelheap.com/openvpn-docker-compose/<p>I recently worked out the correct incantation to get a set of containers to connect to the internet via a VPN using <code>docker-compose</code>. I run it on a QNAP NAS, but it should work on any Linux-like system (I couldn’t get it working on MacOS).</p>
<p>The following <code>docker-compose.yml</code> will create two containers - one to run the VPN client and a second that runs <code>curl</code>. I use this to run <code>curl ipv4.canhazip.com</code> to check that the container has connectivity and the VPN IP address is returned.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">---</span></div><div class="line"><span style="color: #8FBCBB">version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">3</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #8FBCBB">services</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">vpn</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">container_name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">vpn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">image</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">dperson/openvpn-client:latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">cap_add</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">net_admin</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">restart</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">unless-stopped</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">volumes</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">/dev/net/tun:/dev/net/tun</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">./vpn-config:/vpn</span><span style="color: #D8DEE9FF"> </span><span style="color: #616E88"># You'll need to provide this</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">security_opt</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">label:disable</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">ports</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">8001:8001</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">8002:8002</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">8003:8003</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">networks</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">bridge_vpn</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">entrypoint</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">/sbin/tini</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">--</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">/usr/bin/openvpn.sh</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">-d</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">curl</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">image</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">alpine/curl</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">container_name</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">curl</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">restart</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">unless-stopped</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">network_mode</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">service:vpn</span></div><div class="line"><span style="color: #8FBCBB">networks</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">bridge_vpn</span><span style="color: #ECEFF4">:</span></div></code></div></pre>
<p>This compose file will expose ports <code>8001</code>, <code>8002</code> and <code>8003</code> from any containers using <code>network_mode: service:vpn</code> and make them accessible via a bridge network. This is useful when running a service that connects to the internet using a VPN. (There are no exposed ports in this demo, but I wanted to make a note here as in my actual deployment some of the other services expose ports.)</p>
<p>The biggest change to this configuration compared to a lot of guides out there is the <code>entrypoint</code> line for the <code>vpn</code> service. The <code>openvpn-client</code> image supports a <code>-d</code> flag that adds some DNS related pre/post scripts. I found that these are required to make connectivity work via the VPN.</p>
<p>You may have noticed the <code>vpn-config</code> folder being mounted. This is where you’ll provide your VPN configuration and authentication files. I tested this with Private Internet Access. If you’re a PIA customer, you’ll need to create a file named <code>vpn.auth</code> with your PIA username on the first line, and password on the second line, plus a <code>vpn.conf</code> file with the following contents:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">apache</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">client</span></div><div class="line"><span style="color: #D8DEE9FF">dev tun</span></div><div class="line"><span style="color: #D8DEE9FF">proto udp</span></div><div class="line"><span style="color: #D8DEE9FF">remote sweden.privacy.network </span><span style="color: #B48EAD">1198</span></div><div class="line"><span style="color: #D8DEE9FF">resolv-retry infinite</span></div><div class="line"><span style="color: #D8DEE9FF">nobind</span></div><div class="line"><span style="color: #D8DEE9FF">persist-key</span></div><div class="line"><span style="color: #616E88"># persist-tun # disable to completely reset vpn connection on failure</span></div><div class="line"><span style="color: #D8DEE9FF">cipher aes-</span><span style="color: #B48EAD">128</span><span style="color: #D8DEE9FF">-cbc</span></div><div class="line"><span style="color: #D8DEE9FF">auth sha1</span></div><div class="line"><span style="color: #D8DEE9FF">tls-client</span></div><div class="line"><span style="color: #D8DEE9FF">remote-cert-tls server</span></div><div class="line"><span style="color: #D8DEE9FF">auth-user-pass /vpn/vpn.auth # to be reachable inside the container</span></div><div class="line"><span style="color: #D8DEE9FF">comp-lzo</span></div><div class="line"><span style="color: #D8DEE9FF">verb </span><span style="color: #B48EAD">1</span></div><div class="line"><span style="color: #D8DEE9FF">reneg-sec </span><span style="color: #B48EAD">0</span></div><div class="line"><span style="color: #D8DEE9FF">crl-verify /vpn/crl.rsa.</span><span style="color: #B48EAD">2048</span><span style="color: #D8DEE9FF">.pem # to be reachable inside the container</span></div><div class="line"><span style="color: #D8DEE9FF">ca /vpn/ca.rsa.</span><span style="color: #B48EAD">2048</span><span style="color: #D8DEE9FF">.crt # to be reachable inside the container</span></div><div class="line"><span style="color: #D8DEE9FF">disable-occ</span></div><div class="line"><span style="color: #D8DEE9FF">keepalive </span><span style="color: #B48EAD">10</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">30</span><span style="color: #D8DEE9FF"> # send a ping every </span><span style="color: #B48EAD">10</span><span style="color: #D8DEE9FF"> sec and reconnect after </span><span style="color: #B48EAD">30</span><span style="color: #D8DEE9FF"> sec of unsuccessfull pings</span></div><div class="line"><span style="color: #D8DEE9FF">pull-filter ignore "auth-token" # fix PIA reconnection auth error that may occur every </span><span style="color: #B48EAD">8</span><span style="color: #D8DEE9FF"> hours</span></div></code></div></pre>
<p>This configuration connects to Sweden, but you can switch to any endpoint that you like. You’ll also need to download <code>crl.rsa.2048.pem</code> and <code>ca.rsa.2048.pem</code> from the <a href="https://www.privateinternetaccess.com/helpdesk/kb/articles/where-can-i-find-your-ovpn-files">PIA site</a>.</p>
<p>Once you have everything configured, it’s time to create the containers by running <code>docker-compose up -d</code>. Once that’s complete, you can test your connection by logging in to the <code>vpn</code> service and making a HTTP call:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">docker-compose </span><span style="color: #88C0D0">exec</span><span style="color: #D8DEE9FF"> vpn bash -c </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">curl ipv4.canhazip.com</span><span style="color: #ECEFF4">"</span></div></code></div></pre>
<p>If the above command returns an IP address successfully, you can also test it using the <code>curl</code> container which is configured to use the VPN for all network connectivity:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">docker-compose run curl ipv4.canhazip.com</span></div></code></div></pre>
<p>At this point, you have a <code>docker-compose</code> setup that connects all of the containers configured via an OpenVPN connection. Replace the <code>curl</code> service with any other service you may want to run behind a VPN and enjoy as your traffic is safe from snooping eyes.</p>
The GitHub wiki is an anti-pattern2022-01-23T18:08:26Zhttps://michaelheap.com/github-wiki-is-an-antipattern/<p>The “<a href="https://twitter.com/joemasilotti/status/1483124554843058180">should I use the wiki or a docs folder on GitHub?</a>” discussion comes up every 6 months or so, and following Shawn Wang’s <a href="https://www.swyx.io/three-strikes/">three strikes rule</a> I thought it was about time I wrote something down about it.</p>
<p>The initial version of this post opened with “You can use the wiki or a <code>docs</code> folder for your GitHub project, both are valid choices” but as I wrote more, I realised that there is a single reason to use a wiki, and many more reasons <em>not</em> to use the wiki. So many in fact, that I consider <strong>using the wiki on GitHub is an anti-pattern</strong>.</p>
<p>Let’s start with the benefits of using a wiki:</p>
<ol>
<li>You can get to the wiki contents in a single click from anywhere in the repo</li>
<li>There is no 2.</li>
</ol>
<p>Really, the only benefit that I’ve been able to find to using the wiki is that it’s always there.</p>
<p>How about the reasons <strong>not</strong> to use the wiki?</p>
<ol>
<li>Documentation is versioned alongside your code when using the <code>/docs</code> folder. If you need to use an old version, the docs are easy to find</li>
<li>The documentation isn’t available locally when someone clones your repo (you can clone the wiki separately, but this is a hidden feature)</li>
<li>Documentation edits get the same treatment as code. They get a full peer review through the pull request process</li>
<li>You can use GitHub Actions to lint your docs using tools such as <a href="https://github.com/errata-ai/vale-action">Vale</a></li>
<li>People can work with tooling that they already know (e.g. <code>vscode</code> with spellcheck)</li>
<li>Wikis provide limited branding opportunities. They all look pretty much the same</li>
<li>The wiki doesn’t support image uploads, so you have to put images somewhere else anyway</li>
</ol>
<p>Now that you’re sold on the idea of keeping docs alongside your code, how do you make it easy for people to view them?</p>
<ol>
<li>Add your docs to your repository in the <code>/docs</code> folder. Do <em>not</em> use the <code>gh-pages</code> branch as this prevents docs being versioned alongside code</li>
<li>Set up a GitHub pages build to publish the docs
<ul>
<li>If you’re just getting started, I recommend using the <code>just-the-docs</code> theme and letting GitHub build and publish your docs</li>
<li>If you prefer to build your own workflow (e.g. using Hugo), you can use <a href="https://github.com/peaceiris/actions-gh-pages/">this</a> GitHub Action to publish the docs</li>
</ul>
</li>
<li>Add a single wiki page directing people to the hosted documentation</li>
</ol>
<p>Using the <code>/docs</code> folder is the highest effort-to-reward ratio option whilst you’re building out a new product. At some point your docs will outgrow a single folder, and then all bets are off. You’ll want a separate repo with its own build process, pull request review guidelines and a whole host of other things. At that point people are already used to working with docs in a repository, and the migration from <code>/docs</code> to its own repo should be seamless for your contributors.</p>
<p>Whether you agree or disagree, I’d love to hear your thoughts <a href="https://twitter.com/mheap/">on Twitter</a></p>
VSCode only showing one open tab2021-11-22T12:53:51Zhttps://michaelheap.com/vscode-show-tabs/<p>My VSCode instance has been refusing to show more than a single tab for a while, even with preview disabled and when double clicking on a filename to open.</p>
<p>Most blog posts talking about how to fix this have a similar resolution to <a href="https://stackoverflow.com/questions/48589785/vscode-showing-only-one-file-in-the-tab-bar-cant-open-multiple-files">this</a> Stack Overflow thread. That is:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">workbench.editor.showTabs</span><span style="color: #ECEFF4">"</span><span style="color: #D8DEE9FF">: </span><span style="color: #81A1C1">false</span></div></code></div></pre>
<p>This didn't work for me, nor did setting <code>"workbench.editor.enablePreview": false</code></p>
<p>What worked for me is to lock a tab as opened by pressing <code>Ctrl+K+Enter</code> whilst the tab is focused. This updated the UI to show a tab bar and when I double clicked on another file it opened as expected.</p>
Convert a PDF to a stacked jpg2021-11-11T13:36:24Zhttps://michaelheap.com/convert-pdf-to-stacked-jpg/<p>I built some UI mockups in Google Slides today and wanted to share the output as a single <code>jpg</code> file. The way to achieve this is to export the deck as PDF, use <code>imagemagick</code> to convert the PDF to images, then combine the images.</p>
<p>Once you have your PDF, save it as <code>input.pdf</code> in a new folder, then run the following commands:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">convert -density 150 input.pdf -quality 90 output.jpg</span></div><div class="line"><span style="color: #D8DEE9FF">convert -append output-</span><span style="color: #81A1C1">*</span><span style="color: #D8DEE9FF"> final.jpg</span></div></code></div></pre>
<p>You'll now have a file named <code>final.jpg</code> in your folder that contains all of the pages from the PDF, stacked vertically.</p>
<p>If you'd prefer to stack the images horizontally rather than vertically, you can use <code>+append</code> rather than <code>-append</code>.</p>
Test experimental versions with GitHub Actions2021-10-20T12:25:54Zhttps://michaelheap.com/testing-experimental-versions-github-actions/<p>If you’re a library maintainer you may want to test your code against every available version of a language, including pre-release versions. After all, you want to know if your code won’t work on the latest and greatest version of a language before your users do.</p>
<h2 id="using-a-matrix" tabindex="-1">Using a matrix</h2>
<p>GitHub Actions allows you to <a href="https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idstrategymatrix">specify a matrix of values</a> and will spawn one job per combination.</p>
<p>The following workflow will trigger two jobs, <code>build (7.4)</code> and <code>build (8.0)</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div></code></div></pre>
<p>Our libraries aren’t just what we write though, it’s a combination of the code we author and the libraries we depend on. Let’s add another parameter that checks with the <code>lowest</code> version of our dependencies and the <code>highest</code> available version of our dependencies.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">composer-version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">lowest</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">highest</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div></code></div></pre>
<p>This workflow will generate four jobs:</p>
<ul>
<li><code>build (7.4, lowest)</code></li>
<li><code>build (7.4, highest)</code></li>
<li><code>build (8.0, lowest)</code></li>
<li><code>build (8.0, lowest)</code></li>
</ul>
<p>As you can see, the number of jobs can grow exponentially as you add new dimensions to the matrix.</p>
<h2 id="testing-a-prerelease-version" tabindex="-1">Testing a prerelease version</h2>
<p>PHP 8.1 has been released and we’d like to test our code against this version, but we don’t want our build to fail if the tests don’t pass as we don’t officially support PHP 8.1 yet.</p>
<p>GitHub Actions supports a <code>continue-on-error</code> parameter for jobs that if set to <code>true</code> will record a failure for the job, but <strong>won’t</strong> fail the whole workflow run:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">continue-on-error</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div></code></div></pre>
<p>This has the unwanted side effect that our workflow will never fail, even if <code>7.4</code> or <code>8.0</code> don’t pass. Fortunately, we can set <code>continue-on-error</code> conditionally:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">continue-on-error</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.php == '8.1' }}</span></div></code></div></pre>
<p>In this example <code>continue-on-error</code> only evaluates to true for <code>8.1</code>, which achieves the behaviour that we were looking for.</p>
<h2 id="testing-multiple-versions" tabindex="-1">Testing multiple versions</h2>
<p>Checking the value directly in the <code>continue-on-error</code> field works, but it gets hard to read if you’re running multiple experimental versions. Imagine that in addition to the new version (<code>8.4</code>), we want to test on PHP <code>7.3</code> too. We don’t officially support it any more, but our customers are still heavy users of it and it’d be good to know if all the tests pass.</p>
<p>We start with a matrix definition that contains all of our non-experimental versions. This will run two jobs as there is only a single entry in the <code>experimental</code> row.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #81A1C1">false</span><span style="color: #ECEFF4">]</span></div></code></div></pre>
<p>Matrix definitions allow an <code>include</code> parameter to be provided to add new entries to the matrix. Anything added with <code>include</code> adds a single entry, not a new permutation.</p>
<p>Let’s add an experimental <code>8.1</code> job to our workflow:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #81A1C1">false</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">include</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div></code></div></pre>
<p>Which would generate the following set of jobs:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">[</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #ECEFF4">]</span></div></code></div></pre>
<p>In code, the algorithm would look like the following:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">js</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">const</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">jobs</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">=</span><span style="color: #D8DEE9FF"> []</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #81A1C1">for</span><span style="color: #D8DEE9FF"> (</span><span style="color: #81A1C1">let</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">version</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">of</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">matrix</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">php</span><span style="color: #D8DEE9FF">) </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">for</span><span style="color: #D8DEE9FF"> (</span><span style="color: #81A1C1">let</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">experiment</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">of</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">matrix</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">experimental</span><span style="color: #D8DEE9FF">) </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">jobs</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">push</span><span style="color: #D8DEE9FF">(</span><span style="color: #ECEFF4">{</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">version</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #88C0D0">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">experiment</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #ECEFF4">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88">// jobs is currently:</span></div><div class="line"><span style="color: #616E88">// [</span></div><div class="line"><span style="color: #616E88">// { php: '7.4', experimental: false },</span></div><div class="line"><span style="color: #616E88">// { php: '8.0', experimental: false }</span></div><div class="line"><span style="color: #616E88">// ]</span></div><div class="line"></div><div class="line"><span style="color: #81A1C1">for</span><span style="color: #D8DEE9FF"> (</span><span style="color: #81A1C1">let</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">j</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">of</span><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">matrix</span><span style="color: #ECEFF4">.</span><span style="color: #D8DEE9">include</span><span style="color: #D8DEE9FF">) </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #D8DEE9">jobs</span><span style="color: #ECEFF4">.</span><span style="color: #88C0D0">push</span><span style="color: #D8DEE9FF">(</span><span style="color: #D8DEE9">j</span><span style="color: #D8DEE9FF">)</span><span style="color: #81A1C1">;</span></div><div class="line"><span style="color: #ECEFF4">}</span></div><div class="line"></div><div class="line"><span style="color: #616E88">// jobs is currently:</span></div><div class="line"><span style="color: #616E88">// [</span></div><div class="line"><span style="color: #616E88">// { php: '7.4', experimental: false },</span></div><div class="line"><span style="color: #616E88">// { php: '8.0', experimental: false },</span></div><div class="line"><span style="color: #616E88">// { php: '8.1', experimental: true }</span></div><div class="line"><span style="color: #616E88">// ]</span></div></code></div></pre>
<p>We can now use the <code>experimental</code> flag to decide if we want to <code>continue-on-error</code>:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #81A1C1">false</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">include</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">continue-on-error</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.experimental }}</span></div></code></div></pre>
<p>Adding a new experimental version is easy at this point. We can add an additional <code>include</code> entry for any versions that we want to test:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.4</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.0</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #81A1C1">false</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">include</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">8.1</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">php</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">7.3</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">experimental</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">continue-on-error</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.experimental }}</span></div></code></div></pre>
<p>This workflow will run four jobs, and allow <code>7.3</code> and <code>8.1</code> to fail without failing the entire workflow run.</p>
Dynamic matrix generation with GitHub Actions2021-10-18T08:24:23Zhttps://michaelheap.com/dynamic-matrix-generation-github-actions/<p><a href="https://docs.github.com/en/actions/learn-github-actions/managing-complex-workflows#using-a-build-matrix">Using a build matrix</a> with GitHub Actions allows us to run tests across multiple combinations of operating systems, platforms and languages. You can set up huge matrices (a matrix with three parameters, each of which has three values will run 27 jobs!), but most workflows that you see will have a single matrix entry.</p>
<p>In this example, we’re going to run our tests on the currently supported versions of Node using the following workflow:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">push</span></div><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">ci</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span><span style="color: #B48EAD">12</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">14</span><span style="color: #ECEFF4">,</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">16</span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/checkout@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/setup-node@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">with</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">node-version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.version }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm ci</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm test</span></div></code></div></pre>
<p>The list of supported Node.js versions is accurate today, but what happens in 6 months? How about 12 months? We’d have to come back and update this list each time the list of supported versions changes.</p>
<p>Let’s update this workflow to build the list of supported versions dynamically so that it’s always up to date.</p>
<h2 id="create-a-matrix-from-an-output" tabindex="-1">Create a matrix from an output</h2>
<p>In the workflow above we set <code>matrix.version</code> manually, but it doesn’t have to be hardcoded. We can populate it using the output values a previous job. Let’s update the workflow to use an <code>output</code> as the <code>matrix.version</code> value and add a new job that returns a list of currently supported <code>node</code> versions:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">push</span></div><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build-matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">set-matrix</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">echo '::set-output name=version_matrix::["12","14","16"]'</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">outputs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">version_matrix</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ steps.set-matrix.outputs.version_matrix }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">ci</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">needs</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">build-matrix</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ fromJson(needs.build-matrix.outputs.version_matrix) }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/checkout@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/setup-node@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">with</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">node-version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.version }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm ci</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm test</span></div></code></div></pre>
<p>The list of versions is still hardcoded in the <code>set-matrix</code> job, but we’re one step closer to dynamically populating the list of active versions.</p>
<h2 id="fetch-supported-node.js-versions" tabindex="-1">Fetch supported Node.js versions</h2>
<p>The next step is to replace the hardcoded string (<code>["12","14","16"]</code>) with a data source that dynamically updates. For this post I’m using <a href="https://endoflife.date/nodejs">endoflife.date</a> to fetch the list of active versions.</p>
<p>The API returns a list of objects that look like the following:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">cycle</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">16</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">release</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">2021-04-20</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">lts</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">false</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">support</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">2022-10-18</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">eol</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">2024-04-30</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">latest</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">16.10.0</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>Out action doesn’t need all of that data - all we need is the <code>cycle</code> value, and to ensure that the <code>eol</code> date is before today. Fortunately the Actions runners have <code>jq</code> installed, which we can use to manipulate the data.</p>
<p>Here’s the command that we use to return the list of active versions:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">bash</div><div class="code-container"><code><div class="line"><span style="color: #D8DEE9FF">curl https://endoflife.date/api/nodejs.json </span><span style="color: #81A1C1">|</span><span style="color: #D8DEE9FF"> jq -c </span><span style="color: #ECEFF4">'</span><span style="color: #A3BE8C">[.[] | select(.eol > (now | strftime("%Y-%m-%d"))) | .cycle]</span><span style="color: #ECEFF4">'</span></div><div class="line"><span style="color: #616E88"># ["16","14","12"]</span></div></code></div></pre>
<p>We need to use a sub-shell to run the command in our workflow, which means wrapping the command in <code>$()</code>.</p>
<p>Putting it all together, this is what the workflow looks like:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">yaml</div><div class="code-container"><code><div class="line"><span style="color: #81A1C1">on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">push</span></div><div class="line"><span style="color: #8FBCBB">jobs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">build-matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">id</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">set-matrix</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">echo "::set-output name=version_matrix::$(curl https://endoflife.date/api/nodejs.json | jq -c '[.[] | select(.eol > (now | strftime("%Y-%m-%d"))) | .cycle]')"</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">outputs</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">version_matrix</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ steps.set-matrix.outputs.version_matrix }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">ci</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">needs</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">build-matrix</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">runs-on</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">ubuntu-latest</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">strategy</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">matrix</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ fromJson(needs.build-matrix.outputs.version_matrix) }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">steps</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/checkout@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">uses</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">actions/setup-node@v2</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">with</span><span style="color: #ECEFF4">:</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">node-version</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">${{ matrix.version }}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm ci</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">-</span><span style="color: #D8DEE9FF"> </span><span style="color: #8FBCBB">run</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #A3BE8C">npm test</span></div></code></div></pre>
<p>When this workflow runs, it fetches the list of active Node.js versions in the first job then triggers one instance of the <code>ci</code> job per version returned.</p>
<p>Try adding it to one of your own projects to see it work 🥳</p>
<h2 id="conclusion" tabindex="-1">Conclusion</h2>
<p>The example shown above is simple, but super-useful! As soon as there’s a new version of node available, our code will start being tested against it. No more surprises where the community find out that your code doesn’t work before you do.</p>
<p>I’ve seen a couple of interesting applications of this, but none more interesting than RectorPHP <a href="https://github.com/rectorphp/rector/actions?query=workflow%3A%22Packagist+Testing%22">testing their refactoring tool against the top 50 packages in the PHP ecosystem</a>. The top 50 list is populated dynamically to ensure that as new packages are released and picked up, they get early feedback on if their tool is compatible.</p>
Getting started with Problem Matchers2021-10-11T12:58:48Zhttps://michaelheap.com/getting-started-problem-matchers/<p>Problem matchers are a relatively new concept that allow you to watch unstructured log output for specific details and add annotations to your source code based on what it finds.</p>
<p>This means that you don’t need to go reading through hundreds of lines of logs. Any relevant information will be added to your code inline, allowing you to see the information in context.</p>
<p>Imagine that you’re using ESLint for your JavaScript code. Instead of seeing that <code>myVar</code> is declared on line 50, then not used in the rest of the file on line 312 of a 500 log file, it’ll show that information on line 50 of your editor (or in your pull request!)</p>
<p>Here's how it looks in VSCode (I'm using the <a href="https://marketplace.visualstudio.com/items?itemName=usernamehw.errorlens">errorlens</a> plugin to show the problem inline):</p>
<div class="image-wrapper "><picture> <source class="" type="image/webp" srcset="https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-320.webp 320w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-640.webp 640w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-960.webp 960w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-1200.webp 1200w" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" /> <img class="" alt="Problem matchers in VSCode" src="https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-1200.jpeg" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" srcset="https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-320.jpeg 320w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-640.jpeg 640w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-960.jpeg 960w, https://michaelheap.com/images/getting-started-problem-matchers/vscode-problems.png/AAzGmw8Z4b-1200.jpeg 1200w" width="1200" height="580" /> </picture></div>
<p>Then once I push to GitHub it shows in the Actions log:</p>
<div class="image-wrapper "><picture> <source class="" type="image/webp" srcset="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-320.webp 320w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-640.webp 640w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-960.webp 960w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-1200.webp 1200w" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" /> <img class="" alt="Problem matchers in VSCode" src="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-1200.jpeg" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" srcset="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-320.jpeg 320w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-640.jpeg 640w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-960.jpeg 960w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-log.png/maySDxvF7N-1200.jpeg 1200w" width="1200" height="509" /> </picture></div>
<p>Plus it shows as an inline annotation in the files list on the pull request page:</p>
<div class="image-wrapper border-gray-300 border-4 mb-4 py-2"><picture> <source class="m-auto" type="image/webp" srcset="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-320.webp 320w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-640.webp 640w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-960.webp 960w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-1200.webp 1200w" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" /> <img class="m-auto" alt="Problem matchers in VSCode" src="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-1200.jpeg" sizes="(max-width: 320px) 320px, (max-width: 640px) 640px, (max-width: 960px) 960px, (max-width: 1200px) 1200px, 100vw" srcset="https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-320.jpeg 320w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-640.jpeg 640w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-960.jpeg 960w, https://michaelheap.com/images/getting-started-problem-matchers/github-problems-annotation.png/jAk-5Wdw98-1200.jpeg 1200w" width="1200" height="591" /> </picture></div>
<p>If you want to try building your own problem matchers after reading this post, you might find this <a href="https://problem-matcher.netlify.app/">problem matcher tester</a> (<a href="https://github.com/mheap/problem-matcher-tester">source</a>) useful.</p>
<h2 id="how-problem-matchers-work" tabindex="-1">How Problem Matchers work</h2>
<p>If we strip problem matchers right back to their core, they’re <em>just</em> a regular expression (<em>just</em> - as though regular expressions don’t strike fear into all of our hearts!)</p>
<p>The way they work is by scanning each line of the output for a specific pattern and populating a predefined set of values using the strings that match the regular expression.</p>
<p>A minimal problem matcher has three values:</p>
<ul>
<li><code>file</code></li>
<li><code>line</code></li>
<li><code>message</code></li>
</ul>
<blockquote>
<p>There is another type of matcher which specifies <code>{"kind": "file"}</code> rather than specifying a <code>line</code> entry. We’ll cover these later on</p>
</blockquote>
<p>These values allow a <code>message</code> to be shown in the interface in the correct <code>file</code> on the correct <code>line</code>. Imagine that you have a log message containing the following:</p>
<blockquote>
<p>src/sample.js:10:This is the message</p>
</blockquote>
<p>In <a href="https://regex101.com/r/CA7o0E/1/">this regular expression</a>, everything up to the first colon is the filename, the next section is the line number and the remainder of the line is the message to add. To match this message using a problem matcher you’d use the following configuration:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">owner</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">demo-matcher</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">pattern</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">([^:]+):(</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+):(.+)</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">line</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>It’s important to note that in your JSON file, you need to escape any backslashes, which means that (\d+) becomes (\\d+).</p>
<blockquote>
<p>In all future examples I’m only going to show the <code>pattern</code> section of this config file to make the examples smaller</p>
</blockquote>
<p>You could take this problem matcher and deploy it and you’d start seeing annotations in your application, but this is just the start.</p>
<h3 id="single-line-matchers" tabindex="-1">Single line matchers</h3>
<p>Problem matchers can also provide a lot more context than just the <code>file</code>, <code>line</code> and <code>message</code>. You’ll find that most tools also output a <code>severity</code> level and <code>column</code> which you can use to augment your annotations. Using the example above, we can add this additional information into the error message:</p>
<blockquote>
<p>ERROR:src/sample.js:10,12:This is the message</p>
</blockquote>
<p>To capture the <code>severity</code> and <code>column</code> we need to add some additional capture groups to our regular expression, and add some additional values:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">(ERROR|WARNING|INFO):([^:]+):(d+),(d+):(.+)</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">line</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">5</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">severity</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">column</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">4</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p><a href="https://regex101.com/r/4b6DBg/1/">This regular expression</a> will match anything that starts with <code>ERROR</code>, <code>WARNING</code> or <code>INFO</code> (the <code>severity</code>), followed by the <code>file</code>, then a <code>line</code>, a comma and then <code>column</code>, followed by the <code>message</code>.</p>
<p>The example above is well formatted, with colons to help mark the end of fields. Problem matchers can work with unstructured text too:</p>
<blockquote>
<p>Sadly there was an error on line 19. "Something went wrong". It was on column 12 in sample.js</p>
</blockquote>
<p>To match the above string, you’d use the following problem matcher:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">Sadly there was an error on line (</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+).</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+</span><span style="color: #EBCB8B">\"</span><span style="color: #A3BE8C">([^</span><span style="color: #EBCB8B">\"</span><span style="color: #A3BE8C">]+)</span><span style="color: #EBCB8B">\"</span><span style="color: #A3BE8C">.</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+It was on column (</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+) in ([</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">w</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">.]+)</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">5</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">line</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">severity</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">column</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">4</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>Finally, there are a few additional properties that can be added in addition to <code>file</code>, <code>line</code>, <code>message</code>, <code>severity</code> and <code>column</code>:</p>
<ul>
<li><code>location</code> - a shorthand way to provide <code>line</code> and <code>column</code> in a single group. Allows for the format <code>line</code>, <code>line,column</code> or <code>startLine,startColumn,endLine,endColumn</code></li>
<li><code>endLine</code> - for multi-line notices. This will highlight all lines between <code>line</code> and <code>endLine</code></li>
<li><code>endColumn</code> - the same as <code>endLine</code>, but for columns</li>
<li><code>code</code> - capture a standardised error code</li>
</ul>
<p>I’ve not seen these additional properties used, with the exception of <code>code</code> which can be useful when capturing the rule name using a linting tool.</p>
<h3 id="multi-line-matches" tabindex="-1">Multi-line matches</h3>
<p>Problem matchers work on multi-line messages in addition to single line matchers.</p>
<p>Here’s some sample output from the ESLint stylish formatter:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="code-container"><code><div class="line"><span style="color: undefined">test.js
1:0 error Missing "use strict" statement strict
5:10 error 'addOne' is defined but never used no-unused-vars
foo.js
36:10 error Expected parentheses around arrow function argument arrow-parens
37:13 error Expected parentheses around arrow function argument arrow-parens
✖ 4 problems (4 errors, 0 warnings)</span></div></code></div></pre>
<p>We can see that the first line provides the <code>file</code>, then each line after that provides the <code>line</code>, <code>column</code>, <code>severity</code>, <code>message</code> and <code>code</code>.</p>
<p>To match these values, problem matchers allows you to specify multiple regular expressions. Each regular expression will match as many lines as it can before moving on to the next expression if you set <code>loop</code> to <code>true</code>.</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">[</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">^([^</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s].*)$</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">^</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+):(</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(error|warning|info)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(.*)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(.*)$</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">line</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">column</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">severity</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">4</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">code</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">5</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">loop</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #ECEFF4">]</span></div></code></div></pre>
<p>In this configuration, the first regular expression matches everything up to the first space and stores it in <code>file</code>, then as that expression doesn’t match the next line it moves on to the next expression. This regular expression matches lines 2 and three, populating the provided values and looping until the regex no longer matches. At this point it goes back to the first expression and starts looping again.</p>
<h3 id="file-level-matches" tabindex="-1">File level matches</h3>
<p>I mentioned earlier that you need to provide a <code>line</code> value unless you set <code>kind: file</code>. Let’s take a look at what <code>kind: file</code> allows us to do.</p>
<p>It’s unusual to have an error that doesn’t relate to a specific line in a file when you think about unit tests or style linting. However, when the unit of work you’re looking at gets bigger, sometimes error messages only make sense in the context of a whole file.</p>
<p>Imagine that you’ve got a set of acceptance tests that are configured using a YAML file that contains account information for a test user. The same test will run for multiple users, so adding an annotation on the single test wouldn’t make sense. Instead, we want to flag which configuration files failed.</p>
<p>Here’s the sample output from our testing tool:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="code-container"><code><div class="line"><span style="color: undefined">[ i ] PROCESS SUITES/TESTS RESULTS ...
----------------------------------------------------------------------
Suite: acceptance_ui_account (1 tests)
----------------------------------------------------------------------
failed: 1
----------------------------------------------------------------------
| 00:00:05 | failed | User can log in | user/login/alice.yml
| 00:00:12 | passed | User can log in | user/login/bob.yml
| 00:00:18 | error | User can log in | user/login/charlie.yml
----------------------------------------------------------------------
Suite start time : 12:41:09
Suite end time : 12:41:27
Suite elapsed time: 00:00:18
----------------------------------------------------------------------
[ ! ] Failed tests list was stored in 'acceptance_ui_account' suite.</span></div></code></div></pre>
<p>The <a href="https://regex101.com/r/C4MKmJ/1">regular expression</a> for this is a bit more complex, but it allows us to specify a whole file to add an annotation:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">|</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+:</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+:</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">|</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(failed|error)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">|</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+([^</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">|]+)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">|</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(.*)$</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">3</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">severity</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">kind</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">file</span><span style="color: #ECEFF4">"</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>I’ve not seen any real-world usage of <code>kind: file</code>, so if you’re using it <a href="https://twitter.com/mheap/">let me know</a>.</p>
<h2 id="top-level-parameters" tabindex="-1">Top level parameters</h2>
<p>If your logs don’t contain a <code>severity</code> entry, you might be wondering how you can set this value to add annotations as an error or a warning. Problem matchers can specify a default <code>severity</code> value which will be applied to all entries:</p>
<pre class="shiki nord" style="background-color: #2e3440ff; color: #d8dee9ff"><div class="language-id">json</div><div class="code-container"><code><div class="line"><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">owner</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">eslint-stylish</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">severity</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">error</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">pattern</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">[</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">^([^</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s].*)$</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">file</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">},</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">{</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">regexp</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #A3BE8C">^</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+):(</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">d+)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(.*)</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s</span><span style="color: #EBCB8B">\\</span><span style="color: #A3BE8C">s+(.*)$</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">line</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">1</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">column</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">2</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">message</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">4</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">code</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #B48EAD">5</span><span style="color: #ECEFF4">,</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">"</span><span style="color: #8FBCBB">loop</span><span style="color: #ECEFF4">"</span><span style="color: #ECEFF4">:</span><span style="color: #D8DEE9FF"> </span><span style="color: #81A1C1">true</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">}</span></div><div class="line"><span style="color: #D8DEE9FF"> </span><span style="color: #ECEFF4">]</span></div><div class="line"><span style="color: #ECEFF4">}</span></div></code></div></pre>
<p>In addition to <code>severity</code>, you can set a few other values in addition to <code>owner</code> (which don’t have any effect on GitHub, but do in VSCode):</p>
<ul>
<li><code>applyTo</code> - Set to <code>allDocuments</code> to run against all files, not just open files</li>
<li><code>background</code> - Used to detect if a background task is running</li>
<li><code>base</code> - The problem matcher to extend. Any properties specified in <code>pattern</code> will override the values in <code>base</code></li>
<li><code>fileLocation</code> - specify if the file path is <code>relative</code> or <code>absolute</code></li>
</ul>
<h2 id="actions-with-built-in-matchers" tabindex="-1">Actions with built-in matchers</h2>
<p>Everything so far has covered writing your own problem matchers, but for common use cases you don’t need to.</p>
<p>If you’re a JavaScript developer using <code>actions/setup-node</code>, you’ll get the problem matchers for <a href="https://github.com/actions/setup-node/blob/main/src/main.ts#L58-L65">ESLint and Typescript</a> (<code>tsc</code>) added for free. <code>setup-go</code> adds a c<a href="https://github.com/actions/setup-go/blob/main/matchers.json">ompiler matcher</a>, as do <a href="https://github.com/actions/setup-dotnet/blob/main/.github/csc.json"><code>setup-dotnet</code></a> and <a href="https://github.com/actions/setup-elixir/blob/main/.github/elixir.json"><code>setup-elixir</code></a>. <code>setup-java</code> adds a matcher for <a href="https://github.com/actions/setup-java/blob/main/.github/java.json">uncaught exceptions</a> and <code>setup-python</code> will <a href="https://github.com/actions/setup-python/blob/main/.github/python.json">show errors</a>.</p>
<p>Those are just the GitHub maintained <code>setup-*</code> actions too! You can also find available actions that register matchers for known patterns. For example, here’s a <a href="https://github.com/mheap/phpunit-matcher-action">matcher for PHPUnit tests</a> that matches the TeamCity output. A quick search also shows matchers for <a href="https://github.com/jonasb/android-problem-matchers-action">Android</a>, <a href="https://github.com/ammaraskar/gcc-problem-matcher">GCC</a>, <a href="https://github.com/xt0rted/stylelint-problem-matcher">StyleLint</a> and <a href="https://github.com/python/cpython/blob/main/.github/problem-matchers/sphinx.json">Sphinx</a>.</p>
<h2 id="building-your-own" tabindex="-1">Building your own</h2>
<p>The <a href="https://michaelheap.com/getting-started-problem-matchers/#how-problem-matchers-work">how problem matchers work</a> section above covers almost everything you need to know to build your own problem matchers, but I wanted to mention one last trick that I needed to make a custom problem matcher work.</p>
<p>For annotations to be added automatically, the <code>filename</code> value must be relative to the root of the repository. However, some tools (such as PHPUnit) provide the absolute path to the file.</p>
<p>To handle this, I add the <code>GITHUB_WORKSPACE</code> path to my regular expression dynamically and keep it out of the capture group. As we register new matchers using JavaScript, it’s easy to <a href="https://github.com/mheap/phpunit-matcher-action/blob/master/index.js#L20">replace a placeholder</a> in the regex and write out a new matcher file.</p>
<h2 id="conclusion" tabindex="-1">Conclusion</h2>
<p>Problem matchers are used by GitHub Actions and VSCode today, with Actions using a subset of the VSCode functionality.</p>
<p>On GitHub, problem matchers run against the log output for GitHub Actions, and must be registered in a workflow before the output is written to the log.</p>
<p>In VSCode, problem matchers are defined <a href="https://code.visualstudio.com/docs/editor/tasks#_processing-task-output-with-problem-matchers">as part of a task</a>, and run on the output of the task that is run.</p>
<p>I’ve not seen any other usages of problem matchers out in the world, but they feel like a good solution to parsing free text and making it in to something that is contextually useful.</p>
<p>If you’ve seen an awesome use of problem matchers out in the wild, I’d love to hear about it - I’m <a href="https://twitter.com/mheap/">@mheap</a> on Twitter.</p>