Test experimental versions with GitHub Actions

If you’re a library maintainer you may want to test your code against every available version of a language, including pre-release versions. After all, you want to know if your code won’t work on the latest and greatest version of a language before your users do.

Using a matrix

GitHub Actions allows you to specify a matrix of values and will spawn one job per combination.

The following workflow will trigger two jobs, build (7.4) and build (8.0):

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]

Our libraries aren’t just what we write though, it’s a combination of the code we author and the libraries we depend on. Let’s add another parameter that checks with the lowest version of our dependencies and the highest available version of our dependencies.

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        composer-version: ["lowest", "highest"]

This workflow will generate four jobs:

• build (7.4, lowest)
• build (7.4, highest)
• build (8.0, lowest)
• build (8.0, lowest)

As you can see, the number of jobs can grow exponentially as you add new dimensions to the matrix.

Testing a prerelease version

PHP 8.1 has been released and we’d like to test our code against this version, but we don’t want our build to fail if the tests don’t pass as we don’t officially support PHP 8.1 yet.

GitHub Actions supports a continue-on-error parameter for jobs that if set to true will record a failure for the job, but won’t fail the whole workflow run:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: true

This has the unwanted side effect that our workflow will never fail, even if 7.4 or 8.0 don’t pass. Fortunately, we can set continue-on-error conditionally:

yaml
jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        php: ["7.4", "8.0", "8.1"]
    continue-on-error: ${{ matrix.php == '8.1' }}

In this example continue-on-error only evaluates to true for 8.1, which achieves the behaviour that we were looking for.

Testing multiple versions

Checking the value directly in the continue-on-error field works, but it gets hard to read if you’re running multiple experimental versions. Imagine that in addition to the new version (8.4), we want to test on PHP 7.3 too. We don’t officially support it any more, but our customers are still heavy users of it and it’d be good to know if all the tests pass.

We start with a matrix definition that contains all of our non-experimental versions. This will run two jobs as there is only a single entry in the experimental row.

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]

Matrix definitions allow an include parameter to be provided to add new entries to the matrix. Anything added with include adds a single entry, not a new permutation.

Let’s add an experimental 8.1 job to our workflow:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true

Which would generate the following set of jobs:

json
[
  { "php": "7.4", "experimental": false },
  { "php": "8.0", "experimental": false },
  { "php": "8.1", "experimental": true }
]

In code, the algorithm would look like the following:

js
const jobs = [];
for (let version of matrix.php) {
  for (let experiment of matrix.experimental) {
    jobs.push({ php: version, experimental: experiment });
  }
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false }
// ]
for (let j of matrix.include) {
  jobs.push(j);
}
// jobs is currently:
// [
//   { php: '7.4', experimental: false },
//   { php: '8.0', experimental: false },
//   { php: '8.1', experimental: true }
// ]

We can now use the experimental flag to decide if we want to continue-on-error:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

Adding a new experimental version is easy at this point. We can add an additional include entry for any versions that we want to test:

yaml
jobs:
  build:
    strategy:
      matrix:
        php: ["7.4", "8.0"]
        experimental: [false]
        include:
          - php: "8.1"
            experimental: true
          - php: "7.3"
            experimental: true
  continue-on-error: ${{ matrix.experimental }}

This workflow will run four jobs, and allow 7.3 and 8.1 to fail without failing the entire workflow run.