Cache Mocha Test Results And Run Only Failed Test Files On Re-runs

Mocha can generate a JSON file with test results. It might be sometimes useful to find all failed test files, e.g. when I want to re-run only tests from such test files.

Running all automated tests might be time-consuming, so you might want to look for various ways to run only what’s necessary.

One such scenario might be that you run all tests. Some of them fail. You investigate (you should always do that) but realise that those tests failed for reasons you can’t really fix straight away. You decide to run the tests again. But this time, you might want to run only those tests files that contain tests that failed the first time over.

This is something that I implemented in GitHub pipelines. On re-runs, I tell Mocha to run only those test files that contain tests that failed in the previous run. For that, there’re two things I need to sort out:

• caching of the Mocha result file
• parsing the Mocha result file, finding failed test files and handing them to Mocha

Regarding the Mocha result file, I wrote more on that in this article, you can get an idea on how the file structure looks like.

On to the two points.

The caching part can be a bit tricky, because the famous actions/cache@v3 action does not cache anything when any step in a workflow fails. That is a major drawback because I want to do exactly that.

There’s hopefully a solution with its beta version v3.2.0-beta.1 (there are more beta versions as of December 2022, you might want to check those other ones as well). My workflow can then look like this (a simplified example from my mock repo):

- uses: actions/checkout@v3
- name: set chache
  id: set-cache
  run: |
    cache_prefix="test-results-${{ github.workflow }}-${{ matrix.task }}-${{ github.sha }}-${{ github.run_id }}"
    {
      echo "run_attempt=$(( GITHUB_RUN_ATTEMPT - 1 ))"
      echo "cache_prefix=$cache_prefix"
    } >> $GITHUB_OUTPUT
- name: restore test results
  id: restore-test-results
  uses: actions/cache/restore@v3.2.0-beta.1
  with:
    path: results.json
    key: ${{ steps.set-cache.outputs.cache_prefix }}-${{ steps.set-cache.outputs.run_attempt }}
- name: Setup node
  uses: actions/setup-node@v3
  with:
    node-version: 18
- name: install deps
  run: npm install
- name: test
  run: scripts/run-tests.sh
- name: cache test results
  if: always()
  id: save-test-results
  uses: actions/cache/save@v3.2.0-beta.1
  with:
    path: results.json
    key: ${{ steps.set-cache.outputs.cache_prefix }}-${{ github.run_attempt }}

Each run will create a new cache that will contain the Mocha test result file, it will also try to restore the Mocha file from the previous run.

In caches, it can look like this in my mock repo:

Created caches with Mocha test result file

Now on to the second point. If I actually retrieve cache, I want to hand to Mocha only those test files in which some tests failed. It can be done with a little Bash that puts together a few utilities:

#!/usr/bin/env bash

set -euo pipefail

function warn {
  echo "$0: $*" >&2
  echo >&2
}

function die() {
  warn "$@"
  exit 1
}

function remove_path_to_project_from_files() {
  echo "${1//$2/}"
}

function failures_without_file_exist() {
  grep -E '^null$' &>/dev/null <<< "$1"
}

function main() {
  if ! [[ -f results.json ]]; then
    die "No results.json file found"
  fi

  local failed_test_files
  failed_test_files="$(jq '.failures[].file' results.json | uniq)"

  if failures_without_file_exist "$failed_test_files"; then
    die "Some failures do not have \"file\" attribute, something more serious happened"
  fi

  if [[ -z "$failed_test_files" ]]; then
    warn "All tests passed last time, running everything again"
    return 0
  fi

  local path_to_delete="${GITHUB_WORKSPACE-$(pwd)}/"
  failed_test_files="$(remove_path_to_project_from_files "$failed_test_files" "$path_to_delete")"

  echo "$failed_test_files"
}

main "$@"

Let’s say that the failures in Mocha’s test result file look like this:

"failures": [
  {
    "title": "no argument instanceof",
    "fullTitle": "_getYyyyMmDd() no argument instanceof",
    "file": "/home/pavel/testing/useful-test/tests/_getYyyyMmDd.test.js",
    "duration": 1,
    "currentRetry": 0,
    "err": {
      "message": "expected true to equal false",
      "showDiff": true,
      "actual": "true",
      "expected": "false",
      "operator": "strictEqual",
      "stack": "AssertionError: expected true to equal false\n    at Context.<anonymous> (file:///home/pavel/testing/useful-test/tests/_getYyyyMmDd.test.js:19:47)\n    at process.processImmediate (node:internal/timers:471:21)"
    }
  },
  {
    "title": "object with no properties is object",
    "fullTitle": "_isObject() object with no properties is object",
    "file": "/home/pavel/testing/useful-test/tests/_isObject.test.js",
    "duration": 0,
    "currentRetry": 0,
    "err": {
      "message": "expected true to be false",
      "showDiff": true,
      "actual": "true",
      "expected": "false",
      "operator": "strictEqual",
      "stack": "AssertionError: expected true to be false\n    at Context.<anonymous> (file:///home/pavel/testing/useful-test/tests/_isObject.test.js:27:32)\n    at process.processImmediate (node:internal/timers:471:21)"
    }
  }
]

Running this script would produce this output:

$ scripts/find-failed-test-files.sh 
"tests/_getYyyyMmDd.test.js"
"tests/_isObject.test.js"

Now I know what test files to run :)

In package.json, there can be something simple like this script:

{
  "scripts": {
    "test": "mocha"
  }
}

And the scripts/run-tests.sh can look like:

#!/usr/bin/env bash

set -euo pipefail

if ! [[ -f results.json ]]; then
  npm run test "tests/.test.js"
else
  scripts/find-failed-test-files.sh | xargs npm run test --
fi

In other words, if we don’t have any cache, run all tests, if we do, get only failed test files and run those.

When I run my workflow, I can see Mocha was run like this:

mocha tests/*.test.js

First workflow run

When I click on “re-run failed jobs”:

Re-run failed jobs

I can see that Mocha was executed only with those test failed that failed the first time over:

> mocha tests/_getYyyyMmDd.test.js tests/_isObject.test.js

Second workflow run

That said, you can probably see there’s a bit more in the scripts/find-failed-test-files.sh script. There can be three scenarios I know of:

• some tests fail
• some tests fail, but Mocha doesn’t say in what files — file attribute in failures array is optional
• all tests passed but I still want to re-run test jobs

If you pay close attention, you realise that my current implementation I’ve shown up to this point does not cover the third point. Nothing would be run, which is likely not what I want. It’d be better to run everything again in this scenario, just to be on the safe side.

Let’s then change the scripts/run-tests.sh file a little:

#!/usr/bin/env bash

set -euo pipefail

# if cache is restored
if [[ -f results.json ]]; then
  failed_test_files="$(scripts/find-failed-test-files.sh)"

  # and some test files are found
  if [[ -n "$failed_test_files" ]]; then
    # run only those test files
    echo "$failed_test_files" | xargs npm run test --

    exit 0
  fi
fi

# run everything
npm run test "tests/*.test.js"

All in all, this can be a time saver. In my work context, this can save about ~6 minutes off a re-run, which is a significant time when it comes to pipeline runs.

If you find this useful and want to read more posts by me, please follow me here. Thank you.