Cache Mocha Test Results And Run Only Failed Test Files On Re-runs

Mocha can generate a JSON file with test results. It might be sometimes useful to find all failed test files, e.g. when I want to re-run only tests from such test files.

Running all automated tests might be time-consuming, so you might want to look for various ways to run only what’s necessary.

One such scenario might be that you run all tests. Some of them fail. You investigate (you should always do that) but realise that those tests failed for reasons you can’t really fix straight away. You decide to run the tests again. But this time, you might want to run only those tests files that contain tests that failed the first time over.

This is something that I implemented in GitHub pipelines. On re-runs, I tell Mocha to run only those test files that contain tests that failed in the previous run. For that, there’re two things I need to sort out:

  • caching of the Mocha result file
  • parsing the Mocha result file, finding failed test files and handing them to Mocha

Regarding the Mocha result file, I wrote more on that in this article, you can get an idea on how the file structure looks like.

On to the two points.

The caching part can be a bit tricky, because the famous actions/cache@v3 action does not cache anything when any step in a workflow fails. That is a major drawback because I want to do exactly that.

There’s hopefully a solution with its beta version v3.2.0-beta.1 (there are more beta versions as of December 2022, you might want to check those other ones as well). My workflow can then look like this (a simplified example from my mock repo):

Each run will create a new cache that will contain the Mocha test result file, it will also try to restore the Mocha file from the previous run.

In caches, it can look like this in my mock repo:

Created caches with Mocha test result file
Created caches with Mocha test result file

Now on to the second point. If I actually retrieve cache, I want to hand to Mocha only those test files in which some tests failed. It can be done with a little Bash that puts together a few utilities:

Let’s say that the failures in Mocha’s test result file look like this:

Running this script would produce this output:

Now I know what test files to run :)

In package.json, there can be something simple like this script:

And the scripts/run-tests.sh can look like:

In other words, if we don’t have any cache, run all tests, if we do, get only failed test files and run those.

When I run my workflow, I can see Mocha was run like this:

First workflow run
First workflow run

When I click on “re-run failed jobs”:

Re-run failed jobs
Re-run failed jobs

I can see that Mocha was executed only with those test failed that failed the first time over:

Second workflow run
Second workflow run

That said, you can probably see there’s a bit more in the scripts/find-failed-test-files.sh script. There can be three scenarios I know of:

  • some tests fail
  • some tests fail, but Mocha doesn’t say in what files — file attribute in failures array is optional
  • all tests passed but I still want to re-run test jobs

If you pay close attention, you realise that my current implementation I’ve shown up to this point does not cover the third point. Nothing would be run, which is likely not what I want. It’d be better to run everything again in this scenario, just to be on the safe side.

Let’s then change the scripts/run-tests.sh file a little:

All in all, this can be a time saver. In my work context, this can save about ~6 minutes off a re-run, which is a significant time when it comes to pipeline runs.

If you find this useful and want to read more posts by me, please follow me here. Thank you.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store