James Craig

Ways to Improve Your Development Process - Mutation Testing

Mutation, it is the key to our evolution.
Jul 28 2021 by James Craig

One of the things that I've noticed over the years is that our industry tends to be very set in our ways. We hear about a new paradigm or whatever buzz word you want to throw at a concept and we ignore it generally speaking. But give someone an untested new JavaScript library and we're all over it. Testing is generally one of those areas that everyone means to do well and then it falls off a cliff never to be seen again. This is especially true when we have to do example based testing, which basic unit testing is more or less. We give a static input, we expect a static output that equals X. The past couple of posts that I've made have gone over a couple of ways to improve test coverage of the code that we want to ship. Fuzzing can be used to find edge cases or unexpected inputs. Property testing can be used to simplify the example based testing. The joy is that we can add both of them to our setup with minimal cost to us on the development side and they'll find a ton of hidden issues in our code. But what about our tests? How do we know that they're good?

This is one of those areas that a lot of research is being done but it hasn't necessarily made its way to the average developer yet. But there is an easy way to test our tests without too much effort on our part: Mutation testing.

Mutation Testing #

Mutation testing is pretty simple conceptually speaking. It's a process where faults are automatically added to your code. The addition of these faults makes the resulting code into something known as a mutation and is accomplished by doing things like changing an if statement to be greater than instead of less than or addition instead of subtraction. After we make our change, we run the tests. If your tests fail then the mutation is killed and if your tests pass then the mutation lived.

OK, so how does this help us? Well with this we can measure the quality of our tests by figuring out the percentage of mutations that were killed. The more mutations killed, the better our tests are at finding issues in our code base.

As an example, let's assume we have a code base with traditional unit tests. We can even have 100% branch code coverage in this example. We then make a change in our code and everything passes on the tests BUT as soon as we try to run this thing in production, it crashes and burns. This could be due to issues with libraries that we are linking to or a configuration issue. Similarly it could also be due to us making assumptions in our test suite that pass but are bad assumptions.

    Assert.Equal(1, TestObject.MakeObj(2).SomeProperty);

In the above code we have no real knowledge of how SomeProperty gets set to 1. Maybe it's simply taking the 1 we sent in and assigning it or maybe it's a more complex formula. Perhaps it's not even based on the value that we sent in. In any case our one test may pass but it's probably not that great of a test. Mutation testing would help point this out to us. So what's the pros/cons of mutation testing?

Pros

  1. It has been used with success to find bugs in code and is generally better than alternative methods for finding issues with unit tests.

  2. It also leads to improved code coverage quite a bit.

Cons

  1. It's slow. Really slow sometimes, with much of the research being aimed at speeding up the process.
  2. There is potential for getting fatigue from duplicate issues being reported to the developer. A real issue and area of research is finding effective ways to detect equivalent and redundant mutants. Not doing this leads to developers being inundated with the same info over and over again.

So considering the aim of this type of testing and the speed issues, this isn't something that you would run every build. This is more of a check up on your code base that you run late at night or over a weekend to see if it kicks up anything. That said, it's usually fire and forget until it finishes so the trade off is usually worth it.

Mutation Testing Tools #

There are a surprising number of tools in this area, especially considering the lack of use that mutation testing gets in the average corporate dev job. The one that I like to use is Stryker Mutator. It supports JavaScript, TypeScript, Scala, and C#. It tends to run rather quickly relatively speaking, supports most test runners, and has good reporting capabilities.

In the case of C#, it installs as a dotnet tool and from there you just run this from your unit test library:

dotnet stryker

And that's it. It will then go and try to find issues with your code. With that you have mutation testing now in your pipeline and another tool to help reduce issues in your code.

Items in the Series #

  1. Unit Testing and Automation
  2. Fuzzing
  3. Property-Based Testing
  4. Mutation Testing
  5. Fault Injection
Unit Testing Mutation Testing