Kévin Gomez has a recent post to his site sharing some of the knowledge he gained when digging into Humbug, a mutation testing tool for PHP development.

While I’ve already used Humbug a few time, a recent article made my realise that I didn’t really know how it worked.

That’s when I got the idea to dig into Humbug to learn how it works, and publish my findings here.

He starts with a brief overview of Humbug for those not familiar with it - a mutation testing tool that reviews your unit tests to see how well they actually cover your code. It performs various operations (mutations) on the tests and evaluates the response. He then gets into how Humbug does this and what tools it uses to break down and understand your tests. He then goes through the actual code of the tool, walking through the tests, tokenizing the code and performing small changes to re-test and see how the results differ from the original results.

In a tutorial posted to the SitePoint PHP blog Claudio Ribeiro tries to answer the question of "who watches the watchmen" (your application's tests) to ensure they're functioning as expected and are correct. In this new tutorial he introduces the Humbug mutation testing tool and how it can be used to verify your own tests.

Regardless of whether you’re working for a big corporation, a startup, or just for yourself, unit testing is not only helpful, but often indispensable. We use unit tests to test our code, but what happens if our tests are wrong or incomplete? What can we use to test our tests? Who watches the watchmen?

[...] Mutation Testing ( or Mutant Analysis ) is a technique used to create and evaluate the quality of software tests. It consists of modifying the tests in very small ways. Each modified version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Mutations are bugs in our original code and analysis checks if our tests detect those bugs. In a nutshell, if a test still works after it’s mutated, it’s not a good test.

He starts by helping you get it installed (a quick composer require) and creating a simple "calculator" test to show it in use. He then creates the test for the class with some simple testing methods for the basic calculator functionality. He then configures the Humbug installation (via a JSON config file) and executes it on the current tests, sharing the resulting output. He goes through the results showing how to interpret them and points out places where the tests could be improved.

While he's mentioned it in other posts to his site, Pádraic Brady has officially posted an Introduction to Humbug to his site today. Humbug is a mutation testing framework that lets you determine the actual effectiveness of your unit tests through "mutation testing" methods.

You may already be familiar with the concept. In Mutation Testing, defects which emulate simple programmer errors are introduced into source code (your canonical code is untouched) and the relevant unit tests are run to see if they notice the defect. The more defects that are noticed, the more effective the test suite is presumed to be. The methodology relies on the theory that a quantity of relatively simple defects, either in isolation or combined, provide as much useful information as would a series of more complex defects.

He talks about the differences between mutation testing and the more traditional code coverage metrics. He points out that code coverage, while a decent high-level metric, should never be used as a quality metric. Using Humbug allows you to determine the real effectiveness and "coverage" of what you're testing. He then gets into how to use the tool, outlining:

• Installation of the library as a phar
• Generating a configuration file
• Execute the command to run your tests (to ensure they're passing) and execute the mutation testing

The execution is broken into several stages: executing your tests for passing, breaking up the source into tokens to determine mutability, replacement of content with mutations in a temporary version of the source and a final execution of the test suite to determine the mutation results. He includes some example output from the tool on a moderately large codebase and how to interpret these results. He ends the post talking about the logs that Humbug generates, the overall performance of the tool and an experimental feature that's in the works called "Incremental Analysis".

In his latest post Pádraic Brady talks about lies, damned lies and code coverage and how that relates to something called "mutation testing."

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don’t notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There’s a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.