Kévin Gomez has a recent post to his site sharing some of the knowledge he gained when digging into Humbug, a mutation testing tool for PHP development.

While I’ve already used Humbug a few time, a recent article made my realise that I didn’t really know how it worked.

That’s when I got the idea to dig into Humbug to learn how it works, and publish my findings here.

He starts with a brief overview of Humbug for those not familiar with it - a mutation testing tool that reviews your unit tests to see how well they actually cover your code. It performs various operations (mutations) on the tests and evaluates the response. He then gets into how Humbug does this and what tools it uses to break down and understand your tests. He then goes through the actual code of the tool, walking through the tests, tokenizing the code and performing small changes to re-test and see how the results differ from the original results.

In a tutorial posted to the SitePoint PHP blog Claudio Ribeiro tries to answer the question of "who watches the watchmen" (your application's tests) to ensure they're functioning as expected and are correct. In this new tutorial he introduces the Humbug mutation testing tool and how it can be used to verify your own tests.

Regardless of whether you’re working for a big corporation, a startup, or just for yourself, unit testing is not only helpful, but often indispensable. We use unit tests to test our code, but what happens if our tests are wrong or incomplete? What can we use to test our tests? Who watches the watchmen?

[...] Mutation Testing ( or Mutant Analysis ) is a technique used to create and evaluate the quality of software tests. It consists of modifying the tests in very small ways. Each modified version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Mutations are bugs in our original code and analysis checks if our tests detect those bugs. In a nutshell, if a test still works after it’s mutated, it’s not a good test.

He starts by helping you get it installed (a quick composer require) and creating a simple "calculator" test to show it in use. He then creates the test for the class with some simple testing methods for the basic calculator functionality. He then configures the Humbug installation (via a JSON config file) and executes it on the current tests, sharing the resulting output. He goes through the results showing how to interpret them and points out places where the tests could be improved.