Alejandro Celaya has a post on his site that shows how to use a less well-known testing tool - mutation testing - to test for variations on the "good" and "bad" data paths. In this article he makes use of the infection library that replaced the previously active Humbug library.

There's no doubt that having tests in a project allows you to find potential bugs earlier and more easily.

Lots of OSS projects require a minimum code coverage in order to accept new pull requests from contributors, and proprietary projects also tend to have some sort of continuous integration workflow which requires certain metrics to be fulfilled in order to get builds passing. However, the code coverage can lead to a false sense of security, which makes you think that if a certain class has a 100% code coverage, it is also 100% bug-free.

This is not always true since you could be calling a method and yet not being properly testing its output or its real behavior. The code coverage will mark it as covered, but you might introduce a bug and still have a green test. This is where mutation testing comes in.

He starts by briefly introducing the concepts of mutation testing and showing how to get the infection library installed and configured. He then gives a guide on running the tool and some of the command line options that can be used to configure threading, having it only run on covered code and setting the log verbosity. He then offers some advice on troubleshooting the use of the tool and how phpdbg is used to generate reports.

The MyBuilder.com Tech blog has a new post from Edd Mann looking at designing immutable concepts in PHP with "transient mutation".

In a recent project we felt it beneficial to introduce the Money pattern. There are many good resources on this pattern, so I will delegate to those for further definition. We decided that encapsulating this into a immutable value object allowed for a cleaner API and removed the fear of any unexpected mutation bugs. However, we noticed a spike in memory and processor usage when wishing to perform many successive actions on such values i.e. summation.

In such a case, new ‘temporary’ Money objects would be instantiated upon each applied addition. As many of these objects were simply a stepping stone to generating the final result, they were just left for the Garbage collector to clean-up.

The idea of "transients" is something pulled from the Clojure language allowing for the mutation of the immutable object and returning a new immutable one....with with some controls, not free form mutation. He includes showing an example of a "Money" object that implements this concept including a "withMutable" method that handles a callback for the mutation operation.

Kévin Gomez has a recent post to his site sharing some of the knowledge he gained when digging into Humbug, a mutation testing tool for PHP development.

While I’ve already used Humbug a few time, a recent article made my realise that I didn’t really know how it worked.

That’s when I got the idea to dig into Humbug to learn how it works, and publish my findings here.

He starts with a brief overview of Humbug for those not familiar with it - a mutation testing tool that reviews your unit tests to see how well they actually cover your code. It performs various operations (mutations) on the tests and evaluates the response. He then gets into how Humbug does this and what tools it uses to break down and understand your tests. He then goes through the actual code of the tool, walking through the tests, tokenizing the code and performing small changes to re-test and see how the results differ from the original results.

In a tutorial posted to the SitePoint PHP blog Claudio Ribeiro tries to answer the question of "who watches the watchmen" (your application's tests) to ensure they're functioning as expected and are correct. In this new tutorial he introduces the Humbug mutation testing tool and how it can be used to verify your own tests.

Regardless of whether you’re working for a big corporation, a startup, or just for yourself, unit testing is not only helpful, but often indispensable. We use unit tests to test our code, but what happens if our tests are wrong or incomplete? What can we use to test our tests? Who watches the watchmen?

[...] Mutation Testing ( or Mutant Analysis ) is a technique used to create and evaluate the quality of software tests. It consists of modifying the tests in very small ways. Each modified version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Mutations are bugs in our original code and analysis checks if our tests detect those bugs. In a nutshell, if a test still works after it’s mutated, it’s not a good test.

He starts by helping you get it installed (a quick composer require) and creating a simple "calculator" test to show it in use. He then creates the test for the class with some simple testing methods for the basic calculator functionality. He then configures the Humbug installation (via a JSON config file) and executes it on the current tests, sharing the resulting output. He goes through the results showing how to interpret them and points out places where the tests could be improved.

While he's mentioned it in other posts to his site, Pádraic Brady has officially posted an Introduction to Humbug to his site today. Humbug is a mutation testing framework that lets you determine the actual effectiveness of your unit tests through "mutation testing" methods.

You may already be familiar with the concept. In Mutation Testing, defects which emulate simple programmer errors are introduced into source code (your canonical code is untouched) and the relevant unit tests are run to see if they notice the defect. The more defects that are noticed, the more effective the test suite is presumed to be. The methodology relies on the theory that a quantity of relatively simple defects, either in isolation or combined, provide as much useful information as would a series of more complex defects.

He talks about the differences between mutation testing and the more traditional code coverage metrics. He points out that code coverage, while a decent high-level metric, should never be used as a quality metric. Using Humbug allows you to determine the real effectiveness and "coverage" of what you're testing. He then gets into how to use the tool, outlining:

• Installation of the library as a phar
• Generating a configuration file
• Execute the command to run your tests (to ensure they're passing) and execute the mutation testing

The execution is broken into several stages: executing your tests for passing, breaking up the source into tokens to determine mutability, replacement of content with mutations in a temporary version of the source and a final execution of the test suite to determine the mutation results. He includes some example output from the tool on a moderately large codebase and how to interpret these results. He ends the post talking about the logs that Humbug generates, the overall performance of the tool and an experimental feature that's in the works called "Incremental Analysis".

In his latest post Pádraic Brady talks about lies, damned lies and code coverage and how that relates to something called "mutation testing."

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don’t notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There’s a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.

In this recent post to his site Dave Marshall looks at a method for evaluating the overall quality of your suite of unit tests with help from mutation testing.

100% code coverage should never really be a goal. [...] I feel pursuing 100% coverage in a PHP project is a particularly poor idea as our tooling generally only provides Line Coverage. [...] There are more reasonable coverage metrics to use to measure the quality of a test suite. Sebastian Bergmann and Derick Rethans are working hard on bringing some of these options to us, but for now we're limited to line coverage.

He talks about the difference between line, branch and condition coverage types (with code examples) and which allows for more effective and quality tests to be written. He then talks about the results of an experiment to achieve 100% coverage on the Router component in the Aura project. He found the problem using mutation testing - changing values in the production code to make sure the tests break. He also links over to a new mutation testing tool that's been released to help with this kind of thing, humbug, and some of the results it can report.

Mutation testing is a great thing to have a grasp of in theory, but it's not particularly easy to practice. The tools are very hard to write and then their output is often hard to understand or interpret effectively. I wouldn't recommend practicing with mutation testing on a regular basis, but it's certainly worth considering on the odd occasion.

Padraic Brady has posted about a library that he's developed to help with mutation testing in PHP - MutateMe. What's mutation testing? Padraic explains,

Mutation Testing is basically testing...for tests. It ensures that your tests are truly capable of detecting errors and problems with the source code. It does this by mutating the source code itself (using ext/runkit) in such a way that an error is created in the code. If your tests detect the error, all is well with the world. If your tests do not detect the error...well, you better add a new test that does.

He mentions how it compares to code coverage and goes through the entire process you'll need to get MutateMe up and running - the download (or PEAR install), creating the extension, using the command line tool and the results of an example run of the client against a few tests.

In another behavior-driven development related post on his blog, Padraic Brady talks about mutation testing and a library he's put together to support it in PHP - PHPMutagen.

As I said in my previous entry I was thinking about how to write a Mutation Testing engine. The "braindump" was to use PHP's built in Tokenizer to break down a class file into digestible pieces which could be mutated, and then reconstructed into a mutated file. Once you allow for a working copy of the original source (let's not mutate the original!) it turned out to be a reasonable approach.

He's not releasing it just yet ("don't get too excited until I find some time to scan through it again") but he does share some sample output from his test run - one instance where all is well and the other where one mutant "escapes" and causes an exception.

In another behavior-driven development related post on his blog, Padraic Brady talks about mutation testing and a library he's put together to support it in PHP - PHPMutagen.

As I said in my previous entry I was thinking about how to write a Mutation Testing engine. The "braindump" was to use PHP's built in Tokenizer to break down a class file into digestible pieces which could be mutated, and then reconstructed into a mutated file. Once you allow for a working copy of the original source (let's not mutate the original!) it turned out to be a reasonable approach.

He's not releasing it just yet ("don't get too excited until I find some time to scan through it again") but he does share some sample output from his test run - one instance where all is well and the other where one mutant "escapes" and causes an exception.