While he's mentioned it in other posts to his site, Pádraic Brady has officially posted an Introduction to Humbug to his site today. Humbug is a mutation testing framework that lets you determine the actual effectiveness of your unit tests through "mutation testing" methods.

You may already be familiar with the concept. In Mutation Testing, defects which emulate simple programmer errors are introduced into source code (your canonical code is untouched) and the relevant unit tests are run to see if they notice the defect. The more defects that are noticed, the more effective the test suite is presumed to be. The methodology relies on the theory that a quantity of relatively simple defects, either in isolation or combined, provide as much useful information as would a series of more complex defects.

He talks about the differences between mutation testing and the more traditional code coverage metrics. He points out that code coverage, while a decent high-level metric, should never be used as a quality metric. Using Humbug allows you to determine the real effectiveness and "coverage" of what you're testing. He then gets into how to use the tool, outlining:

• Installation of the library as a phar
• Generating a configuration file
• Execute the command to run your tests (to ensure they're passing) and execute the mutation testing

The execution is broken into several stages: executing your tests for passing, breaking up the source into tokens to determine mutability, replacement of content with mutations in a temporary version of the source and a final execution of the test suite to determine the mutation results. He includes some example output from the tool on a moderately large codebase and how to interpret these results. He ends the post talking about the logs that Humbug generates, the overall performance of the tool and an experimental feature that's in the works called "Incremental Analysis".