News Feed

News Archive
feed this:

Looking for more information on how to do PHP the right way? Check out PHP: The Right Way

Pádraic Brady:
Introduction to Humbug A Mutation Testing Framework for PHP
April 08, 2015 @ 09:34:13

While he's mentioned it in other posts to his site, Pádraic Brady has officially posted an Introduction to Humbug to his site today. Humbug is a mutation testing framework that lets you determine the actual effectiveness of your unit tests through "mutation testing" methods.

You may already be familiar with the concept. In Mutation Testing, defects which emulate simple programmer errors are introduced into source code (your canonical code is untouched) and the relevant unit tests are run to see if they notice the defect. The more defects that are noticed, the more effective the test suite is presumed to be. The methodology relies on the theory that a quantity of relatively simple defects, either in isolation or combined, provide as much useful information as would a series of more complex defects.

He talks about the differences between mutation testing and the more traditional code coverage metrics. He points out that code coverage, while a decent high-level metric, should never be used as a quality metric. Using Humbug allows you to determine the real effectiveness and "coverage" of what you're testing. He then gets into how to use the tool, outlining:

  • Installation of the library as a phar
  • Generating a configuration file
  • Execute the command to run your tests (to ensure they're passing) and execute the mutation testing

The execution is broken into several stages: executing your tests for passing, breaking up the source into tokens to determine mutability, replacement of content with mutations in a temporary version of the source and a final execution of the test suite to determine the mutation results. He includes some example output from the tool on a moderately large codebase and how to interpret these results. He ends the post talking about the logs that Humbug generates, the overall performance of the tool and an experimental feature that's in the works called "Incremental Analysis".

0 comments voice your opinion now!
humbug mutation testing framework tool library introduction example


Pádraic Brady:
Lies, Damned Lies and Code Coverage Towards Mutation Testing
January 14, 2015 @ 12:20:29

In his latest post Pádraic Brady talks about lies, damned lies and code coverage and how that relates to something called "mutation testing."

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don't notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There's a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.

0 comments voice your opinion now!
mutation testing humbug unittest codecoverage lines


Dave Marshall:
Probing Test Suite Quality with Mutation Testing
January 08, 2015 @ 12:09:42

In this recent post to his site Dave Marshall looks at a method for evaluating the overall quality of your suite of unit tests with help from mutation testing.

100% code coverage should never really be a goal. [...] I feel pursuing 100% coverage in a PHP project is a particularly poor idea as our tooling generally only provides Line Coverage. [...] There are more reasonable coverage metrics to use to measure the quality of a test suite. Sebastian Bergmann and Derick Rethans are working hard on bringing some of these options to us, but for now we're limited to line coverage.

He talks about the difference between line, branch and condition coverage types (with code examples) and which allows for more effective and quality tests to be written. He then talks about the results of an experiment to achieve 100% coverage on the Router component in the Aura project. He found the problem using mutation testing - changing values in the production code to make sure the tests break. He also links over to a new mutation testing tool that's been released to help with this kind of thing, humbug, and some of the results it can report.

Mutation testing is a great thing to have a grasp of in theory, but it's not particularly easy to practice. The tools are very hard to write and then their output is often hard to understand or interpret effectively. I wouldn't recommend practicing with mutation testing on a regular basis, but it's certainly worth considering on the odd occasion.
0 comments voice your opinion now!
test suite quality mutation testing unittest coverage


Padraic Brady's Blog:
PHP Mutation Testing With MutateMe
February 24, 2009 @ 10:20:54

Padraic Brady has posted about a library that he's developed to help with mutation testing in PHP - MutateMe. What's mutation testing? Padraic explains,

Mutation Testing is basically testing...for tests. It ensures that your tests are truly capable of detecting errors and problems with the source code. It does this by mutating the source code itself (using ext/runkit) in such a way that an error is created in the code. If your tests detect the error, all is well with the world. If your tests do not detect the error...well, you better add a new test that does.

He mentions how it compares to code coverage and goes through the entire process you'll need to get MutateMe up and running - the download (or PEAR install), creating the extension, using the command line tool and the results of an example run of the client against a few tests.

0 comments voice your opinion now!
phpunit test unittest mutation codecoverage tutorial mutateme

Padraic Brady's Blog:
PHPMutagen Mutation Testing for PHP5
December 05, 2007 @ 09:32:00

In another behavior-driven development related post on his blog, Padraic Brady talks about mutation testing and a library he's put together to support it in PHP - PHPMutagen.

As I said in my previous entry I was thinking about how to write a Mutation Testing engine. The "braindump" was to use PHP's built in Tokenizer to break down a class file into digestible pieces which could be mutated, and then reconstructed into a mutated file. Once you allow for a working copy of the original source (let's not mutate the original!) it turned out to be a reasonable approach.

He's not releasing it just yet ("don't get too excited until I find some time to scan through it again") but he does share some sample output from his test run - one instance where all is well and the other where one mutant "escapes" and causes an exception.

0 comments voice your opinion now!
mutation testing php5 phpmutagen behavior development mutation testing php5 phpmutagen behavior development

Padraic Brady's Blog:
Mutation Testing Brain Dump
November 30, 2007 @ 10:26:00

Padraic Brady has made a post about some of the mutation testing thoughts running through his head for the Behaviour-Driven Development framework he's been working on - PHPSpec.

Mutation Testing is like testing, for tests. The idea is actually quite simple. Mutation testing adds small changes to your source code on the assumption that changing something, will most likely break it, which in turn means at least one test/spec should fail. [...] The reason why it's useful it that it gets around the problem of being over reliant on code coverage - just because some code executes when running tests, doesn't prove the tests will detect problems with it!

He wants to create a way that doesn't rely on external dependencies and is looking at working with the tokenizer functions native to PHP. He includes two examples of the way he's thinking - an example of simple token output and a way to go "backwards" to recreate the original source code.

0 comments voice your opinion now!
mutation testing unittest phpspec tokenizer mutation testing unittest phpspec tokenizer

Sebastian Bergmann's Blog:
Update on Mutation Testing
July 26, 2007 @ 07:54:00

Sebastian Bergmann has provided a pointer to the update Mike has posted concerning the Google Summer of Code project he's working on - implementing mutation testing in PHPUnit.

From Mike's update:

The most recent version committed to the mutation_testing branch is in a testable state. I would encourage anyone interested to try out the new mutation testing feature in this branch.

You can get complete information about the project - its methods and goals - from this page on the PHPUnit wiki. If you'd like to get this new code a try and see what his mutation testing is all about, here's a link to the subversion repository to grab the latest copy.

0 comments voice your opinion now!
mutation testing phpunit google summerofcode project mutation testing phpunit google summerofcode project

Sebastian Bergmann's Blog:
PHPUnit Status Update
June 25, 2007 @ 08:22:00

Sebastian Bergmann has posted an update of the status of the PHPUnit project to his blog today including feature additions to PHPUnit 3.1 and ones being added to other branches.

The last weeks have been busy, but now I want to take the time to finally give you an update on the status of PHPUnit. The development of PHPUnit 3.1 is progressing nicely, about the only thing missing is finishing the documentation for all the new features

Among these new features are things like:

0 comments voice your opinion now!
phpunit status project mockobject selenium templatemethod database mutation phpunit status project mockobject selenium templatemethod database mutation

Sebastian Bergmann's Blog:
Google Summer of Code 2007
April 16, 2007 @ 07:52:00

Another mentor for another Google Summer of Code project has been announced and Sebastian Bergmann will be mentoring Mike Lewis as he adds mutation testing for PHPUnit.

Sebastian quotes a definition of "mutation testing" as:

Mutation Testing, or Automated Error Seeding, is an approach where the testing tool makes some change to the tested code, runs the tests, and if the tests pass displays a message saying what it changed.

The key difference is that it makes it easier to run a series of tests rather than just one at a time, aiding in more complex testing abilities. Check out this page on the PHPUnit site for more information.

0 comments voice your opinion now!
googlesummerofcode2007 phpunit mutation testing unittesting googlesummerofcode2007 phpunit mutation testing unittesting

Sebastian Bergmann's Blog:
Ideas for PHPUnit Projects
February 26, 2007 @ 11:26:00

Always looking forward to the future of PHPUnit, Sebastian Bergmann has started gathering some ideas for future additions to the project:

I started to collect ideas for features that I would like to see in PHPUnit but currently lack the time to implement myself.

The two listed in this post are:

  • Mutation testing - an approach where the testing tool makes some change to the tested code, runs the tests, and if the tests pass displays a message saying what it changed
  • Web Interface for Test Result Database - output viewer for the database logging PHPUnit is now capable of.
Looking to help out? Let him know!

0 comments voice your opinion now!
phpunit project future mutation unittest webinterface test result database phpunit project future mutation unittest webinterface test result database

Community Events

Don't see your event here?
Let us know!

composer opinion project community application example php7 symfony programming yii2 framework part2 introduction series laravel interview podcast api list language

All content copyright, 2015 :: - Powered by the Solar PHP Framework