In a new post to his site Matthias Noback has a post showing you how to, when using the Behat functional testing framework, to create local and remote code coverage metrics.

PHPUnit has built-in several options for generating code coverage data and reports. Behat doesn't. As Konstantin Kudryashov (@everzet) points out in an issue asking for code coverage options in Behat: "Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you're doing code testing with StoryBDD - you're doing it wrong."

He starts off by talking about code coverage and why you might want it for the tests created through Behat. Code coverage metrics are usually associated with unit tests but having those numbers for the functional tests can be helpful too. He then covers the two pieces of the puzzle needed to gather the coverage numbers: an extension for the local coverage and one for the remote coverage. He shows how to get both of these installed, configured and used in the code to gather the results. Finally he shows how to use the phpcov tool to merge these results with the PHPUnit results to get a better overall view of coverage numbers.

In this post on his Medium page Elton Minetto shows how to generate the code coverage of your PHPunit tests with better performance using phpdbg.

In a previous post (in portuguese) I explained how to identify tests that are taking too long to execute. In this post, I’ll show you how to increase the performance of code coverage report generation using PHPUnit.

In the phpunit.xml file it’s possible to add configurations to generate reports related to the tests that are being executed. [...] In addition to changing the phpunit.xml file, to generate this information we also need to install the extension XDebug. However, by installing it we get a substantial decrease in performance.

He shows an example of the time difference in running the tests (about 1 minute without versus 22 with XDebug). He went looking for a better way and found this post talking about using phpdbg instead. He includes the "brew" commands to get everything you'll need installed and how to use phpdbg with your coverage calls rather than XDebug. However, as is pointed out at the end of the post, the results are slightly different but they're close enough to help you know what code to target next.

Sebastian Bergmann has posted about some updates in the latest version of the PHPUnit PHP unit testing tool (v4.8) and changes in code coverage handling.

PHPUnit 4.8 introduces a couple of small improvements. For instance, the @testWith annotation was added as "syntactic sugar" for defining data providers and the --no-coverage commandline option was added for ignoring any code coverage configuration from the configuration file.

PHP_CodeCoverage, the library used by PHPUnit to collect, process, and report code coverage information, has been updated for PHPUnit 4.8. It can now collect code coverage information on PHP 7 without the need for a third-party extension such as Xdebug. And PHP_CodeCoverage's HHVM driver has been updated to reflect changes to how HHVM provides code coverage information.

He gets into a bit more detail about the updates to both the PHP 7 handling and changes for HHVM, including how to call it for PHP 7 (phpdbg vs just the normal CLI) and how HHVM no exposes its coverage information publicly. This makes it simpler for PHPUnit to grab the data without special handling code. He also looks ahead to PHPUnit 5, scheduled for release in October 2015, and that some functionality will be deprecated when the version is changed.

Freek Lijten has a new post today continuing his look at the world of PHP extensions and focusing in on testing this time. He hopes to answer the question of what makes a good, effective set of tests to help increase the stability and quality of the extensions you write.

In my previous blog I took you through the process of getting PHP and extensions compiled, generating code coverage and running tests. What I did not talk about was what makes a good test. I hope to correct on this by adding this post and going into more detail on the actual writing of tests itself.

Using the same extension as before (enchant) he goes through the addition of a test for the enchant_dict_add_to_session function. He start by showing how much the function is currently tested (hint: none) and code coverage. He points out that 100% coverage is just one metric in a set that should be considered and not the final goal. He shares a simple test for the function that checks to see if a certain word exists in a dictionary. The coverage report shows all lines being executed, but there's a lot not tested, at least conceptually. He shows how to test "the spirit" of the function with additional tests for non-existent words, spell checking and if a word is not in the dictionary at all. PHP example code shows these tests kinds of tests to illustrate the steps he's talking about.

In his latest post Pádraic Brady talks about lies, damned lies and code coverage and how that relates to something called "mutation testing."

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don’t notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There’s a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.

Derick Rethans has posted the first in a series focusing on the Xdebug tool and the code coverage functionality it can provide via PHPUnit's testing. In this first post he catches the reader up on the current state of things and what all the Xdebug tool can do.

Since ages Xdebug has provided code coverage support for PHPUnit, a way to show which lines are covered by your test cases. But I never really wrote about how it works. A recently filed bug prompted me to write this post, as well as a follow up post on Code Coverage's future.

He starts off with the early days of Xdebug, how it hooked into the Zend Engine (that powers a lot of PHP behind the scenes) and when it was triggered. This came with its own set of problems so Xdebug was updated to overload some opcodes. He talks about how it can calculate the unused lines and determines which lines can be covered in the code coverage results. He provides some example code showing the execution of the coverage report on a simple function and try/catch handler, complete with the HTML output of the results.

There's a good post over on the NetTuts.com site today talking about unit testing, the myths around code coverage and why it may not be as important as you think.

Most cubicle workplaces disappeared and programmers started loving their craft. With the advent of Agile techniques and the Software Craftsmanship movement, many new tools emerged to help the programmer and the process. TDD is slowly becoming the de facto way of writing code and the secrets of SCRUM or Kanban were revealed even to the programmers in the darkest corners of the cubicle world. Automated testing and test driven development (TDD) are some of the essential techniques Agile provided to us programmers. And a tool that comes with those methodologies is used to produce test code coverage, which is the topic of this article.

The article starts with a brief definition of code coverage and gets right into an example class, PHPUnit test and the results of a code coverage generation. They show both output options, the text-only output and the full HTML output with clickable links and visualization of the covered lines of code. There's also an example of generating the coverage inside an IDE (PHPStorm). The post finishes with a look at the myths of code coverage including: "100% covered is bug free" and that "gaming the system" is pretty easy.

In her latest post Anna Filina proposes that full test coverage is an impractical way to measure the quality of your software. It can provide a false sense of security, even if the tests are poorly written.

Many developers claim that to achieve high quality software, developers must create automated tests that ensure that all possible execution routes have been covered. This is also known as full path coverage. I will argue that different types of software require different testing approaches and that full path coverage is impractical in almost every case. Too many tests simply create clutter.

She looks at how it's impractical to expect that all tests will be written efficiently or even correctly. Even simple tests are enough to show up on code coverage reports but may only be painting part of the picture. She also notes that not all software can be tested the same way - things like APIs require different testing skills/methods than something like consumer software.

In the end, there are no exact rules on how much to test. The most important thing to keep in mind is that writing tests for the sake writing tests is futile and costly. [...] Focus on building great software. Tests are a tool to make it better. Just don’t overdo it.

The Codeception testing tool has released a new major update with some interesting new features - the expected feature for generating code coverage reports (similar to other tools) but there's also the idea of "remote code coverage" introduced.

There is no magic in local codecoverage. XDebug and PHP_CodeCoverage libraries do their job. The tricky thing is remote codecoverage. We attach small script into application's front controller. When a special header is sent this script starts to collect coverage information. And in the end of tests, this data is merged, serialized and sent back to Codeception. So you can test and collect coverage report even on staging servers in real environment.

They also mention a few other updates in the release - new Redis and MongoDb modules, UX improvements and the normal bugfixes. You can find out more about the code coverage feature in their manual or just about the project in general from the main site.

Derick Rethans has a new post to his blog today talking about some work that's been done to speed up XDebug's code coverage generation. Changes in the coming 2.2 release have some improvements that make things perform better and put less stress on PHP in the process.

Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

These changes were from a combination of contributions from Taavi Burns and a new ini setting that will allow you to enable or disable the code coverage in XDebug. Benchmarking shows a good amount of time reduction in coverage runs - dropping anywhere from a few seconds to over a minute. He also mentions the idea of "modes", shortcuts to predefined settings for different types of reporting (like "profiling" or "tracing").