Looking for more information on how to do PHP the right way? Check out PHP: The Right Way

ThePHP.cc:
PHPUnit 4.8: Code Coverage Support
Aug 10, 2015 @ 08:48:52

Sebastian Bergmann has posted about some updates in the latest version of the PHPUnit PHP unit testing tool (v4.8) and changes in code coverage handling.

PHPUnit 4.8 introduces a couple of small improvements. For instance, the @testWith annotation was added as "syntactic sugar" for defining data providers and the --no-coverage commandline option was added for ignoring any code coverage configuration from the configuration file.

PHP_CodeCoverage, the library used by PHPUnit to collect, process, and report code coverage information, has been updated for PHPUnit 4.8. It can now collect code coverage information on PHP 7 without the need for a third-party extension such as Xdebug. And PHP_CodeCoverage's HHVM driver has been updated to reflect changes to how HHVM provides code coverage information.

He gets into a bit more detail about the updates to both the PHP 7 handling and changes for HHVM, including how to call it for PHP 7 (phpdbg vs just the normal CLI) and how HHVM no exposes its coverage information publicly. This makes it simpler for PHPUnit to grab the data without special handling code. He also looks ahead to PHPUnit 5, scheduled for release in October 2015, and that some functionality will be deprecated when the version is changed.

tagged: phpunit codecoverage support php7 hhvm changes phpdbg public

Link: https://thephp.cc/news/2015/08/phpunit-4-8-code-coverage-support

Freek Lijten:
Testing PHP extensions - what makes a good test
Mar 23, 2015 @ 09:52:58

Freek Lijten has a new post today continuing his look at the world of PHP extensions and focusing in on testing this time. He hopes to answer the question of what makes a good, effective set of tests to help increase the stability and quality of the extensions you write.

In my previous blog I took you through the process of getting PHP and extensions compiled, generating code coverage and running tests. What I did not talk about was what makes a good test. I hope to correct on this by adding this post and going into more detail on the actual writing of tests itself.

Using the same extension as before (enchant) he goes through the addition of a test for the enchant_dict_add_to_session function. He start by showing how much the function is currently tested (hint: none) and code coverage. He points out that 100% coverage is just one metric in a set that should be considered and not the final goal. He shares a simple test for the function that checks to see if a certain word exists in a dictionary. The coverage report shows all lines being executed, but there's a lot not tested, at least conceptually. He shows how to test "the spirit" of the function with additional tests for non-existent words, spell checking and if a word is not in the dictionary at all. PHP example code shows these tests kinds of tests to illustrate the steps he's talking about.

tagged: test extension phpt spirit codecoverage metric goal

Link: http://www.freeklijten.nl/home/2015/03/22/Testing-PHP-extensions-what-makes-a-good-test

Pádraic Brady:
Lies, Damned Lies and Code Coverage: Towards Mutation Testing
Jan 14, 2015 @ 12:20:29

In his latest post Pádraic Brady talks about lies, damned lies and code coverage and how that relates to something called "mutation testing."

I spent the vast majority of 2014 not contributing to open source, so I kicked off 2015 by making Humbug available on Github. Humbug is a Mutation Testing framework for PHP. Essentially, it injects deliberate defects into your source code, designed to emulate programmer errors, and then checks whether your unit tests notice. If they notice, good. If they don’t notice, bad. All quite straightforward. [...] This article however is mostly reserved to explain why I wrote Humbug, and why Mutation Testing is so badly needed in PHP. There’s a few closing words on Mutation Testing performance which has traditionally been a concern impeding its adoption.

He starts off by talking about the idea of "code coverage" when it comes to writing unit tests...and how 100% coverage usually ends up being a lie. He points out that the current methods of line-based coverage metrics can lead to false results and that it's more about test quality rather than volume of tests. He then moves into talking about mutation testing and where it fits in the test quality puzzle. Finally, he mentions one thing to watch out for when trying out mutation testing and the performance jump (longer, not shorter) it can introduce into your testing cycle.

tagged: mutation testing humbug unittest codecoverage lines

Link: http://blog.astrumfutura.com/2015/01/lies-damned-lies-and-code-coverage-towards-mutation-testing/

Derick Rethans:
Code Coverage: The Present
Dec 02, 2014 @ 11:54:01

Derick Rethans has posted the first in a series focusing on the Xdebug tool and the code coverage functionality it can provide via PHPUnit's testing. In this first post he catches the reader up on the current state of things and what all the Xdebug tool can do.

Since ages Xdebug has provided code coverage support for PHPUnit, a way to show which lines are covered by your test cases. But I never really wrote about how it works. A recently filed bug prompted me to write this post, as well as a follow up post on Code Coverage's future.

He starts off with the early days of Xdebug, how it hooked into the Zend Engine (that powers a lot of PHP behind the scenes) and when it was triggered. This came with its own set of problems so Xdebug was updated to overload some opcodes. He talks about how it can calculate the unused lines and determines which lines can be covered in the code coverage results. He provides some example code showing the execution of the coverage report on a simple function and try/catch handler, complete with the HTML output of the results.

tagged: xdebug codecoverage phpunit coverage history functionality opcode

Link: http://derickrethans.nl/code-coverage.html

NetTuts.com:
Test Code Coverage: From Myth to Reality
Mar 31, 2014 @ 11:50:09

There's a good post over on the NetTuts.com site today talking about unit testing, the myths around code coverage and why it may not be as important as you think.

Most cubicle workplaces disappeared and programmers started loving their craft. With the advent of Agile techniques and the Software Craftsmanship movement, many new tools emerged to help the programmer and the process. TDD is slowly becoming the de facto way of writing code and the secrets of SCRUM or Kanban were revealed even to the programmers in the darkest corners of the cubicle world. Automated testing and test driven development (TDD) are some of the essential techniques Agile provided to us programmers. And a tool that comes with those methodologies is used to produce test code coverage, which is the topic of this article.

The article starts with a brief definition of code coverage and gets right into an example class, PHPUnit test and the results of a code coverage generation. They show both output options, the text-only output and the full HTML output with clickable links and visualization of the covered lines of code. There's also an example of generating the coverage inside an IDE (PHPStorm). The post finishes with a look at the myths of code coverage including: "100% covered is bug free" and that "gaming the system" is pretty easy.

tagged: codecoverage unittest phpunit tutorial introduction

Link: http://code.tutsplus.com/articles/test-code-coverage-from-myth-to-reality--cms-20442

Anna Filina:
Full Test Coverage is Impractical
May 23, 2013 @ 12:06:34

In her latest post Anna Filina proposes that full test coverage is an impractical way to measure the quality of your software. It can provide a false sense of security, even if the tests are poorly written.

Many developers claim that to achieve high quality software, developers must create automated tests that ensure that all possible execution routes have been covered. This is also known as full path coverage. I will argue that different types of software require different testing approaches and that full path coverage is impractical in almost every case. Too many tests simply create clutter.

She looks at how it's impractical to expect that all tests will be written efficiently or even correctly. Even simple tests are enough to show up on code coverage reports but may only be painting part of the picture. She also notes that not all software can be tested the same way - things like APIs require different testing skills/methods than something like consumer software.

In the end, there are no exact rules on how much to test. The most important thing to keep in mind is that writing tests for the sake writing tests is futile and costly. [...] Focus on building great software. Tests are a tool to make it better. Just don’t overdo it.
tagged: full testing codecoverage quality software impractical opinion

Link: http://annafilina.com/blog/full-test-impractical

Codeception:
Codeception released with CodeCoverage support
Jan 09, 2013 @ 11:14:19

The Codeception testing tool has released a new major update with some interesting new features - the expected feature for generating code coverage reports (similar to other tools) but there's also the idea of "remote code coverage" introduced.

There is no magic in local codecoverage. XDebug and PHP_CodeCoverage libraries do their job. The tricky thing is remote codecoverage. We attach small script into application's front controller. When a special header is sent this script starts to collect coverage information. And in the end of tests, this data is merged, serialized and sent back to Codeception. So you can test and collect coverage report even on staging servers in real environment.

They also mention a few other updates in the release - new Redis and MongoDb modules, UX improvements and the normal bugfixes. You can find out more about the code coverage feature in their manual or just about the project in general from the main site.

tagged: codeception unittest tool codecoverage remote report update

Link:

Derick Rethans' Blog:
Xdebug's Code Coverage speedup
Sep 23, 2011 @ 09:56:33

Derick Rethans has a new post to his blog today talking about some work that's been done to speed up XDebug's code coverage generation. Changes in the coming 2.2 release have some improvements that make things perform better and put less stress on PHP in the process.

Code coverage tells you how much of your code base is actually being tested by your unit tests. It's a very useful feature, but sadly, it slows down PHP's execution quite a lot. One part of this slowdown is the overhead to record the information internally, but another part is because I have to overload lots of opcodes. (Opcodes are PHP's internal execution units, similar to assembler instructions) They are always overloaded even if code coverage is not used, because it's only safe to overload them for the whole request.

These changes were from a combination of contributions from Taavi Burns and a new ini setting that will allow you to enable or disable the code coverage in XDebug. Benchmarking shows a good amount of time reduction in coverage runs - dropping anywhere from a few seconds to over a minute. He also mentions the idea of "modes", shortcuts to predefined settings for different types of reporting (like "profiling" or "tracing").

tagged: xdebug codecoverage speed improvement opcode contribution benchmark

Link:

Sebastian Bergmann's Blog:
Towards Better Code Coverage Metrics in the PHP World
Jun 20, 2011 @ 08:10:57

Sebastian Bergmann has a new post to his blog talking about some of the future plans for better code coverage metrics for PHP applications (not just the statistics that we have now as generated from PHPUnit runs combined with Code_Coverage PEAR package and Xdebug).

Xdebug currently only supports what is usually referred to as Line Coverage. This software metric measures whether each executable line was executed. Based on the line coverage information provided by Xdebug, PHP_CodeCoverage also calculates the Function / Method Coverage software metric that measures whether each function or method has been invoked.

The various kinds of coverage they're planning the in future include statement coverage, branch coverage (boolean evaluation), call coverage, path coverage with an alternative of linear code sequence and jump coverage (LCSAJ).

tagged: codecoverage metrics analyze code xdebug phpunit phpcodecoverage pear

Link:

Volker Dusch's Blog:
Dealing with segfaults while PHPUnit code coverage generation for CI
Apr 14, 2011 @ 11:55:13

Volker Dusch has a new post to his blog looking at handling segfaults during code coverage generation that might come up during your continuous integration process (with PHPUnit flavor unit tests).

About half the "Build failed" mails I’ve gotten from Jenkins in the last two weeks where not due to me breaking the tests but just PHPUnit segfaulting. "Wait! I know PHPUnit can’t segfault!", only PHP itself can. And it does, quite often. For some reason that probably has to do with using PHP 5.2.OLD it doesn’t survive generate the clover.xml file or the HTML report about 20% of the times it’s being run.

He finds a solution to keep this from happening - not correcting the segfault as much as running the testing again to get it to pass. He wraps the call to PHPUnit inside of a simple shell script that tries to execute the PHPUnit testing suite as long as a segfault isn't thrown. If something was seriously broken, this could cause all sorts of problems, but in theory it's a simple hack that gets the job done.

tagged: jenkins phpunit build continuousintegration codecoverage segfault

Link: