On the QaFoo blog today Manuel Pichler has written up a guide to help you write a custom rule to use with the PHP Mess Detector (PHPMD) as extended from some of the current rules (it's always easier to start from examples).
Let us start with some architecture basics behind PHPMD. All rules in PHPMD must at least implement the PHP_PMD_Rule interface. You can also extend the abstract rule base class PHP_PMD_AbstractRule which already provides an implementation of all required infrastructure methods and application logic, so that the only task which is left to you is the implementation of the concrete validation code of your rule.
He walks you through some sample code and talks about some of the common interfaces that are available to your testing. He takes one of them, IFunctionAware, and looks at it specifically, showing how to use it to detect and add violations for the validation. He includes an example of writing a rule based on an existing software metric (in this case, the number of public methods). There's also bit at the end showing how to write a rule based on the "abstract syntax tree" to be able to check just about anything in the code you're looking to test.