A new tutorial has been posted to php|architects C7Y community site looking at some practical uses of the PHP tokenizer to work with your source.

In this article we take a look at the PHP tokenizer and its potential at analyzing and processing PHP source code. We will build several working examples, which you can start using and extending for your own purposes.

The author, Stan Vassilev, explains what the tokenizer is for, how it parses the code (via a lexer) and how to get at the tokens that are created. He uses the token_get_all and token_name functions to get the information and creates a wrapper class around them to strip whitespace and comments.