A lot of developers avoid writing parsers because they think it's pretty hard to do so. Writing an efficient parser for a general purpose language (like PHP, Ruby, Java,...) is hard, but fortunately, most of the time we don't need that much complexity. Typically we just want to parse input coming from config files or from a specific problem domain (expressed through DSLs). DSLs (Domain Specific Languages) are pretty cool, because they allow you to express logic and flow in a very specific and convenient way for a limited set of tasks.

He illustrates with an example based on the Lithium framework's routing engine and how it could parse a text file that relates a route to a controller/action combination. He creates a "Lexer" class that defines a few regular expressions to parse the incoming text strings for matches on things like whitespace, URLs and identifiers (words) and return each in the lexer's output.

After looking around for a while [for a good resource on compilers] I settled for Terence Parr's Language Implementation Patterns. This is exactly what I needed - bit sized patterns on compiler and parser design with working code. The book provides a recipe style approach, gradually moving from simple to complex compiler/parser design issues. As I primarily work with PHP, I thought of porting some code to PHP to see how it works.

He shows examples using his custom tool to show a basic lexer output for a list and a complete listing of the code involved. Ultimately, though, he finds that PHP isn't overly suited to the task - anything more than his simple example could be more trouble than it's worth.

His examples include:

• String vs. numeric handling
• That PHP supports octal numbers "by accident"
• A lexer bug with hex values
• A parser bug involving the ternary operator

Comments on the post include those supporting the "die" opinion - that PHP just doesn't have it together like other languages - and those taking a bit more balanced approach on PHP's strengths and weaknesses.

"PHP tokens and opcodes" - When a PHP script is executed it goes through a number of processes, before the final result is displayed. These processes are namely: Lexing, Parsing, Compiling and Executing. In this blog post, I will walk you through all these processes with a sample example. In the end I will list some useful PHP extensions, which can be used to analyze results of every intermediate process.

He touches on the steps the average PHP script takes in its processing - lexing, parsing/compiling and the actual execution of the opcodes. The tokenizer, parsekit and VLD (Vulcan Logic Disassembler) extensions can help you get down into the nuts and bolts of the language and the engine that makes it work.

From time to time, I find that I need to put a parser together. Most of the time I find that I need to do this in C for performance, but other times I just want something convenient, like PHP, and have been out of luck.

His result is two new packages - lemon-php and JLexPHP (under a BSDish license) you can download and compile on your own system.

Also, if you'll remember a while back, Greg Beaver had wanted something similar (as mentioned in the comments) and created his own lexer/generator as well.

Last week, I blogged about completing a port of the Lemon parser generator to PHP 5, which I thought was pretty cool. However, in an email, Alex Merz pointed out that without a lexer generator to accompany lemon, it's pretty difficult to write a decent parser.

After Alex's email, I started thinking about what it would take to write a lexer generator. Basically, a lexer generator requires parsing and compiling regular expressions, then scanning the source one character at a time to find matches. So, it occurred to me that perhaps simply combining regular expressions with sub-patterns could accomplish this task quite easily.

He goes on to explain this process, showing how a simple regular expresion call (and a look at its return arguments) could create a simple, easy solution. Since the re2c format is still unsupported in PHP (without a goto to go to), he opts to stick with the regular expressions and creates a "lex2php" format instead.

He's packaged up both halves of this setup and has already posted proposals for them to the PEAR site:

• PHP_LexerGenerator
• PHP_ParserGenerator