Skip to content
April 24, 2008 / Bob Black

Parser Remade

Yes, I’m still working on Code Toaster in my spare time. Given several bug reports from the last release involving the COM-based parsing toolkit, ProGrammar, from Norken Technologies, along with the fact that development seems to be inactive right now (read: they stole my money), I decided to yank out all ProGrammar dependent code and replace it with pure .Net parsing stuffs.

And that’s working very well. In fact, I think it’s working better than I expected, perhaps because, this time around, I’m using much better development practices. Like Test Driven Development. In retrospect, it was extremely foolish of me to ever approach a large coding project like this without reams of automated tests – I’ll talk about that later in this post. But first, a little about how the parser(s) work…

Parsing a template is comprised of three steps.

  1. A Code Toaster template is parsed into a parse tree by the Template Parser.
  2. The C# Generator translates the parse tree into compilable C# code. In the new design, C# code generation is done as the template is parsed, not as a separate step (as before, with the ProGrammar parser), which yields much improved performance. I maintained modularity by deriving the C# Generator class from the Template Parser base class, and by making parsing event based: as template “nodes” are parsed, an event is raised that allows any interested classes to “do stuff” (like generate code).
  3. The C# Parser parses the generated C# code. The only reason this step is necessary is to give you bodacious IntelliSense as you’re editing template code.

Each step increases in complexity by many-folds. I’ve completed the Template Parser, which was enjoyably challenging but not ridiculously difficult. It’s currently parsing a 1,000+ line template in about 80 milliseconds, which should be fast enough, I think. I’ve done some performance tuning, but may revisit it later to try to speed things up even more. As you’re editing a template, any changes (like variable declarations, new types, etc) need to be available immediately within IntelliSense, which means the backend parser and IntelliSense engine must be lightning fast.

And reliable, which is why I’ve spent a good deal of time fleshing out a suite of unit tests. VS.Net 2008’s new unit testing functionality is just a beautiful thing. It’s actually beyond beautiful. If a test fails, I can easily run just that test (VS.Net is smart enough to figure out what test to run by looking at where my cursor is in the test class), step into the code, and inspect and fix the problem using the stupendous Edit & Continue feature. Yeah, you could do almost the same thing with NUnit, but the process is so much more streamlined now, and just plain easy.

I think I’m on the right track. Getting rid of ProGrammar means there’s no cross-appdomain performance penalties, no COM headaches (like registering and unregistering COM dlls, and all the headaches that could entail, during installation), and most of all, I can tailor the parser architecture to meet the rather specific needs of Code Toaster.

So your next question is probably: what about that C# parser? How long is that bad boy going to take to write from scratch?

And the answer is: not long at all. But that’s a story for another post.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: