At some point early this year, I decided that all code editors were terrible. MSVC is slow, XCode is really weird and a little buggy, Notepad++ and Pe and the Programmers Notepad are not cross platform, and Code::Blocks and Codelite are super buggy.
So I decided to write my own. Which has turned out to also have a few tough bugs in it. But in the process, I wrote a
lexer generator. I started using Flex, but holy cow it is ancient. It's just so...dusty. It works in such a strange way, and generates such huge files. Plus, it's really annoying to use, since it's
so flexible that doing anything at all is really difficult.
I looked at Sci Lexer, but I then found it to be similarly weird.
Probably Sci and Flex are really normal and I just have a weird idea of what a lexer should look like. But that's probably a good reason to write my own--if I am wrong, I will see concretely why I am wrong and why the right way is right.
But I think I may be right...Lunar generates small and fast lexers that cover what I need for syntax highlighting. I wrote a few other tools with it, such as a tool to check for all the functions that a certain JavaScript file will call, and Lunar's own file parser (not really done yet). It seems quite capable to me.
I based the API on what most nedit editors use for styling (I think). That is, you call:
int ParseString(const char * string, unsigned long len, void (*parse_callback)(const char *, unsigned long, int, void *), void * arg);
And when it hits something to lex, it calls:
void ParseCallback(const char *str, unsigned long len, int type, void *arg)
Telling you the position, the length, and the type of the token it found.
I really need to add a couple more features before it becomes really useful, mostly allowing spans to be optionally inclusive or exclusive, and allowing a callback for parts that didn't fit any of its rules.
But it's working so far, and it was fun to write!