Protobuf to serialize AST

I've been thinking about how to incrementally bootstrap the Lys compiler for some years now. The challenge may be huge if not split in smaller achievable tasks. Non intuitively first thing to do will be to create a protobuf library.

The first thing I did to proof that it was going to work was to create the parser. It works fairly well, only allocating memory for the emitted tokens and constructions. Which is ideal for a parser.

But then I hit a wall, the rest of the compiler does most of the heavy-lifting and it is completely written in JS. And since there is no ABI to leverage nor code-generation, sending the AST or tokens back to JS was either a challenging task or a very tedious one with low RoI.

But then this idea of serializing the parsed AST and tokens as protobuf started growing in me. The protoc compiler leverages lots of community created plugins to generate more code or analyze the parsed .proto files of the project. And that is exactly what this compiler needs, to have the parser in one program and the compiler (or plugin in protoc idioms) in a separate program.

The protoc compiler plugins work by first specifying an input for the plugin in .proto format. And then an output for the plugin in .proto format, it is brutal, the tool leverages stdin/stdout to communicate with plugins, making it portable to any language.

The results of the plugin, whereas new files to be generated or linter hints or analysis of the code are serialized in protobuf and read by protoc which then executes the final "file writes" based on the results.

But first, a protobuf code generator or library must be created for Lys.