The Problem?

These snippets themselves. for monkeytype, its easy, you can just paste random english words with variations and you got a good singleplayer experience. But we cannot just throw in random tokens and call it a day. Sure the code does not have to compile but it also cannot be nonsensical. If the person chooses Golang, the snippet should resemble golang.

So what do we do? We really have two options.

• Make a database with snippets of each language and randomly pick one and send it there, can work but again its not elegant and would be taxing on the server if someone wants to just practice on the singleplayer which will be most of the real usecase for it.
• Generate code on the client’s device on the fly, and this had so many advantages. The real backend server won’t be bothered as much and its bandwidth wont be piled up with sending code everywhere. And the app would be very snappy.

And that is why, I made Resrap, the reverse parser.

But what is a Parser?

Traditional Parsers take the “grammar” of the programming language, which contains all the rules to write syntactically correct code and using it validates the given code for its structure and reports syntax errors. It is a step while interpreting or compiling languages. What Resrap does, it reverses the parser’s role. It takes the Grammar and itself generates a syntactically correct snippet from it.

It is a substep in interpreting or compiling the code. We start off with Scanning or Tokenizing. A token (Not in LLM terms ffs) is the smallest piece of a language like

a=b+c

Here a,=,b,+,c and ; all are tokens.

So we first define what all type of tokens there are in a language, and scan through its text to get a list of the tokens.

Parsing comes right next to that step. We take the tokens and generate a Abstract Syntax Tree from it of the code. Then if you are making an interpreter you simply start at the leaf nodes and start collapsing the AST to run the code, a classic Tree Walking Interpreter or You could be making a compiler which is way more complex or a Just in Time compiler which are just legendary. ANYWAYS back to parsing.

Grammar Lesson

There is this one major difference between English and Java, every statement of the latter is deterministic. That is it always means the same thing. It won’t change what it does with change in context. And this level of precision has pushed us to develop grammars for languages. I learnt about from this book “Crafting Interpreters” you can check them out https://craftinginterpreters.com/

Back to Grammar. So, one of the most successful grammars for defining programming languages has been BNF. It will become obvious with a simple example

<sentence> ::= <noun> <verb> <object>
<noun> ::= Sam | Ali | Rohan
<verb> ::= ate | fought | saw
<object> ::= the robot | the food

We are just breaking down the structure and specifying what is a valid order for them. In the example, a sentence is made of noun verb and object. The noun further can be defined as being proper or not and the definitions can go till you have reached the simplest forms. More info about at Computerphile. But obviously BNF is very verbose so modern iterations like EBNF came over it to make it more readable, introducing more operators.

+ -> One or More
program: function+ 
Means a program is made with one or more functions
* -> Zero or More
function: statements*
Means A function is made with zero or more statements
? -> One or None
function: statement* return?
A function is made of one of more statements and it may or may not have a return

Lets right a (minimal) C grammar with it It’ll all make sense.

program :  function+;
function:functionheader'{''\n'functioncontent'}';
functionheader:datatype ' ' identifier '(' ')' ;
functioncontent:block+;
block:(statement+) | ifblock | whileblock ;
ifblock: 'if(' conditionalexpression '){\n' statement+ '}\n';
whileblock :'while(' conditionalexpression '){\n' statement+ '}\n';
conditionalexpression: conditionalexpone (conditionaljoin conditionalexpone)*;
conditionalexpone: (identifier conditionaloperation operand);
conditionaloperation: ' < ' | ' > ';
conditionaljoin: ' && ' | ' || ';
statement:assignment;
assignment: datatype ' ' identifier ' = ' expression ';\n';
expression: operand (operator (operand | '(' expression ')'))*;
operand: identifier | integer | float;
operator : ' + ' | ' - ' | ' * ' | ' / ';
datatype: 'int'|'float'|'double';
identifier: (A|a|B|b)+;
integer:(0|1|2|3|4|5)+;
whitespace: ' ';

Great now that we are on the same page, lets think

How do we generate from this grammar?

My first guess? Make it a graph I mean it literally looks like it is waiting to be converted to a graph! and that is exactly what I did in my first trial. Converting it to a simple graph and just walking it randomly. Just Connect what it needs to go next to and…

Then I was shot 50 times cuz that needed a lot of “state management” and was not a good idea. The problem was that if I am randomly walking, I would have to keep managing where I came from, and where to go from there. For example if I am in “Statement” node, I have no idea where on earth did I come from and what all came before. This would just get worse when we have to generate nested blocks and stuff so we totally could not do that.

To add to our troubles, the special operators like + would over-complicate it tenfold.

Plus we also had to code a recursion free solution since I wanted a more performant solution and easier on the RAM.

Wrapping every Node

If you look closely at the grammar, it has the special characters +,.* etc and then apart from that there are two main nodes,

functionheader:datatype ' ' identifier '(' ')' ;

Here, identifier will invoke the identifier: … statement And ‘{‘ will print something

We take them and wrap them up, with two empty nodes in out graph Lets ignore everything else right now and look into this structure. It makes applying the operators to the graphs super easy just see.

State of The Operators

How a whole statement would play out

Now, we can simply iterate through this mini graph randomly we have made for each statement and it would print things out perfectly

Now we make these mini graphs for each statement. A different one for each and can save them in a map just like:

All we need to do is figure out a way if a node references statement1 how do jump and track out last position and make it work.

The Stack

So now we have a map of graphs we need to iterate through it. For single statement it works, generating that specific sequence perfectly, however we still need a state management for cases like such:

The graph starts at A, then it needs to go to abra, and safely return back to A again. For this, we just use the trusty stack. A simple integer stack whenever you need to jump to a different statement, you will save the id of that node’s inter in the stack. Then the flow can switch to the other graph. And it can happen any number of times, since each jump is only filling a single integer in the stack, we can have practically endlessly deep references. Then when you hit the end node of any statement, you check the stack for what is the top value. You then shift back the control to the top value from the stack and pop that. Simple and elegant

Placing the Seed

This structure can generate code randomly at this point. But lets remember the original problem. It will work just fine with the OSDType interface you provide the grammar and get back a random script that too very fast. But what if you want to play multiplayer. Two players are to be given the same snippet to see who writes that the fastest. How do we build that!?

This is where Pseudo Random Numbers (PRN) help us. We set up a “Seed” initially which is something you would be familiar if you have played minecraft before.

When you first load a Minecraft world, it generates everything you see the biomes, the villages the mountains everything right there. It appears random but is not. All the random numbers needed in its generation are sourced from a single number, the seed. If like a huge integer code in minecraft. If you want the same world loaded into you as a friend or as a youtuber, you would get their world’s seed and enter that. While generating, now all the random numbers used in generation of the world will be same for everyone using the seed.

See: https://minecraft.wiki/w/World_generation

123 -> 432 -> 5435 -> 4123 -> 901 We start from a seed, here its 123 and genrate anay number of random numbers from it

Hence its called Pseudo Random Number Generation. These values are seemingly random, but they are sourced from a single Seed.

Similarly, when I say I am walking the graph inside Resrap, I am not randomly going. I am using PRNG to get values where to go. If the user sets no seed, it picks a random seed, if we get a seed while generation, then the snippet generated will always be identical.

Now think about it, in a multiplayer typing contest where there are a thousand players playing simultaneously say, the server only needs to forward on 64 bit integer to every player and it would automatically be converted to full on snippets on everyone’’s devices. No load on the server at all, and an amazing snappy experience for all the users.

Inventing New Operators

EBNF is amazing but the specific needs of a language generation needs some more operators in order to generate realistic snippets.

Making it truly infinite

We cannot specify how many tokens it will generate. Sure we can specify a max cap to generate after which we can stop the program, but its not guaranteed to reach that specific number, it can go through the graph and reach the end and check the stack and there would be nowhere to go. The program will just stop then. If you want a million tokens you literally cannot.

Hence introducing the infinity operator tadaa: program : functions^ This specifies that program is made of infinite functions. Hence, whenever we reach the end of a statement, and the stack is empty, then we jump to the function node. So now we can generate absurdly long pieces of code, in benchmarks I have gone well over 50 million tokens in a single run.

The Regex-er

We need random numbers and strings to be generated, and to make that less tiresome, I added a “syntax sugar”. Here you can define a simple regex of the code you want to generate and it will generate those. More work needs to be done to make it more adaptable, but it can generate simple identifiers like variable names and all.

identifier: [a-z];
integer:[0-9];

To make them sound less random, we also have a bias function placed which gives a bias to all the numbers like e get highest then other common characters. This allows us to make normal sounding random words. Would probably add support for random real words if that does not cost us too much performance.

Cumulative Frequency Distribution something something

The final one, and this is for the aesthetics of the code itself. In my code, I would prefer there be more normal block statements than if or while blocks in general . Or maybe there be a longer number of statements in a block before it exits.

For this we introduce the probability operator:

block:(statement+)<0.8> | ifblock<0.1> | whileblock<0.1> ;

Underneath, after we have generated the graph, we call a Normalize function, which normalizes all the probabilities to 1. Where there are no probabilities, we add them to ensure a common structure throughout. So each edge has a probability. Now for a node A which has 3 next options to go to. we make a Cumulative Frequency Distribution, which is just an array where we add keep adding the last sum of frequencies. It should make the last value 1.

With this, all we need to do while iterating is generate a random number with prng and normalize it such that it lies below 0 and 1 (evenly). Now all we have to do is iterate through this array and find the first number that is larger than out generated number.

If we get 0.8 for example, then we pick C because its CF index reads 0.9.

We can optimize it even further but altering the position of the elements such that the one with the probabilities are in descending order such that we will usually find the solution in the earlier tries. Something like:

So now 70 percent of the times we will get the value on the very first value of the structure. And that is how we over engineer Random Code Generation

— > Now we can generate infinitely nonsensical code very efficiently

Multi threading Without Safety

If you have one data structure, and many threads accessing it, you need to introduce locks to avoid two threads accessing at once which may lead to race conditions. There is a classic “Readers Writers Problem” in CS which is that what to do when many threads which only read and other threads which also write are accessing the same data structure.

A common solution to this is when the data structure needs to be written to, it blocks everything, every other reader, writer etc. Others have to wait till they acquire the lock before proceeding. However, if a reader gets the lock, every other reader is allowed to access the data structure, and all writers are blocked. This way we can get a good throughput and not lock everyone all the time. Readers can have this selective read only lock.

In our codebase however, there are two distinct phases, One when the graph is being built, and the other when we iterate through it. This means we can have a writer stage in single threaded mode, and then we can literally freeze the ds from any further changes so multiple threads can in parallel access it increasing the throughput substantially. Thanks to this, we can convert the code to walk the graph in parallel once the graph is generated. So we do not need to implement locks and it can safely do that without them.

Contributing

I have made this an actual Golang (and rust) library with docs published. You can find more about this project at https://resrap.osdc.dev/, it comes with documentation and a in browser tester running on a WASM engine.

Imagine we’re handed a file with the following code:

let a = 2;
let b = a * 2;
console.log("The Solution is ", b);

How on earth are we supposed to make sense of this string of text?!

That’s where an interpreter swoops in to save the day.

The Job of an Interpreter

The interpreter’s main task is to turn this cryptic text into something meaningful that a computer can execute. How does it do that? By following 3 steps:

Step 1: Scanning and Tokenizing

The first thing we need to do is scan the text and break it into tokens. A token is the smallest meaningful piece of code. Think of it like the building blocks of the program:

• Keywords (e.g., let, function)
• Literals (e.g., numbers like 2, strings like "hello")
• Identifiers (e.g., variable names like a, b)
• Operators (e.g., =, *)

In any programming language, tokens usually have:

• A type (e.g., IDENTIFIER, NUMBER_LITERAL)
• A value, if it holds one (e.g., 2 is a NUMBER_LITERAL with a value of 2)

For example:

let -> LET_KEYWORD
a   -> IDENTIFIER   "a"
=   -> EQUAL_SIGN   (no value, type says it all)
2   -> NUMBER_LITERAL 2
...etc

This process of scanning and tokenizing is done by the Scanner (or Lexer), which transforms the raw code into a list of meaningful tokens.

For example:

Code: let a = 2;

Tokens:

1. { type: LET_KEYWORD }
2. { type: IDENTIFIER, value: "a" }
3. { type: EQUAL_SIGN }
4. { type: NUMBER_LITERAL, value: 2 }

Step 2: Parsing the Text

Now that we have the list of tokens, we need to parse it. Parsing is all about making sense of it and converting it into an Abstract Syntax Tree (AST).

But before diving into parsing, let’s quickly recap how trees work in programming.

What is a Tree in Programming?

A tree is a special type of data structure that resembles a hierarchy. Think of it like a family tree or the folder structure on your computer.

Here’s how it works:

• The topmost element is called the root.
• Every item (called a node) can have children nodes branching out below it.
• Nodes with no children are called leaves.
• The connections between nodes are called edges.

Key Features of Trees

Parent-Child Relationship: Each node is connected to its parent and may have zero or more children. For example:

     Root
     /  \
  Node1  Node2
   /       \
Leaf1     Leaf2

Single Path Rule: There is only one path between any two nodes, meaning no cycles! It’s not a tree if you can loop back to the root.

Now, let’s see how this logic is used to build an Abstract Syntax Tree (AST).

Quick Quiz: Which of These Makes Sense?

let a = 2 + 3;
let = a + 2 3;

The first one, right?

That’s because the second one breaks the rules of grammar! Without proper grammar, we’d all be speaking gibberish, and both humans and computers would have a hard time deciphering meaning.

Grammar

There are two basic terms in Parsing Grammar: Terminals and Non-Terminals. If you imagine a tree analogy, terminals are essentially the leaf nodes, while non-terminals are all the other nodes.

Example 1

Let’s illustrate this with an example:

Consider:

a = b + c

This is converted into a tree branch that looks like this:

       =
     /  \
    a    +
        / \
       b   c

(Notice I’m calling it a branch because it is part of a much larger tree representing the whole program. This is just a single statement within that tree)

Makes sense? We’re breaking it down into a meaningful structure!

Explanation

• Non-terminals: =, +
  • If these are present, it implies there should be something to the left, right, or both for the given statement to be grammatically correct.
• Terminals: a, b, c
  • These are the leaf nodes and do not enforce any further structure.

Example 2

Let’s enhance this logic with another example:

Consider:

a = b + c * d

This converts into the following branch of the tree:

       =
     /  \
     a    +
         / \
        b   *
           /  \
          c    d

Now it’s even clearer:

• Terminals: a, b, c, d
• Non-terminals: =, +, *

Summary

Operators (=, +, *) are non-terminals, as they structure the statement. Variables (a, b, c, d) are terminals, as they represent the final values.

Great! Now, how do we actually define the rules we need to play with?

Syntax for Grammar

There is a popular formal syntax for grammar called BNF. You can look it up — Computerphile has some amazing videos on it! However, I’ll discuss a much simpler regex-like syntax as explained in the book Crafting Interpreters.

I don’t want to overwhelm you with learning the entire thing, as there isn’t much to it. Instead, let’s construct our grammar for a language side by side.

Dummy Language Example

Here’s a dummy language whose syntax we will build:

void name{
a = 2 + 3;
b = 10 + 20;
}

This language has no arguments in functions for simplicity, avoiding messy string handling for now. Let’s keep things clean!

Building the Grammar

Let’s break down the hierarchy of components in our code and build the structure step by step.

What is a program made of?

A program is made of functions, which we can write in our syntax as:

Program = Function+

• + means 1 or more.
• Similarly, * means 0 or more.
• Since a program needs at least one function, we use +.

What is a function?

Function = header '{' body '}'

Here:

• header: void name
• body: The statements inside the curly braces.

Defining Header and Body

header = DataType Identifier
body = Statement+

• header: Contains a DataType and an Identifier in this order.
• body: Contains 1 or more statements.

Defining DataType and Identifier

DataType = int | float | char | void
Identifier = (alphabet | _ ) (alphanumeric | _ )*

• | means “or”.
• A DataType can be any of the listed types.
• An Identifier starts with an alphabet or an underscore, followed by any number of alphanumeric or underscore characters.

The * here allows 0 or more characters after the first one.

Diagnosing Statements

Statement = identifier '=' expression;

Defining an Expression

For simplicity, let’s say an expression consists of numbers, identifiers, and operators (+, *):

expression = terminal | (expression operator expression)
operator = + | *
terminal = identifier | number

This definition is simple and elegant but lacks operator precedence rules, which are critical for order of operations (BODMAS).

Addressing Operator Precedence

Issue:

Given an expression like:

2 + 3 * 4

Our current grammar might parse it as:

(2 + 3) * 4

Instead of the correct BODMAS evaluation:

2 + (3 * 4)

Solution:

To fix this, we adjust the grammar to handle precedence by splitting it into levels:

expression = term ( '+' term )*
term = factor ( '*' factor )*
factor = identifier | number | '(' expression ')'

Explanation:

• term resolves multiplication first.
• expression handles addition after terms are resolved.
• factor includes parentheses to allow manual precedence overriding.

With this grammar, the expression 2 + 3 * 4 is correctly parsed as:

2 + (3 * 4)

How Does This Work?

term handles multiplication first because it’s defined at a lower level.

• expression handles addition only after terms are resolved.
• factor includes parentheses to allow overriding precedence manually.

Using this grammar, the expression 2 + 3 * 4 is parsed as:

expression -> term '+' term
term -> factor '*' factor
factor -> 3, 4

This ensures multiplication (3 * 4) happens first, followed by addition (2 + 12), yielding the correct result 14.

The parse tree in Action

Grammar till now:

Program = Function+
Function = Header '{' Body '}'
Header = DataType Identifier
Body = Statement+
Statement = Identifier '=' Expression
Expression = Terminal | (Expression Operator Expression)
Operator = '+' | '*'
Terminal = Identifier | Number
DataType = 'int' | 'float' | 'char' | 'void'
Identifier = (alphabet | _ ) (alphanumeric | _ )*

Example Code:

void func1 {
 a = b + c * d;
}
void func2 {
 x = y + z;
}

Parse Tree (ASCII Style)

Program
├── Function (func1)
│   ├── Header
│   │   ├── DataType (void)
│   │   └── Identifier (func1)
│   ├── Body
│   │   └── Statement
│   │       ├── Identifier (a)
│   │       ├── '='
│   │       └── Expression
│   │           ├── Identifier (b)
│   │           ├── '+'
│   │           └── Expression
│   │               ├── Identifier (c)
│   │               ├── '*'
│   │               └── Identifier (d)
├── Function (func2)
│   ├── Header
│   │   ├── DataType (void)
│   │   └── Identifier (func2)
│   ├── Body
│   │   └── Statement
│   │       ├── Identifier (x)
│   │       ├── '='
│   │       └── Expression
│   │           ├── Identifier (y)
│   │           ├── '+'
│   │           └── Identifier (z)

Here is the continuation, word by word, starting from Parse Tree (Lateral Style):

Parse Tree (Lateral Style)

For Function func1:

=
             /   \
            a     +
                 / \
                b   *
                   / \
                  c   d

For Function func2:

=
             /   \
            x     +
                 / \
                y   z

Full Program Tree:

Program
   ├── Function (func1)
   │         \
   │          =
   │         / \
   │        a   +
   │           / \
   │          b   *
   │             / \
   │            c   d
   └── Function (func2)
             \
              =
             / \
            x   +
               / \
              y   z

This breakdown shows how the program can be parsed into a tree format for easier interpretation.

Sneak Peek into the implementation

I would encourage you to go ahead and try writing a small piece of code, maybe just a calculator (That’s what I did first) to experiment with making a parse tree. A very good explanation and a highly detailed one is given in Crafting Interpreters, and you should check that out for constructing the parse tree. However, here is a look into how that’s done:

So we have a counter that keeps track of the current token’s index. It will start at 0 and go all the way till the length of the list of tokens. Now we have two functions, match and consume. match(token) will take a token and check if the current token is the same as the one sent in it and return a boolean.

consume(token, errorMsg) will take the type of token expected there and an error message. If the token is not the one we sent, it will crash the code with the given error message. Again, for implementation details check the book, it’s a goldmine. I am here just to give you the gist of it.

The Walker and Post Order Traversal

Now, we have constructed a complete tree and have to go ahead and parse the tree gracefully, executing it. For that, we will start at the root and do a post-order traversal.

Here is some theory, and then I will give an example with expressions.

Post-Order Traversal (L-R-Root)

In post-order traversal, you visit:

1. Left subtree.
2. Right subtree.
3. Root node.

Example:

For a binary tree:

A
     / \
    B   C
   / \
  D   E

Steps:

1. Visit the left subtree (B → D → E).
2. Visit the right subtree (C).
3. Visit the root (A).

Post-Order Traversal Output:

D → E → B → C → A

Key Rule: Process children first, then the parent.

Example:

Consider the expression represented by the following binary tree:

=
             /   \
            a     +
                 / \
                b   *
                   / \
                  c   d

Where:

• b = 1
• c = 2
• d = 3

Post-Order Traversal Approach:

1. Start at the root: The root is the = sign, representing an assignment. To perform this assignment, we need to first evaluate both the left-hand side (LHS) and the right-hand side (RHS) of the equation. This requires us to traverse both sides of the tree using post-order traversal.

2. Left Subtree (LHS): The left side is simply a, a terminal identifier. No further traversal is needed here, so we just take a as is.

3. Right Subtree (RHS): The right-hand side is the + operator, which is a non-terminal. So, we need to perform post-order traversal on the right subtree, which consists of:

   • The left child: b (terminal with value 1).
   • The right child: * (another non-terminal).

4. Subtree with * (Right of +): The multiplication operator (*) requires post-order traversal as well. It has two children:

   • c (terminal with value 2).
   • d (terminal with value 3).

   Since both c and d are terminals, we evaluate them first: c = 2 and d = 3. The multiplication (c * d) gives us 2 * 3 = 6. Now the * node becomes:

    *
   / \
   2  3
   

   Copy

   We replace the * node with 6:

    +
   / \
   b  6
   

   Copy

5. Final Step: Now that we have resolved both children of the + node (b = 1 and the result of * = 6), we can evaluate the addition: 1 + 6 = 7. So, the + node is replaced with 7:

    =
   / \
   a  7
   

   Copy

6. Solve the Assignment: Finally, we can assign the value of 7 to a: This is the final result of the post-order traversal and evaluation process for this simple expression.

Key Points:

• Post-order Traversal: This process visits the left subtree first, then the right subtree, and finally the root, ensuring that we evaluate operands before operators.
• Recursive Traversal: Non-terminals (like + and *) require recursive traversal to break down the expression into simpler components, ultimately reducing the expression to terminal values that can be computed.
• Evaluation: Once the tree has been fully traversed and reduced to terminal values, the final result can be computed step by step.

Extending This Concept:

This approach can be extended to handle more complex operations such as strings, loops, conditionals (if-else), and even multi-level expressions involving arrays or objects. It is a foundational concept for building parsers and evaluators for your own programming language.

Project Suggestion:

A great beginner project would be to build a calculator that can handle multiple arithmetic operations:

• For example, you could write an expression like:

  a = 2 + 4
  b = a / 2 + 20
  

  Copy

  And print the results of the expressions. Once you’ve implemented that, you can explore scope management (keeping track of variable values) and control flow (like if statements, loops), ultimately creating a simple interpreter or mini-language.

Limitations

Tree walk interpreters are simple and easy to implement but suffer from poor performance due to direct evaluation of the Abstract Syntax Tree (AST) at runtime, without optimizations like operator precedence or memory management. They are slower than compiled languages because they lack optimizations and perform redundant work during interpretation. In contrast, machine code compilation converts code directly to machine language, offering excellent performance but no flexibility. Bytecode compilation translates code into a platform-independent format, offering portability and some optimizations, while Just-In-Time (JIT) compilation dynamically compiles code at runtime, providing performance close to machine code with the advantage of dynamic optimizations, though it introduces initial overhead. I insist you check these other compilation methods out and try stepping up your model later on.

You can check out my interpreter, Solunox.