Introduction

An arithmetic expression parser turns text like 2 + 3 * 4 into a structure a program can evaluate correctly. The hard part is not reading the characters, but preserving operator precedence, associativity, parentheses, and useful error messages.

Start With Tokenization

Before parsing, convert the raw string into tokens. This separates concerns: the tokenizer handles characters, while the parser handles grammar.

For arithmetic expressions, common token types are:

• numbers
• identifiers
• operators such as +, -, *, /
• parentheses

A simple tokenizer in Python can walk the string one character at a time:

1import re
2
3TOKEN_RE = re.compile(r"\s*(?:(\d+)|([A-Za-z_]\w*)|(.))")
4
5def tokenize(text):
6    tokens = []
7    for number, identifier, other in TOKEN_RE.findall(text):
8        if number:
9            tokens.append(("NUMBER", int(number)))
10        elif identifier:
11            tokens.append(("IDENT", identifier))
12        elif other in "+-*/()":
13            tokens.append((other, other))
14        else:
15            raise ValueError(f"Unexpected character: {other}")
16    tokens.append(("EOF", None))
17    return tokens
18
19print(tokenize("sum + 3 * (8 - 2)"))

Without tokenization, the parser becomes cluttered with character-level details.

Encode Precedence In The Grammar

The usual technique is to split the grammar into levels. Each level handles one precedence band.

A classic grammar looks like this:

• expression: addition and subtraction
• term: multiplication and division
• factor: literals, names, unary operators, and parenthesized expressions

This shape naturally gives * and / higher precedence than + and -.

Recursive Descent Is A Practical Choice

For a small arithmetic language, recursive descent is often the best first parser. It is readable, easy to debug, and maps directly onto the grammar.

Here is a compact parser that builds an abstract syntax tree:

1class Parser:
2    def __init__(self, tokens):
3        self.tokens = tokens
4        self.pos = 0
5
6    def peek(self):
7        return self.tokens[self.pos][0]
8
9    def consume(self, expected):
10        token = self.tokens[self.pos]
11        if token[0] != expected:
12            raise ValueError(f"Expected {expected}, got {token[0]}")
13        self.pos += 1
14        return token
15
16    def parse(self):
17        node = self.parse_expression()
18        self.consume("EOF")
19        return node
20
21    def parse_expression(self):
22        node = self.parse_term()
23        while self.peek() in ("+", "-"):
24            op = self.consume(self.peek())[0]
25            right = self.parse_term()
26            node = ("binop", op, node, right)
27        return node
28
29    def parse_term(self):
30        node = self.parse_factor()
31        while self.peek() in ("*", "/"):
32            op = self.consume(self.peek())[0]
33            right = self.parse_factor()
34            node = ("binop", op, node, right)
35        return node
36
37    def parse_factor(self):
38        if self.peek() == "NUMBER":
39            return ("number", self.consume("NUMBER")[1])
40        if self.peek() == "IDENT":
41            return ("ident", self.consume("IDENT")[1])
42        if self.peek() == "-":
43            self.consume("-")
44            return ("negate", self.parse_factor())
45        if self.peek() == "(":
46            self.consume("(")
47            node = self.parse_expression()
48            self.consume(")")
49            return node
50        raise ValueError(f"Unexpected token: {self.peek()}")

This parser already handles precedence, associativity, unary minus, and parentheses.

Build An AST Instead Of Evaluating Immediately

It is tempting to compute results during parsing, but an AST gives you more flexibility. You can:

• evaluate later
• optimize expressions
• generate code
• print useful debugging output

An evaluator can be a separate pass:

1def evaluate(node, env=None):
2    env = env or {}
3    kind = node[0]
4
5    if kind == "number":
6        return node[1]
7    if kind == "ident":
8        return env[node[1]]
9    if kind == "negate":
10        return -evaluate(node[1], env)
11    if kind == "binop":
12        _, op, left, right = node
13        a = evaluate(left, env)
14        b = evaluate(right, env)
15        if op == "+":
16            return a + b
17        if op == "-":
18            return a - b
19        if op == "*":
20            return a * b
21        if op == "/":
22            return a / b
23    raise ValueError(f"Unknown node: {kind}")

Separating parsing from evaluation makes the code easier to test.

Know When To Use Other Techniques

Recursive descent is excellent for simple grammars, but it is not the only option.

• Pratt parsers are great when you want a concise expression parser with many operators.
• Shunting-yard is useful if you want postfix output or stack-based evaluation.
• Parser generators are better when the grammar grows beyond a small expression language.

If your goal is only arithmetic expressions, recursive descent or Pratt parsing is usually enough.

Common Pitfalls

• Using a left-recursive grammar directly in recursive descent. That causes infinite recursion.
• Mixing tokenization and parsing in the same function. The code becomes hard to reason about.
• Forgetting unary operators. Expressions like -5 * 2 then fail or parse incorrectly.
• Evaluating too early. Once parsing and execution are intertwined, later features become harder to add.
• Returning vague errors. Reporting the unexpected token and position saves a lot of debugging time.

Summary

• Tokenize first, then parse.
• Represent precedence through grammar structure such as expression, term, and factor.
• Recursive descent is a practical technique for arithmetic expressions.
• Build an AST instead of evaluating during parsing.
• Consider Pratt or shunting-yard when the operator set becomes more complex.