Writing a simple lexical analyser in C
October 7, 2026 · 11 min read · c, compiler-design, tutorial
When lexical analysis first comes up in a compiler design course, it sounds more complicated than it is. There's talk of finite automata, regular expressions and token classes. The idea underneath is simple, though. You read the source code one character at a time and group the characters into meaningful pieces, which are called tokens.
The easiest way to really get it is to write one. So here's a small but complete lexical analyser (a lexer) in C. It picks out keywords, identifiers, numbers, operators and punctuation, skips spaces and comments, and reports characters it doesn't recognise. The whole thing is about 100 lines.
What a lexer does
Take this line of C:
int total = 0;
You can see at a glance that it's a declaration. A compiler can't. To it, this starts out as a string of characters: i, n, t, a space, t, o, t and so on. The lexer's job is to turn that into a list of tokens:
| Token | Type |
|---|---|
int | keyword |
total | identifier |
= | operator |
0 | number |
; | punctuation |
The next stage of the compiler, the parser, works with these tokens instead of raw characters, which makes its job a lot simpler. The lexer also throws away things the parser doesn't care about, like spaces and comments.
Representing a token
Each token needs three things: its type, the text it was made from, and the line it came from, so that error messages can say where the problem is.
typedef enum { TOK_KEYWORD, TOK_IDENTIFIER, TOK_NUMBER, TOK_OPERATOR, TOK_PUNCT, TOK_EOF, TOK_ERROR } TokenType;
typedef struct {
TokenType type;
char text[64];
int line;
} Token;
TOK_EOF marks the end of the input, and TOK_ERROR marks a character we don't recognise.
The scanner state
The lexer keeps three pieces of state: the source text, the position of the next character, and the current line number.
static const char *src; /* the input being scanned */
static int pos = 0; /* index of the next character */
static int line = 1; /* current line, for error messages */
Skipping what doesn't matter
Before reading a token, we skip whitespace and // comments. Each newline increments line, so error messages stay accurate:
static void skip_whitespace_and_comments(void) {
for (;;) {
char c = src[pos];
if (c == '\n') { line++; pos++; }
else if (isspace((unsigned char)c)) pos++;
else if (c == '/' && src[pos + 1] == '/') { /* // comment */
while (src[pos] != '\n' && src[pos] != '\0') pos++;
}
else return;
}
}
Those (unsigned char) casts look fussy, but they matter. On many systems a plain char is signed, and passing a negative value to isspace or isdigit is undefined behaviour in C.
Reading one token
The real work happens in next_token(). It looks at the current character and decides what kind of token starts there. The rules are the same ones you'd write as regular expressions in class. A letter or _ starts an identifier, which might turn out to be a keyword. A digit starts a number, possibly with a decimal part. Anything else is an operator, a piece of punctuation, or an error.
static Token next_token(void) {
Token t = { TOK_ERROR, "", 0 };
int len = 0;
skip_whitespace_and_comments();
t.line = line;
char c = src[pos];
if (c == '\0') { t.type = TOK_EOF; return t; }
if (isalpha((unsigned char)c) || c == '_') { /* identifier or keyword */
while ((isalnum((unsigned char)src[pos]) || src[pos] == '_') && len < 63)
t.text[len++] = src[pos++];
t.text[len] = '\0';
t.type = is_keyword(t.text) ? TOK_KEYWORD : TOK_IDENTIFIER;
return t;
}
if (isdigit((unsigned char)c)) { /* number, with optional .fraction */
while (isdigit((unsigned char)src[pos]) && len < 63) t.text[len++] = src[pos++];
if (src[pos] == '.' && isdigit((unsigned char)src[pos + 1])) {
t.text[len++] = src[pos++];
while (isdigit((unsigned char)src[pos]) && len < 63) t.text[len++] = src[pos++];
}
t.text[len] = '\0';
t.type = TOK_NUMBER;
return t;
}
/* two-character operators first, so "==" isn't read as "=" "=" */
const char *two[] = { "==", "!=", "<=", ">=", "&&", "||" };
for (int i = 0; i < 6; i++) {
if (strncmp(&src[pos], two[i], 2) == 0) {
strcpy(t.text, two[i]);
pos += 2;
t.type = TOK_OPERATOR;
return t;
}
}
t.text[0] = c;
t.text[1] = '\0';
pos++;
if (strchr("+-*/=<>!", c)) t.type = TOK_OPERATOR;
else if (strchr("(){};,", c)) t.type = TOK_PUNCT;
else t.type = TOK_ERROR; /* anything we don't recognise */
return t;
}
A few things in there are worth slowing down for.
The identifier loop keeps eating letters and digits for as long as it can, so total2 comes out as one token rather than total followed by 2. This is called longest match, and almost every lexer works this way.
Keywords are handled as identifiers with special names. The code reads the whole word first and only then checks it against the keyword list. That's much simpler than trying to recognise while one character at a time.
Two-character operators are checked before single characters. If they weren't, <= would come out as < and then =.
And the len < 63 checks stop a very long word from overflowing the 64-character buffer. It's an easy thing to forget, and C won't warn you about it.
Keywords and readable output
The keyword check is a simple loop over a list:
static const char *KEYWORDS[] = { "int", "float", "if", "else", "while", "return" };
static const int KEYWORD_COUNT = sizeof(KEYWORDS) / sizeof(KEYWORDS[0]);
static int is_keyword(const char *word) {
for (int i = 0; i < KEYWORD_COUNT; i++)
if (strcmp(word, KEYWORDS[i]) == 0) return 1;
return 0;
}
A real compiler would use a hash table here, but for six keywords a plain loop is completely fine.
Running it
To try it out, main feeds in a small program and prints every token until it hits the end. I put an invalid character, @, on the last line on purpose:
int main(void) {
src = "int total = 0; // running sum\n"
"while (total <= 10.5) {\n"
" total = total + 2;\n"
"}\n"
"return total @ 1;\n";
Token t;
do {
t = next_token();
if (t.type == TOK_ERROR)
printf("line %d: unexpected character '%s'\n", t.line, t.text);
else if (t.type != TOK_EOF)
printf("line %d %-12s %s\n", t.line, type_name(t.type), t.text);
} while (t.type != TOK_EOF);
return 0;
}
(type_name() is a small switch that turns each TokenType into a string like "KEYWORD". It's included in the full program at the end.)
Compile and run it:
gcc -std=c99 -Wall -Wextra -o lexer lexer.c
./lexer
The output:
line 1 KEYWORD int
line 1 IDENTIFIER total
line 1 OPERATOR =
line 1 NUMBER 0
line 1 PUNCTUATION ;
line 2 KEYWORD while
line 2 PUNCTUATION (
line 2 IDENTIFIER total
line 2 OPERATOR <=
line 2 NUMBER 10.5
line 2 PUNCTUATION )
line 2 PUNCTUATION {
line 3 IDENTIFIER total
line 3 OPERATOR =
line 3 IDENTIFIER total
line 3 OPERATOR +
line 3 NUMBER 2
line 3 PUNCTUATION ;
line 4 PUNCTUATION }
line 5 KEYWORD return
line 5 IDENTIFIER total
line 5: unexpected character '@'
line 5 NUMBER 1
line 5 PUNCTUATION ;
A few things to notice. The comment // running sum is gone. <= came out as one operator, and 10.5 as one number. The @ was reported along with its line number, and then the lexer just kept going. That's how it should behave: report the problem and carry on, rather than crashing on the first bad character.
Where to go next
This lexer is small on purpose. If you want to keep going, here's what I'd add next:
- string and character literals like
"hello"and'a', including escape sequences such as\n - block comments (
/* … */), which can span several lines, so keep counting lines inside them - reading the source from a file instead of a hard-coded string, with
fgetsorfread - a symbol table that records each identifier the first time it shows up, since the later stages of a compiler depend on one
After that, try generating a lexer with a tool like flex. Once you've written one by hand, it's much easier to see what the tool is doing for you.
If you're studying Compiler Design under Pokhara University, past papers and notes for the subject are on BECE Vault.
The full program
#include <ctype.h>
#include <stdio.h>
#include <string.h>
typedef enum { TOK_KEYWORD, TOK_IDENTIFIER, TOK_NUMBER, TOK_OPERATOR, TOK_PUNCT, TOK_EOF, TOK_ERROR } TokenType;
typedef struct {
TokenType type;
char text[64];
int line;
} Token;
static const char *KEYWORDS[] = { "int", "float", "if", "else", "while", "return" };
static const int KEYWORD_COUNT = sizeof(KEYWORDS) / sizeof(KEYWORDS[0]);
static const char *src; /* the input being scanned */
static int pos = 0; /* index of the next character */
static int line = 1; /* current line, for error messages */
static int is_keyword(const char *word) {
for (int i = 0; i < KEYWORD_COUNT; i++)
if (strcmp(word, KEYWORDS[i]) == 0) return 1;
return 0;
}
static void skip_whitespace_and_comments(void) {
for (;;) {
char c = src[pos];
if (c == '\n') { line++; pos++; }
else if (isspace((unsigned char)c)) pos++;
else if (c == '/' && src[pos + 1] == '/') { /* // comment */
while (src[pos] != '\n' && src[pos] != '\0') pos++;
}
else return;
}
}
static Token next_token(void) {
Token t = { TOK_ERROR, "", 0 };
int len = 0;
skip_whitespace_and_comments();
t.line = line;
char c = src[pos];
if (c == '\0') { t.type = TOK_EOF; return t; }
if (isalpha((unsigned char)c) || c == '_') { /* identifier or keyword */
while ((isalnum((unsigned char)src[pos]) || src[pos] == '_') && len < 63)
t.text[len++] = src[pos++];
t.text[len] = '\0';
t.type = is_keyword(t.text) ? TOK_KEYWORD : TOK_IDENTIFIER;
return t;
}
if (isdigit((unsigned char)c)) { /* number, with optional .fraction */
while (isdigit((unsigned char)src[pos]) && len < 63) t.text[len++] = src[pos++];
if (src[pos] == '.' && isdigit((unsigned char)src[pos + 1])) {
t.text[len++] = src[pos++];
while (isdigit((unsigned char)src[pos]) && len < 63) t.text[len++] = src[pos++];
}
t.text[len] = '\0';
t.type = TOK_NUMBER;
return t;
}
/* two-character operators first, so "==" isn't read as "=" "=" */
const char *two[] = { "==", "!=", "<=", ">=", "&&", "||" };
for (int i = 0; i < 6; i++) {
if (strncmp(&src[pos], two[i], 2) == 0) {
strcpy(t.text, two[i]);
pos += 2;
t.type = TOK_OPERATOR;
return t;
}
}
t.text[0] = c;
t.text[1] = '\0';
pos++;
if (strchr("+-*/=<>!", c)) t.type = TOK_OPERATOR;
else if (strchr("(){};,", c)) t.type = TOK_PUNCT;
else t.type = TOK_ERROR; /* anything we don't recognise */
return t;
}
static const char *type_name(TokenType type) {
switch (type) {
case TOK_KEYWORD: return "KEYWORD";
case TOK_IDENTIFIER: return "IDENTIFIER";
case TOK_NUMBER: return "NUMBER";
case TOK_OPERATOR: return "OPERATOR";
case TOK_PUNCT: return "PUNCTUATION";
case TOK_EOF: return "EOF";
default: return "ERROR";
}
}
int main(void) {
src = "int total = 0; // running sum\n"
"while (total <= 10.5) {\n"
" total = total + 2;\n"
"}\n"
"return total @ 1;\n";
Token t;
do {
t = next_token();
if (t.type == TOK_ERROR)
printf("line %d: unexpected character '%s'\n", t.line, t.text);
else if (t.type != TOK_EOF)
printf("line %d %-12s %s\n", t.line, type_name(t.type), t.text);
} while (t.type != TOK_EOF);
return 0;
}