CKTIS / JavaScript
CSV tokenizer and parser state machine
Use a finite-state tokenizer to identify CSV fields and a parser to assemble records without splitting at commas or line breaks inside quoted values.
Version: ECMAScript 2024 and the current web platform; runtime support varies
Syntax
csvTokens(text) → iterable of { kind: 'field', value: string } or { kind: 'endRecord' }
parseCsv(text) → string[][]
Both functions are defined in the example; neither is a built-in API.A CSV tokenizer and parser state machine separates two jobs: csvTokens recognizes fields and emits field and endRecord tokens; parseCsv collects field values into a record and saves it on endRecord. These are custom JavaScript functions, not built-in APIs. Fields stay strings, spaces are preserved, and a header is treated like any other record. The parser does not check column counts.
The tokenizer has four states: start (at the start of a field), bare (in an unquoted field), quoted (inside a quoted field), and afterQuote (just after a quote inside a quoted field). Only start lets a quote open a field. In quoted, commas, LF, and CR are data. In afterQuote, another quote appends one literal " and returns to quoted; otherwise only a comma, record ending, or end of input is allowed. This is why split(',') alone cannot parse this dialect.
Outside quotes, a comma ends a field and LF or CRLF ends a record. End of input finishes a pending record, including the empty field after a trailing comma. Empty input gives zero records; a blank line gives a record with one empty field; a final record separator adds no extra record. A lone CR outside quotes, an unexpected quote, extra text after a closing quote, or an unclosed quoted field causes SyntaxError. Reported offsets are zero-based UTF-16 code-unit positions, not byte offsets.
The example takes one already decoded string. The generator yields tokens incrementally but does not read input in chunks, and the parser retains all records. A chunked implementation must carry state and partial field data across boundaries, including ones that split CRLF or "". Set size limits for untrusted input, and address spreadsheet formula interpretation before handing fields to a spreadsheet; CSV parsing does not neutralize formulas.
Examples
function* csvTokens(text) {
let state = 'start';
let field = '';
let pending = false; // A comma has opened another field in this record.
for (let i = 0; i < text.length; i++) {
const c = text[i];
if (state === 'quoted') {
if (c === '"') state = 'afterQuote';
else field += c;
continue;
}
if (state === 'afterQuote' && c === '"') {
field += '"';
state = 'quoted';
continue;
}
const crlf = c === '\r' && text[i + 1] === '\n';
const endRecord = c === '\n' || crlf;
if (c === ',' || endRecord) {
yield { kind: 'field', value: field };
if (endRecord) yield { kind: 'endRecord' };
state = 'start';
field = '';
pending = c === ',';
if (crlf) i++;
} else if (state === 'start' && c === '"') {
state = 'quoted';
} else if (c === '\r' || c === '"' || state === 'afterQuote') {
throw new SyntaxError(`Unexpected character at UTF-16 offset ${i}`);
} else {
field += c;
state = 'bare';
}
}
if (state === 'quoted') throw new SyntaxError('Unclosed quoted field');
if (state !== 'start' || pending) {
yield { kind: 'field', value: field };
yield { kind: 'endRecord' };
}
}
function parseCsv(text) {
if (typeof text !== 'string') throw new TypeError('Expected a string');
const rows = [];
let row = [];
for (const token of csvTokens(text)) {
if (token.kind === 'field') row.push(token.value);
else {
rows.push(row);
row = [];
}
}
return rows;
}
const csv = 'name,note,extra\r\nAda,"line 1, still here\nline ""2""",';
console.log(JSON.stringify(parseCsv(csv)));Result
[["name","note","extra"],["Ada","line 1, still here\nline \"2\"",""]]
The input uses CRLF between records, but the comma and LF inside `note` stay in the value. Doubled quotes produce one quote on each side of `2`, and the trailing comma adds an empty field. `JSON.stringify` displays the LF as `\n`.
Related lessons
- JavaScript · Lesson 9 · Advanced · Version: 4