AI-generated unit tests: how to check they really work
A green test suite proves little. Break your code on purpose to check AI-generated unit tests: one version caught 1 of 5 bugs, the stricter one caught all 5.
If AI writes your unit tests, how do you check them? Break your code on purpose, one small spot at a time, and see whether the tests turn red. A test that passes on correct code proves nothing; its value is that it fails on wrong code. Ever trusted a test just because it was green? Let's try it.
Which tests does the Telegram layer need?
In the previous post, I debugged with AI a bug where two people shared one book in the Telegram bot of an expense book app. This series builds that app from start to finish with AI: you send "coffee 35k" and the app records an expense, you send /total and it replies with this month's total. (The app counts in Vietnamese đồng, so "35k" means 35,000 đồng, about $1.40.) At the end of that post I noticed that changing the signature of createBot broke one of my try-it scripts, while all 23 tests stayed green, because the part that connects to Telegram, bot.ts, has no tests.
I want to close that gap, and it's a good job to hand to AI. Tests for a 28-line file are repetitive and easy to write, exactly the kind of work AI does quickly. But the last post taught me that AI is only as good as the information I give it. "Write tests for bot.ts" doesn't say what the tests should protect, so AI will pick whatever is easiest to write.
First, the file under test. The bot uses grammY 1.46.0, and every text message goes through the handleMessage function from earlier posts:
// Connects the expense book app to Telegram through grammY. All the logic still lives in handleMessage.
import { Bot, type BotConfig, type Context } from "grammy";
import { handleMessage, type Entry } from "./expense-book.ts";
// `unit` only exists to reproduce the time-unit bug from an earlier post; real code always uses "seconds".
export function createBot(token: string, config?: BotConfig<Context>, unit: "seconds" | "milliseconds" = "seconds"): Bot {
const bot = new Bot(token, config);
// One book per Telegram user, keyed by their id. Before, the whole bot shared a single array.
const bookByUser = new Map<number, Entry[]>();
// "message:text" only matches NEW messages with text: not photos, stickers or edited messages.
bot.on("message:text", async (ctx) => {
const userId = ctx.from.id;
let book = bookByUser.get(userId);
if (!book) {
book = [];
bookByUser.set(userId, book);
}
// Telegram sends the time in SECONDS (Unix time), but JavaScript's Date needs milliseconds.
const at = new Date(unit === "seconds" ? ctx.message.date * 1000 : ctx.message.date);
await ctx.reply(handleMessage(ctx.message.text, at, book));
});
// catch takes an error-handler FUNCTION. Without it, an error inside a handler stops the bot receiving messages.
bot.catch((err) => console.error(`Error while handling update ${err.ctx.update.update_id}:`, err.error));
return bot;
}It calls two files from earlier posts. You need both to run the examples, so I'm including them here. The first reads the amount in a message, such as "35k" or "1tr2" (Vietnamese shorthand for 1.2 million):
// Reads one expense message according to SPEC.md (v1.1), section "Rules for reading amounts".
export type Expense = { note: string; amount: number };
const MAX_AMOUNT = 1_000_000_000;
// One amount "token" on its own: 35k · 32K · 3tr · 1tr2 · 45.000 · 50000 · 35 thousand · 20 grand · 2 million
const AMOUNT = /(?<![\p{L}\d])(\d{1,3}(?:[.,]\d{3})+|\d+)(?:(k|tr)(\d)?|\s+(thousand|grand|million))?(?![\p{L}\d])/giu;
const UNIT: Record<string, number> = { k: 1_000, thousand: 1_000, grand: 1_000, tr: 1_000_000, million: 1_000_000 };
function toAmount(digits: string, unit: string | undefined, decimal: string | undefined): number {
const base = Number(digits.replace(/[.,]/g, ""));
if (!unit) return base;
const multiplier = UNIT[unit.toLowerCase()];
// "1tr2": the digit after "tr" is hundreds of thousands. The regex only allows this digit right after "k" or "tr".
if (decimal && multiplier === 1_000_000) return base * multiplier + Number(decimal) * 100_000;
if (decimal) return Number.NaN; // "35k2" is not in the spec
return base * multiplier;
}
export function parseExpense(input: string): Expense | null {
// Text typed on a phone may use combining accents (NFD); normalize so notes like "cà phê" compare consistently.
const message = input.normalize("NFC");
const matches = [...message.matchAll(AMOUNT)];
if (matches.length !== 1) return null; // no amount, or more than one
const [token, digits, shortUnit, decimal, wordUnit] = matches[0];
const amount = toAmount(digits, shortUnit ?? wordUnit, decimal);
if (!Number.isInteger(amount) || amount <= 0 || amount > MAX_AMOUNT) return null;
const note = message.replace(token, " ").replace(/\s+/g, " ").trim();
if (!note) return null;
return { note, amount };
}The second file records an expense and answers the /total command. It works out the month in Vietnam time, even when the server runs on UTC:
// Handles one message according to SPEC.md (v1.2): record an expense, or answer a command.
// Data is kept in memory for now; later posts store it in a database.
import { parseExpense } from "./parse-expense.ts";
export type Entry = { note: string; amount: number; at: Date };
const HELP = 'Sorry, I didn\'t understand. Send something like "coffee 35k" to record an expense, or /total for this month\'s total.';
// Create it ONCE and reuse it: a new Intl.DateTimeFormat on every call is dozens of times slower.
// "en-GB" because it gives exactly "12/2026"; "vi-VN" gives "tháng 12, 2026".
const VN_MONTH = new Intl.DateTimeFormat("en-GB", { timeZone: "Asia/Ho_Chi_Minh", month: "2-digit", year: "numeric" });
// Month/year in Vietnam time, even when the server runs on UTC.
function vnMonth(date: Date): string {
const parts = VN_MONTH.formatToParts(date);
const month = parts.find((p) => p.type === "month")?.value;
const year = parts.find((p) => p.type === "year")?.value;
if (!month || !year) throw new Error(`Could not get month/year from ${date.toISOString()}`);
return `${month}/${year}`;
}
const vnd = (n: number) => `${n.toLocaleString("en-US")} đồng`;
export function monthTotal(entries: Entry[], now: Date): string {
const month = vnMonth(now);
const inMonth = entries.filter((entry) => vnMonth(entry.at) === month);
if (inMonth.length === 0) return `No expenses yet in ${month}.`;
const total = inMonth.reduce((sum, entry) => sum + entry.amount, 0);
const count = `${inMonth.length} ${inMonth.length === 1 ? "expense" : "expenses"}`;
return `${month}: ${count}, total ${vnd(total)}`;
}
export function handleMessage(text: string, now: Date, entries: Entry[]): string {
const message = text.trim().normalize("NFC");
// A message starting with "/" is a command and is never an expense (spec v1.2).
if (message.startsWith("/")) {
const command = message.toLowerCase();
if (command === "/total" || command === "/tổng") return monthTotal(entries, now);
return `I don't have a command called ${message.split(/\s/)[0]}. Send /total to see this month's total.`;
}
const expense = parseExpense(message);
if (!expense) return HELP;
entries.push({ ...expense, at: now });
return `Recorded: ${expense.note}, ${vnd(expense.amount)}`;
}To try the bot without a Telegram token or a network, I use the approach from the previous post and put it in a shared file. It builds the bot with fake bot information so grammY doesn't ask Telegram, intercepts every outgoing call with a transformer and records the text the bot meant to send, then feeds fake updates into bot.handleUpdate:
// Tools for testing the bot without a token or a network: build the bot with fake botInfo, intercept every call
// to Telegram with a transformer and record it, then feed fake updates into bot.handleUpdate. Shared by the test files.
import type { Update, UserFromGetMe } from "grammy/types";
import { createBot } from "./bot.ts";
const botInfo = {
id: 1, is_bot: true, first_name: "Expense book", username: "expense_book_bot",
can_join_groups: false, can_read_all_group_messages: false, supports_inline_queries: false,
can_connect_to_business: false, has_main_web_app: false, has_topics_enabled: false,
allows_users_to_create_topics: false, can_manage_bots: false, supports_join_request_queries: false,
} satisfies UserFromGetMe;
const seconds = (vnIso: string) => Math.floor(new Date(`${vnIso}+07:00`).getTime() / 1000);
export function createTestBot() {
const bot = createBot("fake-token", { botInfo });
const sent: string[] = [];
bot.api.config.use(async (_prev, _method, payload) => {
sent.push((payload as { text?: string }).text ?? "");
return { ok: true, result: { message_id: 0, date: 0, chat: { id: 0, type: "private", first_name: "" }, text: "" } } as never;
});
let updateId = 0;
const chatOf = (id: number) => ({ id, type: "private" as const, first_name: `User ${id}` });
const userOf = (id: number) => ({ id, is_bot: false, first_name: `User ${id}` });
return {
bot, // for tests that need to call the bot directly
sent,
/** User `id` sends `text` at `at` (Vietnam time, ISO format). Returns the bot's reply, or null if the bot stays silent. */
async send(id: number, text: string, at = "2026-12-01T06:30"): Promise<string | null> {
sent.length = 0;
const update: Update = { update_id: ++updateId, message: { message_id: updateId, date: seconds(at), chat: chatOf(id), from: userOf(id), text } };
await bot.handleUpdate(update);
return sent.at(-1) ?? null;
},
/** User `id` sends a photo with no text. */
async sendPhoto(id: number, at = "2026-12-01T06:30"): Promise<string | null> {
sent.length = 0;
const update: Update = { update_id: ++updateId, message: { message_id: updateId, date: seconds(at), chat: chatOf(id), from: userOf(id), photo: [{ file_id: "f", file_unique_id: "u", width: 90, height: 90 }] } };
await bot.handleUpdate(update);
return sent.at(-1) ?? null;
},
/** User `id` EDITS an older message so it now says `text`. */
async edit(id: number, text: string, at = "2026-12-01T06:30"): Promise<string | null> {
sent.length = 0;
const update: Update = { update_id: ++updateId, edited_message: { message_id: 1, date: seconds(at), edit_date: seconds(at), chat: chatOf(id), from: userOf(id), text } };
await bot.handleUpdate(update);
return sent.at(-1) ?? null;
},
};
}Put these four files in a src/ folder, create a package.json containing "type": "module", and run npm install grammy@1.46.0. You need Node.js 22.18 or later to run .ts files directly. From here on, we get to the point of the post.
What does a test file that "looks complete" look like?
A caveat before we continue. I didn't copy the two test files below from one specific AI run. Both are my own versions, written to illustrate the point. The first follows the style you often get when you just ask for "tests for bot.ts" and say nothing more: one test per feature, short, and it looks thorough.
// The "looks complete" test file: one test per feature, and all of them pass. The kind you often get when you ask
// AI to write tests without saying what they need to protect.
import assert from "node:assert/strict";
import { test } from "node:test";
import { createTestBot } from "./bot-harness.ts";
test("the bot replies when it gets an expense message", async () => {
const bot = createTestBot();
const reply = await bot.send(42, "coffee 35k");
assert.ok(reply);
assert.ok(reply.includes("Recorded"));
});
test("/total gets a reply", async () => {
const bot = createTestBot();
await bot.send(42, "coffee 35k");
const reply = await bot.send(42, "/total");
assert.ok(reply);
assert.ok(reply.includes("total"));
});
test("a message it can't understand gets the help text", async () => {
const bot = createTestBot();
const reply = await bot.send(42, "hello");
assert.ok(reply?.includes("didn't understand"));
});Run it with Node.js's built-in test runner, node:test, which needs nothing extra installed:
# tests 3
# pass 3
# fail 0Three green tests, and on a read-through they look sensible. Each has a clear description, and each checks one feature of the bot. If you were reviewing this pull request, would you approve it? I think many people would.
What does 100% coverage tell you?
A common way to judge tests is code coverage: how many lines of bot.ts ran while the tests ran. Node.js measures this with the --experimental-test-coverage flag, as described in the documentation on collecting code coverage:
# file | line % | branch % | funcs % | uncovered lines
# bot.ts | 100.00 | 80.00 | 66.67 |Every line of bot.ts ran. Someone looking at that number would relax, and I think this is where many people who ask AI for tests get fooled. Coverage only says a line ran. It doesn't say any test checked the result of that line. The first test runs through the code that creates a user's own book, but it only uses one user, so it can't tell "one book per user" from "one book for the whole bot".
Research on this topic says much the same thing. Meta's paper Mutation-Guided LLM-based Test Generation at Meta describes using bugs deliberately planted in code to guide test generation, instead of looking only at coverage. I'm only borrowing the main idea from that paper, none of its numbers. For this post, I'll show you the idea with a small experiment.
How do you test a test by breaking the code on purpose?
The idea is simple. If a test is any use, it must fail when the code is wrong. So I deliberately make the code wrong, one spot at a time, and count how many times the tests notice. The technique is called mutation testing, and tools exist that do it automatically. Here I do it by hand so you can see each step.
I chose five broken spots, each for a reason. Two are real bugs from earlier posts in this project: the whole bot sharing one book, and mixing up seconds and milliseconds. Two come from the spec: photos must not reach the handler, and an edited message must not record another expense. The last one is the most obvious bug of all: the bot no longer calls handleMessage.
// "Breaking the code on purpose" (mutation testing by hand): change one small thing in bot.ts, then run one test file.
// A good test must go RED when the code is broken. If it stays green, it doesn't protect the spot you just changed.
// Usage: node mutate.ts src/bot-weak.test.ts
import { spawnSync } from "node:child_process";
import { cpSync, readFileSync, rmSync, writeFileSync } from "node:fs";
const testFile = process.argv[2];
if (!testFile) throw new Error("Pass a test file name, for example: node mutate.ts src/bot.test.ts");
// Each entry: a name, the original code in bot.ts, and the broken code to put in its place.
const MUTATIONS: [string, string, string][] = [
["the whole bot shares one book (bug from post 5)", "const userId = ctx.from.id;", "const userId = 0;"],
["seconds mixed up with milliseconds (bug from post 4)", 'unit === "seconds" ? ctx.message.date * 1000 : ctx.message.date', "ctx.message.date"],
['"message:text" becomes "message": photos reach the handler too', 'bot.on("message:text"', 'bot.on("message"'],
["edited messages are recorded as new expenses too", 'bot.on("message:text"', 'bot.on(["message:text", "edited_message:text"]'],
["the bot no longer calls handleMessage", "await ctx.reply(handleMessage(ctx.message.text, at, book));", 'await ctx.reply("Recorded");'],
];
const DIR = ".mutate"; // temporary copy of src/, deleted when done; the real bot.ts is never touched
const original = readFileSync("src/bot.ts", "utf8");
const run = () => spawnSync(process.execPath, ["--test", `${DIR}/${testFile.replace(/^src\//, "")}`], { encoding: "utf8" }).status === 0;
console.log(`Checking ${testFile}:`);
cpSync("src", DIR, { recursive: true });
console.log(` untouched code: ${run() ? "green (as expected)" : "RED, fix the tests first"}`);
let caught = 0;
for (const [name, from, to] of MUTATIONS) {
if (!original.includes(from)) throw new Error(`Could not find the code to change in bot.ts: ${from}`);
writeFileSync(`${DIR}/bot.ts`, original.replace(from, to));
const detected = !run();
if (detected) caught += 1;
console.log(` ${detected ? "caught " : "MISSED "} ${name}`);
}
rmSync(DIR, { recursive: true, force: true });
console.log(` => caught ${caught}/${MUTATIONS.length} broken spots`);The script copies the src folder to a temporary folder, changes bot.ts in the copy and runs the tests there. That way the real bot.ts is never touched, even if the script crashes halfway. The original.includes(from) check makes the script fail loudly if someone edits bot.ts and the code to break no longer exists, instead of quietly reporting "caught" for a bug that was never planted. Here it is running against the "looks complete" tests:
Checking src/bot-weak.test.ts:
untouched code: green (as expected)
MISSED the whole bot shares one book (bug from post 5)
MISSED seconds mixed up with milliseconds (bug from post 4)
MISSED "message:text" becomes "message": photos reach the handler too
MISSED edited messages are recorded as new expenses too
caught the bot no longer calls handleMessage
=> caught 1/5 broken spotsThree green tests, 100% line coverage, and only 1 of 5 broken spots caught. The two real bugs that cost this project a whole post to find slipped straight through. Put the "whole bot shares one book" bug back and the tests stay green, because every test uses a single user. The seconds-versus-milliseconds bug slips through too, because no test checks which month an expense lands in.
What I find striking is that the one test that caught something caught the easiest bug. When bot.ts stops calling handleMessage, the reply changes completely, so any test that checks the content of the reply goes red. The subtler bugs get through. Those are also the kind you tend to meet in real code.
What makes the stricter tests different?
The second version differs in one core way: every test starts from something worth protecting, then the code is written. That something can be a bug that really happened or a line in the spec. I put the reason into the test name, so whoever reads it later knows what protection they lose by deleting it:
// Tests for bot.ts: each test protects ONE thing that once broke or is written in SPEC.md, and says which.
import assert from "node:assert/strict";
import { test } from "node:test";
import { createTestBot } from "./bot-harness.ts";
test("one book per person: my /total doesn't add up my friend's expenses (bug from post 5)", async () => {
const bot = createTestBot();
await bot.send(42, "coffee 35k");
await bot.send(77, "milk tea 32k");
assert.equal(await bot.send(42, "/total"), "12/2026: 1 expense, total 35,000 đồng");
assert.equal(await bot.send(77, "/total"), "12/2026: 1 expense, total 32,000 đồng");
});
test("Telegram sends the time in seconds: 6:30 a.m. on 1 December in Vietnam belongs to December (bug from post 4)", async () => {
const bot = createTestBot();
await bot.send(42, "milk tea 32k", "2026-12-01T06:30");
assert.equal(await bot.send(42, "/total", "2026-12-01T06:31"), "12/2026: 1 expense, total 32,000 đồng");
});
test("a photo message, with no text, leaves the bot silent", async () => {
const bot = createTestBot();
assert.equal(await bot.sendPhoto(42), null);
});
test("editing an old message does not record another expense", async () => {
const bot = createTestBot();
await bot.send(42, "coffee 35k");
assert.equal(await bot.edit(42, "coffee 45k"), null);
assert.equal(await bot.send(42, "/total"), "12/2026: 1 expense, total 35,000 đồng");
});Two more differences. The checks use assert.equal against the full reply instead of includes, so a wrong number or a wrong word turns the test red. And the first test uses two users instead of one, because the shared-book bug only shows up when there are two. I ran the same script:
Checking src/bot.test.ts:
untouched code: green (as expected)
caught the whole bot shares one book (bug from post 5)
caught seconds mixed up with milliseconds (bug from post 4)
caught "message:text" becomes "message": photos reach the handler too
caught edited messages are recorded as new expenses too
caught the bot no longer calls handleMessage
=> caught 5/5 broken spotsI also measured this version's coverage: 100% of lines, 80% of branches and 66.67% of functions, identical to the first version. Two test files with the same coverage, one catching 1 of 5 broken spots and the other catching all 5. That's why I don't use coverage to decide whether to trust a test.
The "photo stays silent" test catches the message bug through a small detail worth knowing. When the handler receives a photo, ctx.message.text doesn't exist and handleMessage throws. bot.handleUpdate passes that error back to its caller, so the test goes red because of an exception rather than a wrong reply. When the bot runs for real with bot.start(), the error goes into bot.catch and is only logged. The path differs, but the test still does its job: it tells you something unusual happened.
The whole project now has 27 tests:
# tests 27
# pass 27
# fail 0expense-book.test.ts and parse-expense.test.ts are the test files from the earlier posts, 23 tests between them, as mentioned above. If you don't have them, running bot.test.ts alone is enough to follow this post.
What are four steps to check a test that AI wrote?
After this experiment, here is the short routine I'll use whenever I ask AI for tests:
- Say what needs protecting when you ask. Instead of "write tests for bot.ts", I'll list it: two users must not share a book, time is counted in seconds, photos must not record an expense. That list is also the list of broken spots to try in step 3.
- Run the tests against the correct code. They must pass. But this is only the first step and proves nothing yet.
- Break the code on purpose. For each item from step 1, change the code so it's wrong and run the tests again. They must go red. If they stay green, ask AI to fix the tests, and this time tell it which broken spot got through.
- Read every `assert` line.
assert.ok(reply)orincludes("Recorded")sound like they check a lot, but they pass as long as the bot sends any string containing that word. Where I can, I prefer comparing against the complete string.
Doing step 3 by hand, as in this post, gets slow on a big project, but it works fine for the few important broken spots you already know about. For bigger projects, tools such as Stryker generate many broken spots and run them automatically. I haven't tried Stryker in this series, so I won't comment on it.
What kind of mistake does this experiment not catch?
One more thing before I finish. When you ask AI to write tests, there's a kind of mistake the experiment above can't catch: the AI writes a test that calls a function that doesn't exist. Here's a short example using the harness above and a function called processUpdate, which grammY 1.46.0 doesn't have:
// A line that sounds very reasonable, but grammY 1.46.0 has no processUpdate function.
import assert from "node:assert/strict";
import { test } from "node:test";
import { createTestBot } from "./bot-harness.ts";
test("the bot receives an update and replies", async () => {
const harness = createTestBot();
const update = { update_id: 1, message: { message_id: 1, date: 1796000000, chat: { id: 42, type: "private" }, from: { id: 42, is_bot: false, first_name: "A" }, text: "coffee 35k" } };
await harness.bot.processUpdate(update);
assert.ok(harness.sent.length > 0);
}); error: 'harness.bot.processUpdate is not a function'
name: 'TypeError'
# pass 0
# fail 1This time the test goes red immediately, because Node.js finds the missing function when it runs. But that's only because the line sits inside a test that runs. If a made-up function sits in a branch that rarely executes, it blows up only when a user reaches it. And AI can make up both the function name and its parameters while sounding perfectly sure. Checking every line of code AI writes, not just the tests, is the subject of later posts.
You can follow the whole series at Building a side project with AI. If you're preparing for interviews, the questions on using AI while coding are in the interview question bank.