Verify AI-generated code: catching made-up APIs

What does it mean when AI makes up an API? It writes code that calls a function or option the library doesn't have, yet reads perfectly well. You catch it by checking with machines: the compiler, the library's own types and tests. Have you ever pasted AI code in and only then found out it doesn't exist? Let's try.

The AI sounds right, but is it right?

In the previous post, my expense book app (a Telegram bot that records expenses, written in TypeScript with the grammY library) got tests for the layer that connects it to Telegram. While writing that post, something happened: one test called a function processUpdate that grammY doesn't have, and it only showed up when the test ran. That's not rare, and I'd like to look at it properly instead of treating it as bad luck.

Here's the situation. The app's spec moves to v1.3 with a new requirement: when a user edits a message they already sent, say from "coffee 35k" to "coffee 45k", the matching expense is updated instead of being ignored as it is now. I ask AI to write that feature. Below is the code an AI tool might return. To be clear up front: this is a version I built to illustrate, not taken from one specific AI run, but all three kinds of mistakes in it are kinds I've seen AI make.

// The code "the AI wrote" when asked: "when a user edits a message, update the expense that was recorded".
// This is a version I built to illustrate, not taken from one specific AI run. It reads very reasonably and has three made-up spots.
import { Bot } from "grammy";

import { parseExpense } from "../src/parse-expense.ts";
import type { Entry } from "../src/expense-book.ts";

const bot = new Bot("fake-token");
const bookByUser = new Map<number, Entry[]>();

bot.on("edited_message.text", async (ctx) => {
  const book = bookByUser.get(ctx.from.id) ?? [];
  const newExpense = parseExpense(ctx.editedMessage.text ?? "");
  book.updateEntry(ctx.editedMessage.message_id, newExpense);
  await ctx.replyWithMarkdown("Expense updated");
});

console.log("Edited-message handler registered.");

Two things are imported from earlier posts: parseExpense reads a message like "coffee 35k" into { note, amount } (the amount in Vietnamese đồng, so 35k means 35,000 đồng, about $1.40), and Entry is a recorded expense (note, amount, time).

At a glance the whole snippet looks fine. There are three suspicious spots, and I'd like you to try finding them before reading on:

1. The filter "edited_message.text" is written with a dot.
2. book.updateEntry(...), where book is an array of Entry.
3. ctx.replyWithMarkdown(...), a function from Telegraf.

All three fit the "taste" of a library: a method name like updateEntry, a replyWithMarkdown that does exist somewhere, and a dot is the familiar way to write a property path. That is exactly why they slip past human eyes.

Does running the code catch it?

The first thing everyone does is run the file. Install the library, then run it with Node.js 22.18 or later (from that version Node runs .ts files directly). I pin grammY to version 1.46.0, because the results below come from exactly that version:

npm init -y
npm pkg set type=module
npm install --save-exact grammy@1.46.0
npm install --save-exact --save-dev typescript@7.0.2 @types/node@22

Error: Invalid L1 filter 'edited_message.text' given in 'edited_message.text'. Permitted values are: 'message', 'edited_message', 'channel_post', 'edited_channel_post', 'business_connection', ...
    at check (node_modules/grammy/out/filter.js:110:15)

I shortened the error line (it lists a long set of update types) and removed the full file path. This run caught the first mistake right when the handler was registered. But look at what didn't happen: the other two made-up spots sit inside the handler, and the handler only runs when a real person edits a message. If grammY hadn't validated the filter at registration time, all three mistakes would have sat quietly in the code until a user edited a message. Then the bot would not reply, and you'd only find out from a log line at two in the morning.

That's the basic limit of verifying by running: JavaScript only finds errors on the paths the program actually takes. Each run shows one error, you fix it, and only then do you reach the next.

How many does the TypeScript compiler catch?

The second way is to let the compiler read the whole file without running it. Add two configuration files: tsconfig.json for the main code, and a separate one for the ai-made-up/ folder so the sample's errors don't mix with the rest:

{
  "compilerOptions": {
    "target": "ES2023",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "allowImportingTsExtensions": true,
    "noEmit": true,
    "strict": true,
    "skipLibCheck": true,
    "types": ["node"]
  },
  "include": ["src"]
}

{
  "extends": "./tsconfig.json",
  "include": ["ai-made-up"]
}

ai-made-up/bot-edit.ts(11,8): error TS2345: Argument of type '"edited_message.text"' is not assignable to parameter of type 'ComputeFilterQueryList[] | ComputeFilterQueryList'.
ai-made-up/bot-edit.ts(12,31): error TS18048: 'ctx.from' is possibly 'undefined'.
ai-made-up/bot-edit.ts(13,35): error TS18048: 'ctx.editedMessage' is possibly 'undefined'.
ai-made-up/bot-edit.ts(14,8): error TS2339: Property 'updateEntry' does not exist on type 'Entry[]'.
ai-made-up/bot-edit.ts(14,20): error TS18048: 'ctx.editedMessage' is possibly 'undefined'.
ai-made-up/bot-edit.ts(15,13): error TS2339: Property 'replyWithMarkdown' does not exist on type 'FilteredContext<Context, ...

I cut the end of the last line, because that message runs on with grammY's types. The compiler reports all six errors in one go, with nobody fixing them one at a time. Three of the errors belong to the three made-up spots (lines 11, 14 and 15). The other three, "possibly undefined", are side effects: because the filter is wrong, grammY doesn't know ctx is an edited message, and treats every field as possibly missing.

There's a small lesson here that I find worth remembering. When the compiler reports many errors, fix the first one and run again before touching the whole list, because one made-up spot can cause several knock-on errors. I changed the filter to "edited_message:text" (with a colon) and ran it again:

ai-made-up/bot-edit.ts(14,8): error TS2339: Property 'updateEntry' does not exist on type 'Entry[]'.
ai-made-up/bot-edit.ts(15,13): error TS2339: Property 'replyWithMarkdown' does not exist on type 'FilteredContext<Context, ...

Only the two real errors remain. The three "possibly undefined" errors disappeared, because grammY now knows ctx.from and ctx.editedMessage are definitely there.

If AI makes up a function name at the head of a call chain (say a non-existent bot.onEditedMessage(...)), the parameter ctx of the function inside loses its type, and the compiler may not report the errors inside the function body. So a clean tsc run doesn't mean every line is correct, only that every line the compiler could understand is. I use tsc as a coarse net, not as final proof.

What does the library really have?

For the remaining two errors, I need to know what the right function is. Asking the AI again, "is there a replyWithMarkdown function?", is the easiest way and also the least reliable, because it may answer confidently in the same old way. A more trustworthy source sits right in the node_modules folder: the library's type files (.d.ts) are the real contract between the library and your code.

grep -rn "replyWithMarkdown" node_modules/grammy/out/
grep -n "^    reply(" node_modules/grammy/out/context.d.ts

(the first command prints nothing: no function is called replyWithMarkdown)
483:    reply(text: string, other?: Other<"sendMessage", "chat_id" | "text">, signal?: AbortSignal): Promise<Message.TextMessage>;

The first command found nothing, so replyWithMarkdown isn't in grammY. The second shows that reply takes a second parameter, other, which holds the options of Telegram's sendMessage call. Among those options is parse_mode, which chooses Markdown or HTML. So the right form is ctx.reply(text, { parse_mode: "Markdown" }). I don't need Markdown for a line like "Updated", so I just use ctx.reply(text).

It's worth cross-checking with the official documentation too. The section on filter queries in the grammY documentation explains that the filter string joins the update type and the content type with a colon, as in "message:text" or "edited_message:text". And on the point that each message has its own identifier, the Telegram Bot API documentation says message_id is unique inside a single chat. I need this for the next step, and it's the kind of information the library's types can't tell me, so I had to read the docs.

The error on book.updateEntry is different: it isn't made up from any library, it's a function made up on my own code. The AI doesn't see all of your code at every moment. A JavaScript array has no updateEntry, and nothing like that exists here, so the fix is to write the logic out: find the expense by message id and change it in place.

Rewrite it with the real API, and test it

To find the right expense again when a message is edited, I need to store the id of the message that created it. Two small changes in src/expense-book.ts: add a messageId field to Entry, and export the vnd function (money formatting) so it can be shared:

// messageId: the id of the Telegram message that created the expense, so it can be found again when the user edits that message.
export type Entry = { note: string; amount: number; at: Date; messageId?: number };

export const vnd = (n: number) => `${n.toLocaleString("en-US")} đồng`;

Then src/bot.ts, the whole file. The older parts from earlier posts, a separate book per person and converting seconds to milliseconds, stay as they were, and I add the new part:

// Connects the expense book app to Telegram through grammY. All the logic still lives in handleMessage.
import { Bot, type BotConfig, type Context } from "grammy";

import { parseExpense } from "./parse-expense.ts";
import { handleMessage, vnd, type Entry } from "./expense-book.ts";

// `unit` only exists to reproduce the time-unit bug from an earlier post; real code always uses "seconds".
export function createBot(token: string, config?: BotConfig<Context>, unit: "seconds" | "milliseconds" = "seconds"): Bot {
  const bot = new Bot(token, config);
  // One book per Telegram user, keyed by their id. Before, the whole bot shared a single array.
  const bookByUser = new Map<number, Entry[]>();

  // "message:text" only matches NEW messages with text: not photos, stickers or edited messages (see the handler below).
  bot.on("message:text", async (ctx) => {
    const userId = ctx.from.id;
    let book = bookByUser.get(userId);
    if (!book) {
      book = [];
      bookByUser.set(userId, book);
    }
    // Telegram sends the time in SECONDS (Unix time), but JavaScript's Date needs milliseconds.
    const at = new Date(unit === "seconds" ? ctx.message.date * 1000 : ctx.message.date);
    const countBefore = book.length;
    const reply = handleMessage(ctx.message.text, at, book);
    // Remember which message created the expense that was just recorded, so it can be found again when the user edits it.
    if (book.length > countBefore) book[book.length - 1].messageId = ctx.message.message_id;
    await ctx.reply(reply);
  });

  // The user edits a message they already sent (spec v1.3): update exactly the expense that message created, don't add another.
  // The filter is "edited_message:text" (with a colon): an edited message is its own kind of update, different from "message".
  bot.on("edited_message:text", async (ctx) => {
    const book = bookByUser.get(ctx.from.id) ?? [];
    const entry = book.find((item) => item.messageId === ctx.editedMessage.message_id);
    if (!entry) return; // edited a message that wasn't an expense, or the bot restarted and forgot the book
    const newExpense = parseExpense(ctx.editedMessage.text.trim().normalize("NFC"));
    if (!newExpense) return void (await ctx.reply("I can't read an expense after the edit, so I kept the old one."));
    entry.note = newExpense.note;
    entry.amount = newExpense.amount;
    await ctx.reply(`Updated: ${newExpense.note}, ${vnd(newExpense.amount)}`);
  });

  // catch takes an error-handler FUNCTION. Without it, an error inside a handler stops the bot receiving messages.
  bot.catch((err) => console.error(`Error while handling update ${err.ctx.update.update_id}:`, err.error));
  return bot;
}

Two points in this code deserve your attention, because each came from reading documentation rather than from the AI's memory. First, the search for the expense happens in the sender's own book (bookByUser.get(ctx.from.id)). Since message_id is only unique within one chat, two different people can have messages with the same id, and if I searched a shared book, one person could accidentally edit the other's expense. Second, if no expense is found (that message wasn't an expense, or the bot just restarted and the in-memory book is gone), the bot stays silent instead of recording a new one.

To run tests against this bot, the edit function in bot-harness.ts from the previous post takes one more argument, the id of the message to edit. Here's the change; the default is message number 1 so the old tests still run:

/** User `id` EDITS the message with id `messageId` so it now says `text` (by default the first message the bot received, id 1). */
async edit(id: number, text: string, messageId = 1, at = "2026-12-01T06:30"): Promise<string | null> {
  sent.length = 0;
  const update: Update = { update_id: ++updateId, edited_message: { message_id: messageId, date: seconds(at), edit_date: seconds(at), chat: chatOf(id), from: userOf(id), text } };
  await bot.handleUpdate(update);
  return sent.at(-1) ?? null;
},

And the tests. The old test "editing an old message does not record another expense" described v1.2 behavior and is no longer correct, so I replace it with four v1.3 tests. Each has a reason:

test("editing a message updates exactly that expense, without adding another", async () => {
  const bot = createTestBot();
  await bot.send(42, "coffee 35k"); // message number 1
  assert.equal(await bot.edit(42, "coffee 45k", 1), "Updated: coffee, 45,000 đồng");
  assert.equal(await bot.send(42, "/total"), "12/2026: 1 expense, total 45,000 đồng");
});

test("editing into text that can't be read keeps the old expense", async () => {
  const bot = createTestBot();
  await bot.send(42, "coffee 35k");
  assert.equal(await bot.edit(42, "coffee", 1), "I can't read an expense after the edit, so I kept the old one.");
  assert.equal(await bot.send(42, "/total"), "12/2026: 1 expense, total 35,000 đồng");
});

test("editing a message that wasn't an expense leaves the bot silent", async () => {
  const bot = createTestBot();
  await bot.send(42, "/total"); // message number 1 is a command, it creates no expense
  assert.equal(await bot.edit(42, "/total this month", 1), null);
});

test("editing your own message doesn't touch anyone else's book", async () => {
  const bot = createTestBot();
  await bot.send(42, "coffee 35k"); // message number 1, from user 42
  assert.equal(await bot.edit(77, "coffee 99k", 1), null); // user 77 "edits" message 1: it isn't in their book
  assert.equal(await bot.send(42, "/total"), "12/2026: 1 expense, total 35,000 đồng");
});

Run the compiler over the src folder and then all the tests:

(prints nothing: no errors, exit code 0)

# tests 30
# pass 30
# fail 0

I left out the per-test detail lines and copied only Node.js's summary at the end. Thirty tests are green, including the four new ones.

What do all three layers of checking still miss?

I want to be clear about the limits of this approach, so you don't feel too safe. Running catches errors on the paths that ran. The compiler catches errors in the shape of calls. Tests catch the behavior you thought of checking. What's left is behavior that has the right shape but the wrong meaning, and machines have no way to know.

For example, if AI wrote bot.on("edited_message:text", ...) and then searched for the expense in one book shared by everyone instead of in each person's own book, all three layers above would stay green. That bug would only surface when two people use the bot at the same time, and I only catch it because of the test "editing your own message doesn't touch anyone else's book", which I wrote because I'd read that message ids are unique only within a chat. In the end it's still a human who thinks of the cases to check.

The way I use AI after this post: when it gives me a function or an option I haven't seen, I treat it as a question to answer, not as an answer. I search for the name in the library's code, and if I can't find it, I don't use it. It's a cheap habit, under a minute each time.

What happens when the Git history turns into a mess?

By now the message-editing feature works and has tests. But if you did all of this in one evening and committed each step with names like "fix", "fix again" and "ok", then the next morning nobody can read the Git history and understand what a commit did, including you. One change that spans a spec update, a code fix, a test change and a new verification file gets sliced up by the order you remembered things, not by meaning. How to write commits and pull request descriptions that later readers can follow, even when you ask AI to help write them, is the job of later posts.

You can follow the whole journey at Building a side project with AI. If you're preparing for interviews, the questions on using AI while coding are in the interview question bank.