AI Generated Comments And Docs You Can’t Trust (And What To Do Instead)

When Documentation Looks Solid But Isn’t

This article builds on an earlier piece about debugging an AI generated startup codebase, but you can also read it on its own.

In that same AI heavy startup codebase I was helping to stabilize, one thing kept bothering me.

On paper, the project was “well documented.” There were long comments everywhere, big architecture docs, and auto generated README sections.

In practice, I could not trust any of it.

This is an honest look at why AI generated comments and documentation can quietly hurt teams, and some simple rules that made my own work easier when AI is in the loop.

The new problem: confident but untrustworthy docs

If you have worked in older codebases, you know the classic documentation pain:

Outdated wiki pages
Cryptic comments in key functions
No big picture diagrams at all

With AI, the surface problem can invert:

It is trivial to ask a model to document a module, write a high level overview, or produce a “system design.”
Modern coding assistants integrate inline comment generation into the IDE.

It feels great. Until you notice some patterns.

In that startup codebase, I kept seeing things like:

Function comments that explained “what” in generic terms, but skipped the weird edge cases that actually matter
Docs that described flows that no longer existed because the code had been refactored without regenerating the docs
Diagrams and high level docs written in the same generic tone across the whole project, which is a smell on its own

The dangerous part is not that docs are missing. It is that they pretend to be reliable while being wrong or incomplete.

That matches what Google’s helpful content guidance warns against on the content side: verbose text that looks informative but is not grounded in real expertise or fresh verification.

Inside a codebase, this “fake helpfulness” can be worse than silence.

Why AI generated docs drift so easily

A few reasons this happens, especially in fast paced teams:

Docs are generated once, then never updated
- The team ships something, runs “generate docs” in their IDE, commits, and moves on.
- Nobody owns keeping those docs in sync.
The model describes the intention, not the reality
- If you prompt “explain what this module is supposed to do,” it will often paraphrase the code as if everything is correct.
- It does not know that the implementation is partial, buggy, or obsolete.
Verbosity hides uncertainty
- Long explanations make it harder to notice the missing pieces.
- Readers assume “if someone wrote this much, they must know what they are talking about.”
No one feels responsible for AI written text
- It is “the tool’s” writing.
- People are less likely to delete or aggressively edit text they did not hand write themselves.

When you mix these with a “ship fast” culture, you get walls of documentation that do not deserve the trust they ask for.

How I now treat AI generated comments by default

My own mental model shifted during that experience.

I started treating AI comments and docs as:

Suggestions, not truth
Something to evaluate against the code, not to trust blindly.
Potentially harmful if they are too generic
A vague “this function validates user input” comment is not helpful in a file full of edge cases.
Disposable
If a comment did not add concrete value, I deleted it without guilt, even if it looked nice.

Search engines are moving toward rewarding people first content that reflects real experience, rather than text that merely looks polished, and breakdowns from folks like Backlinko make the same point from the SEO side.

Inside a team, your “search engine” is future you and your colleagues. They deserve more than generic filler.

Practical guidelines that worked better for me

Here are a few simple rules I would recommend to teams that are already using AI for docs.

1. Use AI as a first draft, not the final word

Let the model draft a function comment or a short doc.
Then edit it aggressively:
- Delete obvious fluff.
- Add the parts that are truly non obvious: invariants, gotchas, business rules, weird constraints.
Only keep what you are willing to stand behind personally.

If nobody is willing to own a piece of documentation, it is better not to have it.

2. Prefer small, local comments over giant essays

Instead of:

“This module is responsible for managing all sandbox orchestration, including creation, deletion, routing, error handling, and logging…”

Prefer small comments next to tricky bits:

Why a timeout is set to a weird value
Why a fallback exists for a specific provider
What assumptions you are making about the external API

AI is good at filling in missing description around a small area. It is bad at accurately summarizing the entire system in a few paragraphs without help.

3. Make docs describe current behavior, not ideal behavior

A common failure pattern I saw:

Developers asked AI to document how the system “should” work.
The doc described an idealized flow.
The implementation quietly diverged.

When I write docs now, I ask:

“What does the system do today, including the ugly parts?”
“Which parts are known to be wrong or temporary?”

Then, if I want a vision document, I label it explicitly as such.

4. Delete AI generated docs that nobody understands

In that startup, there were files of architecture docs that nobody on the team had fully read.

If you cannot find a single person who:

understands a document
believes it is accurate
and is willing to maintain it

then it is probably safer to remove it than to keep it as a fake map.

A simple test for your own project

Take a random module in your codebase that has AI generated comments or docs.

Ask yourself:

If a new engineer joined tomorrow, would this text actually help them understand the code faster?
Does it describe non obvious knowledge, or is it just restating the code in English?
If the code changed last week, is this still true?

If the honest answer is “no” to most of those, you have two options:

Fix the docs and comments so they become real assets.
Or delete them and rely on the code until someone has time to document it properly.

The goal is not poetic documentation. It is trustworthy documentation.