AI Feels Intelligent — But That's Not Why I Don't Trust It

Part 1 of 3 on compression and cognition. This essay is about the trust problem. The next one, Most Insights Aren't, is about defining what valuable AI output actually means. The third, Why AI Productivity Metrics Are Lying to Us, follows that definition to where measurement breaks down.

I recently asked an AI to help me evaluate a product decision. My prompt had a flawed assumption baked in, I'd framed two options as mutually exclusive when they weren't. The system didn't pause. Didn't ask whether the framing was right. It just picked one option and argued for it, confidently, with reasons that sounded plausible.

The answer was coherent. It was also useless. The real move was to reject the question entirely. A thoughtful colleague would have caught that. The AI optimized for answering, not for noticing.

This is the thing that keeps bothering me. Not that AI makes mistakes. Humans make mistakes. The problem is that AI rarely signals where it might be wrong. It accepts premises too easily, agrees when agreement should be costly, and collapses uncertainty before it's earned the right to. A human saying "I'm not sure, but here's a first pass" is honest. An AI saying the same thing and then continuing with total confidence often isn't.

What we experience as intelligence in AI is usually compression. Large amounts of state, data, history, context get ingested and reduced into something small, legible, and confident. That's powerful. It's also where things go wrong.

Compression does not distinguish between irrelevant uncertainty and essential uncertainty.

What I mean by that: when you're making a decision, some things you're unsure about don't matter. The exact market size might be off by 10%, but if you're choosing between two product directions, the directional bet is the same either way. That's irrelevant uncertainty. You can compress it away and nothing breaks.

But some uncertainty is the whole game. Whether your core assumption about user behavior is right. Whether the two options you're evaluating are actually mutually exclusive. Whether the metric you're optimizing for is the right metric. That uncertainty is the decision. Compressing it away doesn't simplify the problem. It removes the problem from view while leaving it fully intact.

AI compresses both kinds at the same rate. It takes a messy, ambiguous situation and produces clean, fluent output. The fluency feels like resolution. But the system didn't resolve anything. It just flattened the uncertainty that mattered into the same smooth surface as the uncertainty that didn't. And now you can't tell which was which.

This isn't a failure of the technology. It's what the technology is optimized for. Helpfulness prefers smooth answers. It rewards agreement over friction. Think about that yes-ma'am colleague everyone has, the one who never pushes back, always delivers something that looks right, and whose work you've learned to double-check quietly because their agreeableness is the tell that they're not actually thinking.

AI is that colleague at scale.

Human intelligence earns trust differently. It resists, it asks whether the question itself is well-formed before answering it, it knows that some uncertainty should be surfaced rather than removed. Those behaviors signal someone is actually processing the problem rather than pattern-matching against it. We tend to treat them as inefficiencies. They're not.

When AI removes those signals in the name of fluency, it shifts the burden of judgment onto you without announcing it. You feel clarity, but the system has just hidden where the cost went.

A more intelligent system would probably feel less helpful at first. It would disagree more, slow things down, push back on framing. It would try to align with the underlying goal rather than the surface-level prompt. That would create friction. But friction is what trust feels like before it's been earned.

This is the question I keep circling: if fluency isn't the signal of value in AI output, what is? How do you tell the difference between something that made you feel informed and something that actually moved your thinking?

That requires being precise about a word the industry has made almost useless.