This is one of the biggest trust issues with AI support. A confident wrong answer can be much worse than no answer at all because customers may act on it before realizing it’s incorrect. Good AI support should know when to say “I’m not sure,” ask for clarification, or hand the conversation to a human. Accuracy and graceful escalation matter just as much as automation.

I think they look for any sought of relation in the source documents to what a customer is asking and that leads to wrong or opaque answers

Yeah, this can be really frustrating, i’ve had situations where an AI gives an answer that sounds completely confident, so you only realize later that something is off, i’d much rather it admit when it’s unsure and give me the option to talk to a human than confidently send me in the wrong direction.

I would prefer an AI to admit uncertainty and escalate rather than confidently inventing an answer. In many cases, the problem isn’t that the model is bad, it’s that the system hasn’t been given clear instructions about what to do when the answer cannot be verified.

A confident wrong answer is way worse than just saying “I don’t know.” Trust takes a hit when customers act on bad info, so clear uncertainty and smooth human escalation matter just as much as getting it right.

LLMs are literally trained to predict the next most plausible word , which means they are mathematically optimized to sound confident even when making things up

That’s one of the most frustrating AI problems because the answer can sound completely reasonable. Customers don’t necessarily have a way to know it’s wrong until they’ve acted on it. I’d much rather have the system communicate uncertainty and escalate than give a confident answer without enough evidence.

I’ve definitely been there. I once had a billing bot tell me that I was eligible for a discount that didn’t exist in the first place.