It builds sentences rather than finding answers
A conversational AI learns from a huge body of text which words tend to follow which. Given a question, it uses those patterns to choose the most plausible next piece, attaches it, chooses the next, and repeats until the answer is complete. It is not consulting an organised list of facts; it is composing sentences from a sense of what answers to this kind of question usually look like. That is why well-covered topics come out accurate and rare topics come out strange.
Why plausible wrong answers happen
Nothing in this process says "if you do not know, say so" by default. It can produce an answer-shaped sentence for a question it cannot answer, in exactly the same tone as a correct one. Nonexistent book titles, papers that were never written and convincing fake statistics are born this way. In 2023 a lawyer in the United States was sanctioned after filing court papers that cited cases an AI had made up. The more assertive the tone, the more reason to doubt it once more.
It may not know anything after its training date
A model learns from text collected up to a certain point, so not knowing later events, changed rules or new prices is normal. The trouble is that instead of saying so, it sometimes presents old information as current fact. Services with a search feature can pull in recent material, but then the quality of the answer depends on which sources it picked. When dates matter, ask it to state the point in time its answer refers to, and check the official site yourself.
Why counting and arithmetic trip it up
AI processes text not letter by letter but in chunks of several characters. So it gets questions like "how many characters is this sentence" or "how many times does this letter appear" wrong surprisingly often. Long multiplication and multi-step calculations drift as well. For values that must be exact, such as an application essay's character limit, measure with a character counter, and recheck calculations in a calculator or spreadsheet. Asking it to show the working step by step also makes the error easier to spot.
The same question can get different answers
A little randomness goes into each answer, so asking again can change the wording or even the conclusion. It also tends to agree with the person asking: ask "this is right, isn't it?" and it leans towards yes; ask "this is wrong, isn't it?" and it leans towards no. For decisions that matter, ask neutrally instead of leading, and request the strongest case for the opposite view as well. If the answer keeps changing, that itself signals a question nobody should be confident about.
It is genuinely good at some things
Limits do not make it useless. On tasks where you can read and judge the output immediately, it saves a great deal of time. The key is whether you can tell if the result is right.
- Getting a first draft past the blank page
- Pulling key points and an outline from a long text
- Softening stiff sentences or shortening long ones
- Explaining a hard concept with plain words and examples
- Sorting scattered notes into tidy sections
Common mistakes
The most common mistake is copying a plausible answer without checking it, and sentences containing sources or numbers are the riskiest. The second is asking the AI whether its answer is correct and relaxing when it says yes; an answer produced the same way verifies nothing. The third is seeing one wrong answer and deciding the tool is worthless. The fix is to separate the tasks it suits from the ones it does not. Even summaries can pick up details that are not in the original, so for important documents put the summary next to the source and check that the key statements are really there.
Questions to ask before you use an answer
Asking yourself these four questions whenever you get an answer avoids most accidents. If any of them applies, keep the answer as a reference only and confirm it somewhere else before using it.
- If this is wrong, who loses, and how much?
- Can I tell whether it is right or wrong?
- Does it contain dates, numbers, names or sources?
- Could this have changed recently?
🌍 Search the web for this
Each button runs this keyword on that search engine