OpenAI explains why language models ‘hallucinate’; evaluation incentives reward guessing over uncertainty

In the design of large language models (LLMS), Openai has identified a fundamental defect that causes confident and wrong information known as “hallucinations”. This detailed discovery in a recent research article challenges existing assumptions about AI reliability and proposes a paradigm shift in model assessment.

The hallucinations in AI refer to the examples produced by expressions that are actually wrong but presented with high confidence. For example, when a leading researcher was questioned by XYZ about the title of the doctoral dissertation, the model provided three different titles, none of which was not right. Similarly, it offered three wrong birth date for Kalai.

The main problem identified by Openai researchers lies in the educational and evaluation processes of LLMs. Traditional methods focus on correct or wrong binary ratings, without taking into account the confidence in the model’s responses. This approach accidentally rewards models to make educated predictions, even if it is uncertain, because the correct prediction gives a positive result, whereas accepting uncertainty results in zero points. As a result, models are trained to give an answer to accept lack of information. The research article refers to:
According to the futurism website, hallucinations “most of them continue because of the rating of the evaluation, the language models will be good test buyers and they estimate when it improves uncertain test performance,” he says.

In order to address this problem, Openai proposes transition to evaluation methods that value uncertainty and punish confident mistakes. By applying the trust thresholds, the models will be encouraged to avoid responding when they are not sure and thus reduce the likelihood of hallucination. This approach aims to increase the reliability of AI systems, especially in critical applications where factual accuracy is very important.

“Most scorebord models prioritize and list it based on accuracy, but errors are worse than abstainrs,” Openai wrote in a accompanying blog post. The recommended changes have wider effects for AI development, including the potential effects on user participation. Models that often accept uncertainty can be perceived as less competent, possibly affect user trust and adoption. Therefore, it continues to be a critical issue to balance accuracy with the user experience.