AI Is Artificial And Un-Intelligent

 Mind the hype.

The research tasked the Large Language Model (LLM) Mixtral with grading written responses to middle school homework. Rather than feeding the LLM a human-created rubric, as is usually done in these studies, the UG team tasked Mixtral with creating its own grading system. The results were abysmal.

The LLM graded the work correctly 33.5% of the time. "Even when supplied with a human rubric, the model had an accuracy rate of just over 50 percent."

What’s going on? Follow the money.

While LLMs can adapt quickly to scoring tasks, they often resort to shortcuts, bypassing deeper logical reasoning expected in human grading," wrote the researchers.

"Students could mention a temperature increase, and the large language model interprets that all students understand the particles are moving faster when temperatures rise," said Xiaoming Zhai, one of the UG researchers. "But based upon the student writing, as a human, we’re not able to infer whether the students know whether the particles will move faster or not."

Though the UG researchers wrote that "incorporating high-quality analytical rubrics designed to reflect human grading logic can mitigate [the] gap and enhance LLMs’ scoring accuracy," a boost from 33.5 to 50 percent accuracy is laughable. Remember, this is the technology that's supposed to bring about a "new epoch" — a technology we've poured more seed money into than any in human history.

All that seed money has to be buying something valuable and with a very good ROI, right? But as the study points out, AI doesn’t “understand.” Because it’s incapable of understanding. That’s the thing computers still can’t do. And, as Shrub famously asked: “Is our machines learning?” The answer is: “No, apparently not.”

In fact, there's mounting evidence that AI's comprehension abilities are getting worse as time goes on and original data becomes scarce. Recent reporting by the New York Times found that the latest generation of AI models hallucinate as much as 79 percent of the time — way up from past numbers.

Which means, at the very least, AI needs to stay in the laboratory. It’s certainly not ready for prime time