This gets to the core of the issue. LLMs are a model of the statiscal relationship between words in texts, in a very large number of dimensions. The intelligence they appear to exhibit is that which existed in their source material in the first place. They don’t have a model of the world itself. If you consider how midjourney can produce photorealstic images of people yet very often it will get hands wrong. How is that? It’s because when you train on images, you get a statistical representation of what hands look like without the world model that let’s you know that hands only have 5 fingers and how they’re arranged. AIs like this are very clever copiers. They are not intelligent