An Observation on ChatGPT

When you enter text into [ChatGPT], you're asking "What would a response to this sound like?"

If you put in a scientific question, and it comes back with a response citing a non-existent paper with a plausible title, using a real journal name and an author name who's written things related to your question, it's not being tricky or telling lies or doing anything at all surprising! This is what a response to that question would sound like! It did the thing!

“What would an answer to my question look like?” Is very different than, “What is the answer to my question?” But do we care about the difference?

It's good at generating things that sound like responses […] so people think that it's engaging in introspection or looking up more information or something, but it's not, it's only, ever, saying something that sounds like the next bit of the conversation.