34

After seeing StackOverflow's banning of ChatGPT, I explored it out of curiosity. It's marvellous as it can write code by itself!

Later to check if it knows chess as well like Google-Deepmind's AlphaZero AI, I asked below questions:

Me: Does openai know how to play chess?
OpenAI: It is not clear whether OpenAI knows how to play chess. As an artificial intelligence research organization, OpenAI has developed a number of advanced machine learning algorithms and techniques that allow AI systems to learn and adapt to new tasks and situations. ...

Me: In a chess if white has a king & bishop and if black has only a king, then who can win with a perfect play?
OpenAI: In a chess game where white has a king and a bishop and black has only a king, it is possible for either player to win with perfect play. This type of endgame is known as a king and bishop versus king endgame, and it is considered one of the most challenging in chess. ...

Now this answer is entirely wrong. Because as soon as only a bishop (or a knight) remains apart from the kings, it's instantly declared as a draw!

Question: Shouldn't the AI reject my question on a subject which is not in its known criteria?
It does so for many other subjects.

Note: Should we replace bishop = rook then ChatGPT answers exactly the same answer with replacing those pieces. However that happens to be true.

Community
  • 1
iammilind
  • 443
  • 1
  • 2
  • 9
  • 34
    I think a good way to think about ChatGPT is that it doesn't know anything about anything, and its goal is to bluff its way through a conversation by guessing what you want to hear. Sometimes, the bluff is easier to call. – Stef Dec 13 '22 at 10:41
  • 3
    "that happens to be true" (in the rook case) - really? The GPT answer appears to be saying that **either** player can win; I'm not a chess expert but I'm struggling to see how black can force a win with just a king? – psmears Dec 13 '22 at 15:57
  • @psmears, sorry.. A small correction here is that, if i replace "bishop" with "rook" then the chatgpt correctly says that white *can* win. No mention of Black's win. – iammilind Dec 13 '22 at 18:11
  • 1
    Related: The [Chinese room](https://en.wikipedia.org/wiki/Chinese_room) argument – Ouroborus Dec 13 '22 at 18:45
  • 2
    "*that happens to be true*" you might want to experiment by retrying the same prompt multiple times and observe that the response may change or even contradict previous response. – Andrew T. Dec 14 '22 at 04:37

6 Answers6

72

ChatGPT is a large language model. That means it's very good at stringing together words in ways that humans tend to use them. It's able to construct sentences that are grammatically correct and sound natural, for the most part, because it's been trained on language.

Because it's good at stringing together words, it's able to take your prompt and generate words in a grammatically correct way that's similar to what it's seen before. But that's all that it's doing: generating words and making sure it sounds natural. It doesn't have any built-in fact checking capabilities, and the manual limitations that OpenAI placed can be fairly easily worked around. Someone in the OpenAI Discord server a few days ago shared a screenshot of the question "What mammal lays the largest eggs?" ChatGPT confidently declared that the elephant lays the largest eggs of any mammal.

While much of the information that ChatGPT was trained on is accurate, always keep in mind that it's just stringing together words with no way to check if what it's saying is accurate. Its sources may have been accurate, but just writing in the style of your sources doesn't mean that the results will themselves be true.

Mithical
  • 2,885
  • 5
  • 27
  • 39
  • 11
    Exactly. It finds the words that are most likely to follow the prompt, and has no understanding of anything else. – Oliver Mason Dec 12 '22 at 12:47
  • 3
    I know that on previous GPT models, these kinds of questions with no right answer could lead to some amusing answers. More like this: [Baltimore Orioles effect](https://www.aiweirdness.com/baltimore-orioles-effect/) and [Botsplaining](https://www.aiweirdness.com/botsplaining/) and [Galactica: the AI knowledge base that makes stuff up](https://www.aiweirdness.com/galactica/) and when humans do it [Comparative Illusion](https://en.wikipedia.org/wiki/Comparative_illusion) – user253751 Dec 12 '22 at 16:32
  • 26
    It's fairly easy to confuse ChatGPT once you understand this principle. For example, if you ask it to describe the Avatar movies it'll confuse James Cameron's Avatar with Avatar: The Last Airbender. Or if you ask it to describe how the planets follow probabilistic orbitals it'll happily confuse Quantum Mechanics and Newtonian gravity for you. All it takes is two concepts that use the same set of words, because the language model has no deeper understanding. – David Dec 12 '22 at 22:40
  • I'd like to point out that being a "large language model" does not imply "it's very good at stringing together words in ways that humans tend to use them", so I'd rephrase your first sentence, as it implies that every large language model is good at doing that, which may not be the case. What if it's not trained or does not have the correct inductive bias or was trained with "bad" or little data? These are all factors that affect the quality of the LM. Given the attention that this answer and post got, you can be more precise, although some people understand that one does not imply the other – nbro Dec 14 '22 at 09:32
  • 1
    Moreover, you claim "making sure it sounds natural". Right now, I am not familiar with the details of the ChatGPT, but I will soon be, but does it really **ensure** it sounds natural? If it's a statistical model with no constraints on "naturalness", then I doubt that claim is true. – nbro Dec 14 '22 at 12:36
  • 3
    This is why it's less impressive than it looks. Still impressive, sure, but more showy than anything else. – A. R. Dec 14 '22 at 13:50
  • 1
    @David : thanks for the hint about concepts using the same words for different meanings. I asked some questions which mix the concept of entropy from thermodynamics with the concept of entropy from information theory, and the results were hilarious. But the wording was so natural and so confident, that it could easily fool someone who has only a very superficial knowledge about the topic. I fear ChatGPT will be used by pseudoscientists a lot, because it can produce very scientific-sounding articles which seem to have a deeper meaning even when they don't. – vsz Dec 18 '22 at 12:37
  • Seems [that know it generates the correct answer](https://i.stack.imgur.com/M9Qge.png) for that specific question. It's learning. – Braiam Mar 15 '23 at 17:15
  • 1
    If all it does is *stringing words together*, how is it able to write code by itself? – Shub Apr 29 '23 at 16:45
  • @Shub You can think of it as LLM generating words letter by letter (what are words anyway but a sequence of letters?) Internally, LLM uses subwords (which may be individual letters) stringed together to form a text output. A code is a text, so it can generate it. – justhalf Jun 06 '23 at 05:44
20

ChatGPT does not actually know anything. But more importantly even, it does not know this fact! Hence, it does not know that it does not know.

It is only good at combining text.

DrCommando
  • 309
  • 2
  • 5
    Note that this is just a philosophical point of view. Who decides that people also know anything? What do you mean by "knowing"? Maybe we also don't know anything. We have memories, which we recall with some algorithm, based on some context. You need to define "to know". Because, if "to know" means "have knowledge of something" and "knowledge" means "info or facts", then ChatGPT certainly knows something, because it can give you some facts. Either it's aware of what it says, it's another question, which also depends on what you mean by "awareness". – nbro Dec 14 '22 at 12:38
  • 1
    So, what one can say for sure is that ChatGPT may give you some facts and correct info, but these facts are just the "most likely" outputs given the data and the inductive bias, i.e. it's a statistical model, and thus it's limited by how the model is defined, the data and the training algorithm. So, it can also give you completely wrong info. The way it produces some output $x$ is, we can say, not equal to how we produce the same output and, most likely, not even close to how we produce it. – nbro Dec 14 '22 at 12:48
  • 4
    There is [a great article](https://journal.sjdm.org/15/15923a/jdm15923a.html) describing this phenomena (which it calls "pseudo-profound bullshit") when done by humans. Cult leaders and snake-oil salesmen are great at stringing together buzzwords into grammatically correct but meaningless sentences, which the unaware listener might think to hold some deep knowledge. – vsz Dec 15 '22 at 19:09
  • 3
    @nbro the aspect of awareness is relevant, I agree. One way to naively put it, is when there is knowledge/awareness regarding the fact, that a specific knowledge is not available. Humans, in the best case, are aware of their individual lack of knowledge. We have some sort of table of content about things we know. We also know that we may have heard something, but never thought it through ourselves. ChatGPT and other AI models store and recombine/associate. – DrCommando Dec 15 '22 at 20:58
  • So interesting comments. Knowledge about facts, and knowledge about how to check new facts against our knowledge base. How to expand our table of contents through valid reasoning. – Jaume Oliver Lafont Dec 16 '22 at 08:59
15

ChatGPT and other GPT-based machine learning models don't actually know anything in the sense you're thinking of. ChatGPT is a distant descendant of Markov chain text generators such as Dissociated Press, and works by predicting what is most likely to follow a given sequence of text.

So, when you ask it "Does openai know how to play chess?", what you're actually getting is the answer to the question "What is most likely to follow the phrase 'Does openai know how to play chess?'?"

Mark
  • 420
  • 2
  • 9
  • Then how does it write code? – Shub Apr 29 '23 at 16:47
  • @Shub, the same way it answers questions. Code isn't fundamentally any different from written language, and ChatGPT tends to make the same sorts of mistakes, producing code that is syntactically correct but semantically nonsensical. – Mark Apr 30 '23 at 04:35
9

“It” does nothing. Don’t think just because every conman out there calls our really shockingly primitive neural nets “AI”, and wants to convince you that it’s actually an autonomous intelligence, that it’s not just a glorified function (a list of commands to blindly execute, not a person), to apply a set of biases onto a given input pattern, that have been programmed into it in a way that the programmer doesn’t “have to” know what he’s doing or even what precisely he wants. :)

It is just biasing for the patterns in its training data. And giving you whatever that results in for your your input. In this case, if I am correct, applying its output to its input too again, and again, with diminishing sanity.

So the answer is that your input will just be treated like it is a spectrum of those patterns, no matter what.
In other words: If all it knows is a hammer, everything will look like a nail. :)

So it is quite mundane, and nothing magical at all.

Everything beyond that, attributed to such systems, is deliberate dupery, to get people to “invest” money.

(Don’t get me wrong: This technology is useful when writing an actual algorithm really is beyond a human’s capabilities. (E.g. by definition a brain cannot imagine a model of itself in its entirety. Or you cannot write down the entire set of experiences of a lifeform, find all the patterns, and turn them into a set of rules manually. Even if the Pauli Exclusion Principle would not exist.) But nowadays it is abused by people who proudly can’t even define what they want, to just throw examples at, and expect to get an universal function out, so they can call themselves geniuses and get lots of money.)

Evi1M4chine
  • 207
  • 3
  • 13
    I don't really disagree with any of those statements. However, what else do humans do? Our brains are also just glorified functions that map sensory inputs (plus some nondeterministic crud) to commands to the muscles. If a human tries to answer a question about a subject she doesn't know, she will also make an attempt based on patterns she has seen before that seem to resemble those in this new situation. The main difference with ChatGPT is that it trumpets out wildly extrapolated guesses with the same matter-of-factly confidence, as answers where it really does have good training basis. – leftaroundabout Dec 12 '22 at 19:03
  • 5
    @leftaroundabout Your view is an extremely common fallacy about how our brains work. [Our brains are not information processors](https://aeon.co/essays/your-brain-does-not-process-information-and-it-is-not-a-computer). We actually don't really know how our brains work. – Nelson Dec 13 '22 at 05:27
  • 4
    @Nelson I also agree with that we don't understand how our brains work, but this is besides the point. Neither do we understand how deep learning works. Certainly they don't work the same, but at this point it's clear that systems like ChatGPT do behave a lot like, specifically, _humans tasked with writing pseudo-profound essays_. It's tempting to rest on the pillow of smugness for all the things we can still do better than ChatGPT, and that these are the “truely human” skills, but unfortunately it's already [No-True-Scotsman](https://en.wikipedia.org/wiki/No_true_Scotsman)–“true”. – leftaroundabout Dec 13 '22 at 08:19
  • 5
    @leftaroundabout If as a human being I was asked this chess question I could separate my thought process into two parts a) check my factual knowledge of chess for an answer and b) formulate this answer into a nice text that other humans can read and understand. ChatGPT appears to do very well on part b) but doesn't do anything related to part a) If my human brain comes up empty on a) I could still apply b) and produce an answer similar to ChatGPT but in general I don't. – quarague Dec 13 '22 at 09:05
  • Humans with brain damage act like ChatGPT. It is called "confabulation" but specific examples seem to be rare online. – user253751 Dec 14 '22 at 08:46
  • 4
    @Nelson That author seems to be using terminology in a pretty bizarre way. He asserts that memorizing something doesn't store it in the brain, recalling something doesn't retrieve it from the brain, and reacting to stimuli according to a complex rule doesn't involve computing. He gives no justification whatsoever for the assertion that computers store things but brains don't. I'm not necessarily claiming that the article is totally wrong, but it doesn't make any sense to me. – Tanner Swett Dec 14 '22 at 19:47
  • @Tanner-reinstateLGBTpeople They used the recognition of money as an example. We can recognize money, but our "memories" of it are so laughingly poor there's no AI in the world that can use what we "remember" to positively ID the bills we are referring to, but WE can. Our memories don't see to make "copies" of anything at all, hence the absence of truly "storing" anything. Just flip it around. Imagine you saved a picture of a bank note on a computer, and when you retrieved it, it looked like what a person drew out of memory. That's a broken computer, but it's not a broken person. – Nelson Dec 15 '22 at 00:49
  • 7
    @Nelson Well, if a computer outputs a perfect word-for-word copy of the poem "The Raven" without using any external sources, we conclude that the computer must have contained a representation of the poem. When a human brain performs the same feat, why would we not come to the same conclusion? – Tanner Swett Dec 15 '22 at 02:20
  • 4
    @Nelson even accepting the article's conclusions, I don't think it really applies to/disputes what leftroundabout is saying. Signals enter the brain, trigger complex nonlinear processes, signals leave brain. This is indisputable physical fact. The "correct" way to understand what those processes do, how they do it, what they "mean", are structured, etc are of course up for debate, but to claim the system as a whole cannot be described in this manner would need something non-physics-based to be going on. – mbrig Dec 15 '22 at 03:53
  • 1
    @leftaroundabout: The difference is that a AI is an **individual**. It makes its *own* experiences, *learns* independently, evolves its mental model *freely*, so it can actually make its own choices. And isn’t de-facto programmed into a certain model of patterns of the programmer and then frozen like that. … Yeah, most humans nowadays clearly aren’t individuals anymore either, and seem to turn into more like cells in a bigger body (ideology, church, Apple, etc), and they can live however they “want”, but there, the bigger body still is the individual. … Don’t muddle the definitions. – Evi1M4chine Dec 15 '22 at 14:36
  • mbrig: Correct or not is irrelevant here, as he’s completely missing the point. Sad, how “lost” the concept of independent thought is on some people nowadays. Like they think it isn’t even a thing… Maybe the space-faring civilisations aren’t actually civilisations, but swarm lifeforms that see humanity and themselves a single lifeforms. Like the Borg. ;) – Evi1M4chine Dec 15 '22 at 14:39
  • @Tanner: Memorizing and recalling are actually not separable processes in in a real biological neural net. It is for that reason, that our childhood memories are mostly false. As each time we remembered, we interpreted it in the context of that moment, and then memorized that interpretation on top of the old memory (since it follows mostly the same neural pathways). Ditto for new memories, which are always also a recalling of similar memories, as that is how pattern detection can do its work. Being able to separate those steps is a major advancement in computers over our brains. – Evi1M4chine Dec 15 '22 at 14:43
  • 1
    @Evi1M4chine well, clearly I _am_ missing the point you're making, because I don't know what it is. For example, I genuinely can't tell if you had a typo in “The difference is that a AI is an individual [sic]”. Did you mean it _is_ an individual, or that it isn't? The arguments with which you back it up aren't very convincing either way. – leftaroundabout Dec 15 '22 at 15:25
  • 1
    If your claim is that we should make a _philosophical_ distinction between human thought and AI thought, then I couldn't agree more. In fact I daresay we must soon find some good ethics platform to define and keep up the distinction. But linking it to any fundamental differences in the mechanics of the thought processes of humans and AIs is bound to get harder and harder, and soon impossible. – leftaroundabout Dec 15 '22 at 15:26
  • @leftaroundabout: No, I am absolutely not making a distinction between human and AI *thought*. But I still agree with you: I’m making a distinction between human/AI and the fake thing that is currently called AI by the media and certain scammers. ;) // BTW, that always trying to keep a “distinction” between ”animals” or black people or women or whatever, and “us chosen ones that literally the universe revolves around and that are the only ones that can think”, is as old as it is bad. (And I’m not a SJW.) My grandma still believed we’re not animals and animals can’t feel or think. ^^ – Evi1M4chine Dec 15 '22 at 15:34
  • @leftaroundabout: Ah, sorry for the confusing wording. When I sad “AI” without any qualifiers, I mean actual AI. Which does not exist yet, and would be an actual lifeform and individual. And what I criticize in the answer, is this fake “AI”, which is what exists, what the media and scammers call “AI”, and what is not a lifeform, and is not an individual. … Whether one thinks they are thinking, is a matter of definition, as was rightly pointed out in these comments, but was unrelated to my statements. – Evi1M4chine Dec 15 '22 at 15:37
  • 1
    Ok, well then I disagree with... everything, pretty much. Whether ChatGPT is an “actual AI” is what's irrelevant; we're beyond the point where it's useful to draw lines between “non-actual” and “actual” AI because they will just keep getting crossed every year by the next version. Meanwhile, analogies like _humans—AI_ ⇐⇒ _men—women_ are relevant but wrong. Men did not create women, nor are women based on a different physical architecture that allows them to evolve at orders-of-magnitude faster pace than men. – leftaroundabout Dec 15 '22 at 16:21
  • @leftaroundabout: No, words have meanings. AI always meant what I described. And it’s only recently, that literal conmen and clueless PHBs and media spreading their stuff ruined the term. And now kids entering the field, who confidently believe it always meant that new nonsensical definition. It is not acceptable to then go “Well, now it means something different.” Because we do not have to obey by the definitions of clueless people and criminals. … The analogy to the term “hacker” having been ruined, is almost perfect here. – Evi1M4chine Dec 18 '22 at 11:46
  • @leftaroundabout: My argument about us always thinking we are the superior chosen ones had nothing to do with men/women or any of the other examples specifically, so you can’t mistake it as a “men”/“women” argument. That’s just the current madness of society, and I categorically do not operate in this space. :) Please read that part again without that trigger. (Hint: I’m with you.) – Evi1M4chine Dec 18 '22 at 11:50
  • @leftaroundabout: The new definition is useless. It is deliberately muddled and there already is a term for that (“ANN“, artificial neural nets, or “NN” for non-artifical ones or generally neural nets). The **whole point** of the term “AI” is, that it is more than that. And the new definition removes that exact distinction. So why use a new word for it anyway? … Simple? To con people into believing you got an actual Ai when all your have an a true train wreck of pathetic matrix multiplication hacks that everybody should be ashamed of for being *that* bad. – Evi1M4chine Dec 18 '22 at 11:55
  • 1
    @Evi1M4chine well, I suppose we can agree that the term “AI” is rather meaningless. But how, then, is there any point in debating what is or isn't an AI? Your notion of “AI” seems to be rather what is now conventionally called [AGI](https://en.wikipedia.org/wiki/Artificial_general_intelligence). — At any rate, what really doesn't make sense is to tie the distinction to patheticness of matrix multiplications. You may think that matrices are a horrible implementation, and I would agree, but it is at this point beyond doubt that they _are_ sufficient to emulate a brain, given enough energy. – leftaroundabout Dec 18 '22 at 12:24
2

The original question asked about "an AI" generally, yet most of the responses here focus on OpenAI's ChatGPT specifically. Seems like the answer would depend on the specific type of AI being used, not limited to just large language model-based chatbots, but considering other types of knowledge representation systems more generally.

Not being an AI expert, I can't speak to this, but here's some good background reading: https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning

Trutane
  • 121
  • 3
0

I agree with most the former answers here, and apologies I don't have high enough points yet to comment so had to do this as an answer, but I believe due to inaccuracies, lack of fact checking capabilities etc... in the data its trained on up to 2021 currently at the time of writing, fine tuning via it's API has become popular I believe. Although you were mainly asking about ChatGPT itself I felt its a good idea to consider it's API as part of that which is just as easily accessible, I am currently learning how to fine tune it my self for more specific and accurate results.

So in respect of the question: Shouldn't the AI reject my question on a subject which is not in its known criteria? It does so for many other subjects.

It can possibly do this if you train / fine tune it another way on the API, but for the latter part, using the normal ChatGPT and ChatGPT plus user interface part of the site, not the API the AI mostly will not reject your question unless it breaches ethical and moral constructs, which I think is also another important aspect to consider when gaining information from ChatGPT.

jamiecropley
  • 85
  • 5