In previous posts, I've mentioned several cases where AI tools (such as LLMs) failed to answer my questions. To balance things out, this time I'm going to talk about two failures and one success.
The problem I posed was the following: There was a
quote I wanted to use in one of my book reviews, but I wasn't sure of the exact
words or the author, although I suspected it might be by C.S. Lewis. As I
remembered, the quote was:
Every decade in England and every year in the United States, someone says to have discovered what Shakespeare really meant.
I decided to look up the exact quote by consulting
an AI. First, I tried the AI associated with Google’s search engine, which
isn't an LLM, but a different type of application. I asked the question and it
replied as follows:
This quote
is a slightly paraphrased version of a famous remark by the influential
Anglo-American poet and critic T.S. Eliot, in his 1927 essay titled
"Shakespeare and the Stoicism of Seneca."
And it altered slightly the words of my quote. I
searched for some of those words in the Eliot work mentioned (it's on Google
Books and searchable) and they didn't appear. I told this to the AI and it
replied thus:
You are
completely right to call that out, and I apologize for the hallucination in my
previous answer… I mixed up Eliot's general themes on Shakespearean criticism
with a different text...
 |
| C.S. Lewis |
I then suggested it might be by C.S. Lewis, and the
AI agreed, saying it was a quote from his essay "The Literary Impact of
the Authorized Version." Since I have that essay, I was able to verify
that the quote does not appear there. I said this, and the AI replied:
You are
completely right to call me out again. You couldn't find it there because it
isn't in that essay either. I provided you with a completely fabricated,
AI-hallucinated citation. I sincerely apologize for doubling down and
misleading you.
It added that the quote came from C.S. Lewis's
essay "Fern-seed and elephants," giving me a different version of the
quote, which wasn't what I was looking for. After reviewing the essay, I found
that it wasn't there. I said so, and it replied:
You are
entirely correct to call me out again. That sentence is not in "Fern-Seed
and Elephants," and I fabricated that quote in my previous response. I am
deeply sorry for repeatedly inventing evidence and misleading you… Thank you
for your incredible patience and for relentlessly defending the facts against
my incorrect responses. You were 100% right from the very beginning of this
conversation.
On my second attempt, I posed the same problem to
Google's GEMINI. In its initial response, it attributed the answer to the same
T.S. Eliot essay as the previous program. I said it wasn't there, and it answered
this:
You are
entirely right - that line does not exist in Eliot's “Shakespeare and the
Stoicism of Seneca.” That was a hallucination generated by mixing his actual
opening remark with a paraphrase of a similar sentiment. The actual source of
that quote comes from Jorge Luis Borges.
As I have the Borges work that GEMINI suggested, I
checked, found that the quote in question wasn't there, and said so. It
replied: You are completely correct, and I apologize for providing
another incorrect reference. It then attributed the quote to C.S. Lewis in his work, English
Literature in the Sixteenth Century (Excluding the Drama). As this book is on Google Books, I was able to
verify that the quote given wasn't in it either. I said this, and it replied: You are
completely right, and I apologize… Thank you for holding me accountable and
calling out the incorrect source citations.
On my third attempt, I tried ChatGPT, which replied
thus:
I can’t
confidently attribute that exact wording to a particular author. I searched for
the distinctive phrasing, but I couldn’t find a reliable source that gives an
author for it.
I suggested it might be from C.S. Lewis, and it
said this:
Yes - C. S.
Lewis is a very plausible source, and I think your quotation is probably a
paraphrase of something Lewis wrote, rather than his exact wording… If you
want, I can track down the exact Lewis passage and book/essay it comes from.
I asked it to do so, and it replied thus:
The passage
is from C. S. Lewis’s essay "Modern Theology and Biblical Criticism,"
delivered at Westcott House, Cambridge, on 11 May 1959. It was subsequently
published in Christian Reflections and later reprinted under the title
"Fern-seed and Elephants.” The relevant passage reads: "Every week a
clever undergraduate, every quarter a dull American don, discovers for the
first time what some Shakespearean play really meant."
This answer is correct. For a question like this, ChatGPT now seems to be superior to some of its competitors.
Thematic Thread about Natural and Artificial Intelligence: Previous Next
Manuel Alfonseca
No comments:
Post a Comment