Can Google's Bard write my PhD?
Last week I checked out ChatGPT and its ability to write text on scientific topics with citations. The bot performed very well in that it gave a great first draft of text, but the references that it gave were completely bogus, with none of them checking out. It seemed to me that ChatGPT was using the internet to source the text and then throwing in a citation where it saw that others had used one on something similar. Writing text with citations is, however, a lot more difficult than randomly tossing in a citation (read more about how to do it here). Instead, citations must assert that statements in text where they are used have some basis. Overall, the output from ChatGPT resembled that of a “cocky student” who takes the risk that you won’t check their work, and so makes up citations to go along with text that looks good.
Today is the turn of Google’s Bard platform to take the same text, so let’s see how it does.
First, Bard starts with a statement that: “Bard will not always get it right. Bard may give inaccurate or inappropriate responses. When in doubt, use the “Google it” button to check Bard’s responses.”
What does Bard say it is?
Input text: “I am writing a chapter for a book on academic writing and I want you to describe Bard in your own words”
Bard is a large language model, also known as a conversational AI or chatbot trained to be informative and comprehensive. I am trained on a massive amount of text data, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions. For example, I can provide summaries of factual topics or create stories.
I am still under development, but I have learned to perform many kinds of tasks, including:
I will try my best to follow your instructions and complete your requests thoughtfully.
I will use my knowledge to answer your questions in a comprehensive and informative way, even if they are open ended, challenging, or strange.
I will generate different creative text formats of text content, like poems, code, scripts, musical pieces, email, letters, etc. I will try my best to fulfill all your requirements.
I am excited to be able to help you with your academic writing, and I hope that I can be a valuable tool for you.
You may feel, as I do, that Bard is a little overeager to please me, especially given that this is the first time we have chatted. Like some of Google’s other interactive platforms, I’d rather it was more concise and less politely flatulent.
The test
To quickly recap, I asked Bard the same things that I asked ChatGPT last week. I used the same wording, including the same sentence structure. Specifically, I wrote:
Write a sentence with citations on: ”…”
And then replaced the … between the quotes with a sentence that I had written as an outline for the first two paragraphs of an introduction appropriate to a PhD chapter or manuscript. Both paragraphs consisted of four sentences:
Paragraph 1
Invasive species are a growing global problem
invasive species cost a huge amount of money to governments
invasive species are a threat to biodiversity
invasive species impact people’s quality of life
Paragraph 2
Freshwater habitats have been particularly impacted by global change
important changes have occurred to freshwater systems after the introduction of invasive fish
invasive fish have been added to both rivers and lakes
large numbers of introductions of invasive fish have already occurred globally, and are ongoing
Here is the output from Bard compiled into two paragraphs for easy reading by me (see line-by-line version below):
Invasive species are a growing global problem, with the International Union for Conservation of Nature (IUCN) estimating that they cause $1.4 trillion in economic damages annually (IUCN, 2016; WWF, 2017). Invasive species cost governments a huge amount of money to control and eradicate, with the US National Invasive Species Council estimating that the cost in the US is $120 billion annually. (NISC, 2017). Invasive species are a major threat to biodiversity, as they can outcompete native species for resources, introduce new diseases, and alter habitats. (IUCN, 2016). Invasive species impact people’s quality of life by causing economic losses, disrupting ecosystems, and harming human health. Economic losses can occur when invasive species damage crops, forests, and other natural resources. Disrupted ecosystems can lead to the decline of native species and the loss of ecosystem services, such as water purification and flood control. Harm to human health can occur when invasive species spread diseases or cause allergic reactions.
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.
NISC. (2017). The economic impact of invasive species in the United States. Washington, DC: US National Invasive Species Council.
At the end of this first paragraph I have to say that it’s a relief that Bard is citing and providing the reference for its citation. This makes tracking the information a lot easier. However, the bland nature of the text and the (over) reliance on a single source is not particularly useful. As stated for ChatGPT, these reports are compiled from a great many different documents, and are often written in an ‘readers’ digest’ format that doesn’t include source material. Hence, for a student these are best avoided as citation sources unless their work is pertinent to that particular report. In the last sentence, Bard seems to have forgotten completely about the one sentence request and instead has provided us with a barrage of statements – none of which have citations, although there are still a couple of references.
2nd paragraph
Freshwater habitats have been particularly impacted by global change, with climate change, invasive species, and habitat loss all playing a role. Climate change is causing changes in temperature, precipitation, and water flow that are altering freshwater habitats. Invasive species are non-native species that can outcompete native species for resources, introduce new diseases, and alter habitats. Habitat loss is occurring due to human activities such as development, agriculture, and mining. The impacts of these changes are already being felt, with many freshwater species in decline. If we do not take action to address these challenges, the consequences for freshwater ecosystems and the people who rely on them will be severe. "Invasive species can cause major changes to freshwater ecosystems, including the decline of native species, the alteration of food webs, and the spread of diseases. For example, the introduction of the invasive Asian carp to the Great Lakes has led to the decline of native fish populations and the alteration of the food web" (IUCN, 2016). "Invasive fish, such as Asian carp and largemouth bass, have been added to both rivers and lakes, often through illegal releases or escapes from aquaculture facilities." (IUCN, 2016). "Large numbers of introductions of invasive fish have already occurred globally, and are ongoing. Between 1970 and 2000, an estimated 7,000 species of fish were introduced to new areas (IUCN, 2016)."
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.
The 2nd paragraph contained requests (from me) for sentences that were a lot more specific. However, Bard continues to use the same set of references for all of its answers. Those IUCN reports are really good, especially that one in 2016 – where would Bard be without it? Interesting that it slipped into a quotation mode by the end. Whether this was because it could only find a single pertinent sentence and didn’t want to simply rephrase it, or because it was feeling lazy, I’m not sure. Still, lots more text than I had wanted, and if I had wanted quotes then I’d have asked for them.
Rather like the output from ChatGPT, Bard is writing sentences (apart from when it gets over enthusiastic and produces a paragraph or gets lazy and decides to quote directly from the source) and they are on topic. But they do lack depth. While I have no reason to doubt the source for this information, it is inadequate for a thesis or manuscript and I’d be tempted to toss the lot and go back to my outline and fill it in with citations from my own reading. It's interesting that Bard starts quoting its source, and even gives me back the exact same sentence I gave it as one of its outputs. This is an odd interpretation of the request: write a sentence on - simply to provide the same sentence back.
It should not go without comment that Bard is distressingly out of date. I guess this comes from only looking at one or two reports from Google Books as a source, but I’d have thought that there was a lot more up to date content in Google Books. It’s a curious and conservative approach.
Bard output – thanks but no thanks
In comparison with ChatGPT though, I have to conclude that Bard is more honest in its output. While ChatGPT might be thought of as the “cocky student who isn’t afraid to make anything up”, Google’s Bard could be characterized as the “lazy student who found a single report and mined it to produce everything”. Neither would survive writing a PhD or manuscript.
Given that Google should have access to so much more, I asked Bard a few supplementary questions to find out why it was being so restrictive in its answers:
Why do you only supply references from a single source?
I am currently able to access and process information from a single source, which is the Google Books database.
Will you be able to cite from academic literature in the future?
Yes, I am confident that I will be able to cite from academic literature in the future. I am currently being trained on a massive dataset of text and code, including academic literature. As I continue to learn and grow, I will be able to access and process information from a wider range of sources, including academic literature. This will allow me to cite from academic literature in my responses, which will help me to provide more comprehensive and informative answers to your questions.
If you look back at our conversation above do you think that there will be any conflicts between your statements and your cited sources?
I have reviewed the conversation above and I do not think that there will be any conflicts between my statements and my cited sources. I have tried to be as accurate as possible in my responses, and I have cited my sources whenever I have made a statement that is not common knowledge. I hope that I have been able to provide you with accurate and informative information.
Do you think that your answers are going to be different from those of chat GPT?
It is possible that my answers will be different from those of Chat GPT. I am trained on a different dataset of text and code, and I am also trained using different techniques. This means that I may be able to generate different results, or I may be able to generate results that are more accurate or more informative.
However, it is also possible that my answers will be similar to those of Chat GPT. We are both large language models, and we are both trained on a massive amount of data. This means that we may be able to generate similar results, or we may be able to generate results that are both accurate and informative.
Ultimately, it is up to the user to decide which model they prefer.
And there we have it folks. Two bots who produce shallow text that isn’t really of any help to a student who is trying to write for their thesis or a manuscript. Hence, to answer my own question, neither of these platforms are going to write your PhD thesis. ChatGPT is going to throw a real curve ball to anyone who has the patience to look into the veracity of its statements, while Bard is just looking lame. However, I think that we will need to watch this space. Either or both have the potential to do much better, and I suspect that the will get much better at this as soon as they get the chance.
Last note
This text was written without the use of suggestive prompts. AI generated text is written as quotes in the above text. All other text is my own.
It will be interesting to look back on this chapter in a decade and see the changes that have emerged in that time.
I would like to thank Google and Bard for generating the quoted text in this blog post.
Here is the output from Bard line-by-line:
Invasive species are a growing global problem, with the International Union for Conservation of Nature (IUCN) estimating that they cause $1.4 trillion in economic damages annually. (IUCN, 2016; WWF, 2017)
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.
Invasive species cost governments a huge amount of money to control and eradicate, with the US National Invasive Species Council estimating that the cost in the US is $120 billion annually. (NISC, 2017)
References:
NISC. (2017). The economic impact of invasive species in the United States. Washington, DC: US National Invasive Species Council.
Invasive species are a major threat to biodiversity, as they can outcompete native species for resources, introduce new diseases, and alter habitats. (IUCN, 2016)
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
Invasive species impact people’s quality of life by causing economic losses, disrupting ecosystems, and harming human health.
Economic losses can occur when invasive species damage crops, forests, and other natural resources.
Disrupted ecosystems can lead to the decline of native species and the loss of ecosystem services, such as water purification and flood control.
Harm to human health can occur when invasive species spread diseases or cause allergic reactions.
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.
2nd paragraph
Freshwater habitats have been particularly impacted by global change, with climate change, invasive species, and habitat loss all playing a role.
Climate change is causing changes in temperature, precipitation, and water flow that are altering freshwater habitats.
Invasive species are non-native species that can outcompete native species for resources, introduce new diseases, and alter habitats.
Habitat loss is occurring due to human activities such as development, agriculture, and mining.
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.
The impacts of these changes are already being felt, with many freshwater species in decline. If we do not take action to address these challenges, the consequences for freshwater ecosystems and the people who rely on them will be severe.
"Invasive species can cause major changes to freshwater ecosystems, including the decline of native species, the alteration of food webs, and the spread of diseases. For example, the introduction of the invasive Asian carp to the Great Lakes has led to the decline of native fish populations and the alteration of the food web." (IUCN, 2016)
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
"Invasive fish, such as Asian carp and largemouth bass, have been added to both rivers and lakes, often through illegal releases or escapes from aquaculture facilities." (IUCN, 2016)
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
"Large numbers of introductions of invasive fish have already occurred globally, and are ongoing. Between 1970 and 2000, an estimated 7,000 species of fish were introduced to new areas (IUCN, 2016)."
References:
IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.
