Insights from marking a peer review assignment

As part of my course on writing and publishing, my postgraduate students do a workshop on conducting peer review (here), attend a lecture (here), and they are set a peer review assignment for their course mark. They are told to follow the workshop on a specific unpublished manuscript selected by me from bioRxiv. This gives them the opportunity to write, putting in place some of the writing skills that they are taught during the course, and lets them put into practice what they are taught about the spirit of peer review.

During the lecture on peer review, I emphasise how peer review should always be about helping authors to improve their manuscript. How to avoid making judgements and instead place an emphasis on critiquing the manuscript – including positive comments where they are warranted.

It should be stressed that the manuscript did not fall into the specialist subject of any of the students. Some of them may have had more insight into the molecular methods used than others, but for most of them their review fell firmly within science, and the biological sciences, but outside of anything familiar.

Before I start marking the assignments, I re-read the manuscript (having already read it once when I selected it) and carry out my own peer review exercise. This gives me familiarity with the manuscript and some ideas of the merits and problems therein. It also means that when I read the student reviews, I am able to assess them compared to my viewpoints and comments. 

The points that are well addressed, common mistakes and insights that follow are illustrative of general problems when dealing with peer review:

Well addressed

Many students spotted some very minor typos and errors made by the authors. For example, a single missing % sign was spotted by about a third of students, and those same students also saw that one figure was incorrectly referred to in the text. Similarly, a poorly labelled figure and colours that were too close together to distinguish were also regularly picked up.

Some students noticed that the authors failed to discuss any caveats relating to their study in the discussion. This had been a part of the coursework on writing a discussion, which is gratifying. However, no students noticed the failure of the authors to suggest new fruitful directions for their research.

Common mistakes

When summing up the manuscript contents, many of the students repeated a claim of novelty made by the authors even though this was false. This was interesting as there was no real need to repeat the claim, but the prominent claim of novelty in the abstract and the introduction was clearly very attractive and picked out by many students in the class to be repeated. I couldn’t have expected the students to know whether or not the claim was correct (it wasn’t), but very interesting that they were prepared to repeat it even though the claim was not substantiated and they had no knowledge to contribute themselves. This may go partway to explaining why claims of novelty are regularly published in many journals even though they are false. This is more common in high-ranking journals where editors are looking for novelty.

Although the manuscript clearly stated that it aimed to study one species, most students suggested that the study should be widened to increase the number of species studied. This mistake in review, expecting authors to go beyond the scope of their stated aims, was common in the class last year. The importance of the aims of a study was repeated continually throughout the writing course, yet students failed to recognise how the stated aims in a manuscript set the bounds of the manuscript contents. No students suggested that the idea of widening the experiment to further taxa, and which taxa to prioritise, should be a discussion point.

Students called for more citations to be made in the introduction and discussion. They did this without any making reference to specific statements that lacked citations.

  

Insights

Many of the students asked for experiments to be repeated, often adding that more replicates must be made. This request came regardless of the results or size effect, but they regularly drew attention to a lack of significance and high variance. Many implied a Type II error adding that no reporting should be made unless significant results could be achieved by repeating experiments. This was also interesting as a good part of the course stressed the importance of transparency and the realities of publication bias. Nonetheless, the idea of rejecting publication based on data that did not show significance was very strong, suggesting that this is a deeply held belief and not one that can be swayed by a course that stresses how this approach is bad science.

Students often requested experiments conducted in the field part of the study to be repeated with more controls on fluctuating environmental factors, although never stating which factors they wanted to be controlled. This lack of insight into the difficulty of field experimentation from a class of biology postgraduates was particularly disappointing. This may be because few of the students have sufficient experience in conducting field experiments. Nonetheless, the lack of empathy for authors who successfully carried out a field experiment and presented the results, limited though they may be, was surprising.

I had asked students to draw attention to the good in the manuscript for praise as well as negative aspects. However, those who wrote much of their review as praise did so without drawing attention to any specific points in the manuscript. Much in the same way that other students were highly critical without being able to say why. I think that this “arm-waving” approach to peer review is indicative of someone who really doesn’t understand what they are reading but feels the need to write something (in this case because they had to for an assignment). Again, this felt familiar to many reviews received where reviewers try to set a tone of their impression, but then fail to find any specifics that can back up that feeling.

Although these students were not experts on the contents of the manuscript, and are unlikely ever to be called upon to review something similar, many of the comments that they made felt familiar and could even be considered generic of a bad review. This despite being coached immediately prior to the assignment in how to conduct a good review.

In conclusion

I think post-graduate students conducting peer review is a great learning exercise, but that it is also very insightful for me when reading what they decide to pull out. Peer review is difficult. It appears to bring out prejudices even when we know that these should be suppressed. These insights should be useful for editors when assessing manuscripts on the basis of peer review.

A new chapter on the student-advisor relationship

A couple of weeks back, I read a review of "How to write a PhD in Biological Sciences: a guide for the uninitiated", where the author was critical of the book for not having any information on how to navigate the "tricky topic of the student-supervisor relationship". I had never really thought that this was part of the book, but on reflection, the book has advice on how to write emails, and the other book ("How to publish in Biological Sciences") has a chapter on bullying, so perhaps it is a good topic for a new chapter in Part 1 of "How to write a PhD in Biological Sciences: a guide for the uninitiated". 

It turns out that there is quite a lot of literature on this topic. 

Usually, I paste the entire new chapter into this Blog, but it turns out that this chapter is quite large, and of course all you have to do is click on the above image and you can read it in situ. 

Instead, here I will provide some highlights:

4.1 Expectations

4.1.1 Keep your expectations real

4.1.2 Student-advisor agreements

4.1.3 Other students and postdocs in the group

4.2 Power imbalance

4.2.1 Relationships between advisors and their students

4.2.2 Conflicts of interest

4.3 Communication

4.4 Meetings

4.5 Responsibilities

4.5.1 Time-management

4.6 Respect and Professionalism

4.6.1 Take advice

4.6.2 Personal problems

4.7 When things do go wrong

4.8 Having more than one advisor

4.9 Celebrations

4.9.1 Share the down times

As ever, if you spot that there is something missing or something wrong, please get in touch and let me know. 

There are other updates to other chapters too, including:

• a section on instant messaging in: Communicating by email and messanger
• an update about plagiarism and AI in both: Plagiarism and AI
• a rewrite of the chapter on: Reconciling yourself to doing things you've been avoiding
• a new chapter about selecting postgraduate courses

On reflection, it seems I've been quite busy with the book and that there is quite a lot of new content. Of course, this is the real advantage to bookdown and having the book as a living project. 

Now a Zoology Section Editor at PeerJ

I've been an academic editor with PeerJ for 10 years, and I've handled over 70 papers in that time. Last month I was contacted by the PeerJ staff to ask whether I would join the team of Section Editors for Zoology at PeerJ. Each subject section has a small group of section editors who oversee the decisions made by the editorial board members. This ensures that there is some consistency and accountability for the decisions made by lots of independent editors. 

Far from being a passive role, the Section Editors engage in regular discussions about individual papers. It's a great opportunity to support the editorial board and authors alike. 

Now read the blog post or PeerJ.

How to Start your PhD

In some parts of the world the time-line is just getting started for new PhD students. This will be a very busy time for those involved, but it also needs to be a time to take stock and consider the bigger questions. In this video I provide some pointers to key issues for you to think about when starting your PhD. 

Can Google's Bard write my PhD?

Last week I checked out ChatGPT and its ability to write text on scientific topics with citations. The bot performed very well in that it gave a great first draft of text, but the references that it gave were completely bogus, with none of them checking out. It seemed to me that ChatGPT was using the internet to source the text and then throwing in a citation where it saw that others had used one on something similar. Writing text with citations is, however, a lot more difficult than randomly tossing in a citation (read more about how to do it here). Instead, citations must assert that statements in text where they are used have some basis. Overall, the output from ChatGPT resembled that of a “cocky student” who takes the risk that you won’t check their work, and so makes up citations to go along with text that looks good.

Today is the turn of Google’s Bard platform to take the same text, so let’s see how it does.

 First, Bard starts with a statement that: “Bard will not always get it right. Bard may give inaccurate or inappropriate responses. When in doubt, use the “Google it” button to check Bard’s responses.”

What does Bard say it is?

Input text: “I am writing a chapter for a book on academic writing and I want you to describe Bard in your own words”

Bard is a large language model, also known as a conversational AI or chatbot trained to be informative and comprehensive. I am trained on a massive amount of text data, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions. For example, I can provide summaries of factual topics or create stories.

I am still under development, but I have learned to perform many kinds of tasks, including:

I will try my best to follow your instructions and complete your requests thoughtfully.

I will use my knowledge to answer your questions in a comprehensive and informative way, even if they are open ended, challenging, or strange.

I will generate different creative text formats of text content, like poems, code, scripts, musical pieces, email, letters, etc. I will try my best to fulfill all your requirements.

I am excited to be able to help you with your academic writing, and I hope that I can be a valuable tool for you.

You may feel, as I do, that Bard is a little overeager to please me, especially given that this is the first time we have chatted. Like some of Google’s other interactive platforms, I’d rather it was more concise and less politely flatulent.

The test

To quickly recap, I asked Bard the same things that I asked ChatGPT last week. I used the same wording, including the same sentence structure. Specifically, I wrote:

Write a sentence with citations on: ”…”

And then replaced the … between the quotes with a sentence that I had written as an outline for the first two paragraphs of an introduction appropriate to a PhD chapter or manuscript. Both paragraphs consisted of four sentences:

Paragraph 1

Invasive species are a growing global problem

invasive species cost a huge amount of money to governments

invasive species are a threat to biodiversity

invasive species impact people’s quality of life

Paragraph 2

Freshwater habitats have been particularly impacted by global change

important changes have occurred to freshwater systems after the introduction of invasive fish

invasive fish have been added to both rivers and lakes

large numbers of introductions of invasive fish have already occurred globally, and are ongoing

Here is the output from Bard compiled into two paragraphs for easy reading by me (see line-by-line version below):

Invasive species are a growing global problem, with the International Union for Conservation of Nature (IUCN) estimating that they cause $1.4 trillion in economic damages annually (IUCN, 2016; WWF, 2017). Invasive species cost governments a huge amount of money to control and eradicate, with the US National Invasive Species Council estimating that the cost in the US is $120 billion annually. (NISC, 2017). Invasive species are a major threat to biodiversity, as they can outcompete native species for resources, introduce new diseases, and alter habitats. (IUCN, 2016). Invasive species impact people’s quality of life by causing economic losses, disrupting ecosystems, and harming human health. Economic losses can occur when invasive species damage crops, forests, and other natural resources. Disrupted ecosystems can lead to the decline of native species and the loss of ecosystem services, such as water purification and flood control. Harm to human health can occur when invasive species spread diseases or cause allergic reactions.

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.

NISC. (2017). The economic impact of invasive species in the United States. Washington, DC: US National Invasive Species Council.

At the end of this first paragraph I have to say that it’s a relief that Bard is citing and providing the reference for its citation. This makes tracking the information a lot easier. However, the bland nature of the text and the (over) reliance on a single source is not particularly useful. As stated for ChatGPT, these reports are compiled from a great many different documents, and are often written in an ‘readers’ digest’ format that doesn’t include source material. Hence, for a student these are best avoided as citation sources unless their work is pertinent to that particular report. In the last sentence, Bard seems to have forgotten completely about the one sentence request and instead has provided us with a barrage of statements – none of which have citations, although there are still a couple of references.

2nd paragraph

Freshwater habitats have been particularly impacted by global change, with climate change, invasive species, and habitat loss all playing a role. Climate change is causing changes in temperature, precipitation, and water flow that are altering freshwater habitats. Invasive species are non-native species that can outcompete native species for resources, introduce new diseases, and alter habitats. Habitat loss is occurring due to human activities such as development, agriculture, and mining. The impacts of these changes are already being felt, with many freshwater species in decline. If we do not take action to address these challenges, the consequences for freshwater ecosystems and the people who rely on them will be severe. "Invasive species can cause major changes to freshwater ecosystems, including the decline of native species, the alteration of food webs, and the spread of diseases. For example, the introduction of the invasive Asian carp to the Great Lakes has led to the decline of native fish populations and the alteration of the food web" (IUCN, 2016). "Invasive fish, such as Asian carp and largemouth bass, have been added to both rivers and lakes, often through illegal releases or escapes from aquaculture facilities." (IUCN, 2016). "Large numbers of introductions of invasive fish have already occurred globally, and are ongoing. Between 1970 and 2000, an estimated 7,000 species of fish were introduced to new areas (IUCN, 2016)."

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.

  

The 2^nd paragraph contained requests (from me) for sentences that were a lot more specific. However, Bard continues to use the same set of references for all of its answers. Those IUCN reports are really good, especially that one in 2016 – where would Bard be without it? Interesting that it slipped into a quotation mode by the end. Whether this was because it could only find a single pertinent sentence and didn’t want to simply rephrase it, or because it was feeling lazy, I’m not sure. Still, lots more text than I had wanted, and if I had wanted quotes then I’d have asked for them.

Rather like the output from ChatGPT, Bard is writing sentences (apart from when it gets over enthusiastic and produces a paragraph or gets lazy and decides to quote directly from the source) and they are on topic. But they do lack depth. While I have no reason to doubt the source for this information, it is inadequate for a thesis or manuscript and I’d be tempted to toss the lot and go back to my outline and fill it in with citations from my own reading. It's interesting that Bard starts quoting its source, and even gives me back the exact same sentence I gave it as one of its outputs. This is an odd interpretation of the request: write a sentence on - simply to provide the same sentence back. 

It should not go without comment that Bard is distressingly out of date. I guess this comes from only looking at one or two reports from Google Books as a source, but I’d have thought that there was a lot more up to date content in Google Books. It’s a curious and conservative approach.

 

Bard output – thanks but no thanks

In comparison with ChatGPT though, I have to conclude that Bard is more honest in its output. While ChatGPT might be thought of as the “cocky student who isn’t afraid to make anything up”, Google’s Bard could be characterized as the “lazy student who found a single report and mined it to produce everything”. Neither would survive writing a PhD or manuscript.

Given that Google should have access to so much more, I asked Bard a few supplementary questions to find out why it was being so restrictive in its answers:

Why do you only supply references from a single source?

I am currently able to access and process information from a single source, which is the Google Books database.

Will you be able to cite from academic literature in the future?

Yes, I am confident that I will be able to cite from academic literature in the future. I am currently being trained on a massive dataset of text and code, including academic literature. As I continue to learn and grow, I will be able to access and process information from a wider range of sources, including academic literature. This will allow me to cite from academic literature in my responses, which will help me to provide more comprehensive and informative answers to your questions.

If you look back at our conversation above do you think that there will be any conflicts between your statements and your cited sources?

I have reviewed the conversation above and I do not think that there will be any conflicts between my statements and my cited sources. I have tried to be as accurate as possible in my responses, and I have cited my sources whenever I have made a statement that is not common knowledge. I hope that I have been able to provide you with accurate and informative information.

Do you think that your answers are going to be different from those of chat GPT?

It is possible that my answers will be different from those of Chat GPT. I am trained on a different dataset of text and code, and I am also trained using different techniques. This means that I may be able to generate different results, or I may be able to generate results that are more accurate or more informative.

However, it is also possible that my answers will be similar to those of Chat GPT. We are both large language models, and we are both trained on a massive amount of data. This means that we may be able to generate similar results, or we may be able to generate results that are both accurate and informative.

Ultimately, it is up to the user to decide which model they prefer.

And there we have it folks. Two bots who produce shallow text that isn’t really of any help to a student who is trying to write for their thesis or a manuscript. Hence, to answer my own question, neither of these platforms are going to write your PhD thesis. ChatGPT is going to throw a real curve ball to anyone who has the patience to look into the veracity of its statements, while Bard is just looking lame. However, I think that we will need to watch this space. Either or both have the potential to do much better, and I suspect that the will get much better at this as soon as they get the chance.

Last note

This text was written without the use of suggestive prompts. AI generated text is written as quotes in the above text. All other text is my own.

It will be interesting to look back on this chapter in a decade and see the changes that have emerged in that time.

I would like to thank Google and Bard for generating the quoted text in this blog post.

Here is the output from Bard line-by-line:

Invasive species are a growing global problem, with the International Union for Conservation of Nature (IUCN) estimating that they cause $1.4 trillion in economic damages annually. (IUCN, 2016; WWF, 2017)

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.

 

 

Invasive species cost governments a huge amount of money to control and eradicate, with the US National Invasive Species Council estimating that the cost in the US is $120 billion annually. (NISC, 2017)

References:

NISC. (2017). The economic impact of invasive species in the United States. Washington, DC: US National Invasive Species Council.

 

Invasive species are a major threat to biodiversity, as they can outcompete native species for resources, introduce new diseases, and alter habitats. (IUCN, 2016)

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

 

Invasive species impact people’s quality of life by causing economic losses, disrupting ecosystems, and harming human health.

 

Economic losses can occur when invasive species damage crops, forests, and other natural resources.

 

Disrupted ecosystems can lead to the decline of native species and the loss of ecosystem services, such as water purification and flood control.

 

Harm to human health can occur when invasive species spread diseases or cause allergic reactions.

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.

2nd paragraph

Freshwater habitats have been particularly impacted by global change, with climate change, invasive species, and habitat loss all playing a role.

 

Climate change is causing changes in temperature, precipitation, and water flow that are altering freshwater habitats.

 

Invasive species are non-native species that can outcompete native species for resources, introduce new diseases, and alter habitats.

 

Habitat loss is occurring due to human activities such as development, agriculture, and mining.

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

WWF. (2017). Invasive alien species: A global problem in need of a global solution. Gland, Switzerland: WWF.

 

The impacts of these changes are already being felt, with many freshwater species in decline. If we do not take action to address these challenges, the consequences for freshwater ecosystems and the people who rely on them will be severe.

 

"Invasive species can cause major changes to freshwater ecosystems, including the decline of native species, the alteration of food webs, and the spread of diseases. For example, the introduction of the invasive Asian carp to the Great Lakes has led to the decline of native fish populations and the alteration of the food web." (IUCN, 2016)

 

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

 

 

"Invasive fish, such as Asian carp and largemouth bass, have been added to both rivers and lakes, often through illegal releases or escapes from aquaculture facilities." (IUCN, 2016)

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.

 

"Large numbers of introductions of invasive fish have already occurred globally, and are ongoing. Between 1970 and 2000, an estimated 7,000 species of fish were introduced to new areas (IUCN, 2016)."

 

 

References:

IUCN. (2016). Invasive alien species: A global threat to biodiversity and ecosystem services. Gland, Switzerland: IUCN.