Tips and tricks to evaluate the impact of ChatGPT in scientific writing

Writing a scientific paper is a challenging endeavour that demands deep knowledge, critical thinking, and clear communication. Increasingly, researchers are turning to artificial intelligence (AI) tools like ChatGPT to assist with writing, yet this reliance brings significant risks. While AI can improve grammar and structure, it falls short in generating new scientific ideas or questioning established ones, which undermines the critical thinking essential to research. Moreover, when AI contributes heavily to a paper, ethical concerns emerge regarding authorship and accountability, prompting journals to insist on full disclosure of AI involvement.^1,2 Overdependence on AI may also lead to a uniformity of expression that stifles diversity of thought, and the potential for accidental plagiarism rises if the technology fails to paraphrase appropriately.^1,2 These challenges raise pressing questions about how reviewers can detect AI influence and the broader implications for maintaining research integrity.

How to spot AI-written papers: AI-generated text often has certain patterns that make it recognizable. Here are some key signs:

• Generic and repetitive language: AI-generated text often sounds polished but vague. While it may be grammatically correct and polished, it avoids making strong claims or introducing novel insights. It reuses sentence structures and phrases, creating a repetitive and unnatural flow.

• Depth of argumentation: The depth of argumentation should be assessed. Papers that rely heavily on summarization without critical analysis or that do not engage with alternative perspectives may indicate AI involvement. Scientific papers require a logical flow of arguments backed by data and citations.

• Superficial conclusions: AI-generated text often presents overly simplified conclusions, lacking the complexity and nuance expected in rigorous academic discourse.

• Inconsistent terminology: Experts use precise terms in technical fields like imaging research. AI can sometimes mix up or misuse terminology, inconsistently applying definitions or failing to align with domain-specific language conventions. If a manuscript fluctuates between different definitions for the same concept or misuses key terminology, it may indicate AI assistance.

• Fake or incorrect references: ChatGPT or other tools sometimes generate references that do not exist or misattribute citations, making fact-checking essential.

The challenge of AI detection: Despite these indicators, detecting AI-generated text remains difficult, even for experts. Researchers worldwide are developing automated tools to detect AI-written content. Early efforts include GLTR (Giant Language Model Test Room) from Harvard and MIT (2019), designed to analyse texts produced by GPT-2. However, as AI models advance, detection tools have struggled to keep pace. Studies show that even well-educated individuals can misidentify AI-generated texts nearly half the time.

Several free tools claim to detect AI-generated text based on statistical-linguistic metrics such as ‘burstiness’, which measures the unpredictability of language patterns. AI-generated texts tend to have higher burstiness due to their reliance on statistical modelling rather than natural human variation.

AI detection tools: Do they work? Various online tools attempt to determine whether a text was AI-generated:

• AI Text Classifier (OpenAI): Assigns a probability score but requires at least 1000 characters. Performs best in English but struggles with some AI-generated texts.

• GPTZero: Detects AI-generated content but has limitations, particularly in non-English languages.

• ZeroGPT: Found only 30% of AI-generated content in an article written entirely by ChatGPT.

• CopyLeaks, Writer, and NoPlagio: Offer detection services with varying success rates.

• Originality.ai, GetMerlin, and Corrector: Use NLP models and deep learning to compare AI.

While none of these tools are foolproof, employing multiple detection methods may enhance accuracy. However, experts caution that AI detection tools will become less reliable as AI-generated text increasingly resembles human writing. AI models are evolving to closely imitate human style, making systematic detection more challenging.

Conclusion: ChatGPT and other AI tools are reshaping the way researchers approach scientific writing. While AI can serve as a useful assistant, over-reliance on it can weaken the quality and credibility of research. The growing reliance on AI signals a troubling cultural drift away from deep, critical inquiry, challenging us to preserve the human element that fuels genuine scholarly innovation. To preserve academic integrity, journals, reviewers, and authors must work together to ensure AI is used responsibly—enhancing writing without replacing human expertise. The future of scientific communication depends on maintaining these standards.

Funding

None declared.

Data availability

No new data were generated or analysed in support of this research.

Lead author biography

Dr Alessia Gimelli is a cardiologist and nuclear medicine specialist. She is the Director of the Health Quality System at Fondazione Toscana G. Monasterio, Pisa, Italy. Dr Gimelli has deep experience in clinical research on cardiovascular diseases and cardiovascular imaging and has contributed to several international high-profile projects. She is the Editor-in-Chief of the European Heart Journal – Imaging Methods and Practice.

References

Author notes