Writing for Pediatric Critical Care Medicine: Engaging With Citations to References in the Chatbot Generative Pre-Trained Transformer Era

Abstract

Writing a clinical research article for Pediatric Critical Care Medicine (PCCM) involves reproducing for readers the sequence from ideas to measurements, observations, and evidence, to reasoning and conclusions (1,2). In academic medical writing a Citation (i.e., the act of referring to others) involves three steps: 1) identifying some aspect of other’s work that is useful for readers; 2) quoting a Reference to the source; and 3) providing bibliographic details (3). The references highlight articles that substantiate the topicality of a particular question in the field, the provenance of an idea, and the appropriateness and validity of experimental methods, as well as studies that serve as comparisons to the results (4). At PCCM, we recommend that a 3,000-word Structured Clinical Research Report include up to 40 references: 10–15 in the Introduction, possibly some in the Methods, leaving 20 for the Discussion (1). Table 1 summarizes the traditional rules when compiling a Reference list and it is a given that for every citation the PCCM author acts as the intermediary between the content of the primary source and our readers. TABLE 1. - Rules for Citations to References Section Rules for Citations to References Introduction 10–15 references: Cite references for statements of fact; not necessary if it is common knowledge Cite your other publications that precede or overlap with the current work Cite essential background references, but no more than five studies Cite a recent review if it is a better summary of the field Methods 5–10 references: Cite references for measurements using an assay, instrument, or questionnaire Cite references that define diagnostic criteria used in the study Do not cite a reference for standard or commonly used statistical tests; citing the software package will suffice Cite a reference for complex or less-known statistical analyses that will be unfamiliar to readers Results No references: If you think references are needed, they probably belong to the Methods or Discussion sections Discussion 15–20 references (total of 40 in a Clinical Research Report): Cite references to prior studies that are used as comparison for the current work Cite references that support possible explanations or implications Cite references that contextualizes the study WHAT IS ON THE HORIZON FOR CITATIONS TO REFERENCES? The International Committee of Medical Journal Editors describes the responsibilities of authors and journals for cited references Table 2 (5). That guidance is followed by PCCM, but changes in 2023 have meant that some recalibration is needed. TABLE 2. - Responsibilities for Publication of Citations to References Citation to References Responsibilities Reference list review Journals (by some, but not all): Risk of citation errors in published articles Accuracy of references Authors: Verify source and content of original article by reading the entire article Check that no reference cites retracted articles Identification of retracted articles Authors: Verify identity in PubMed before the proofing stage In June 2023, two academics produced a narrative and statistical analysis about the impact of fruit and vegetable consumption and physical activity on diabetes risk in adults (6,7). The authors started with the 2015 survey results from the U.S. Centers for Disease Control and Prevention Behavioral Risk Factor Surveillance System. Then, a data-to-paper tool served as an assistant acting between themselves and the artificial intelligence (AI) language model, the Chatbot Generative Pre-trained Transformer (ChatGPT). The final “writing” took an hour and shows how any research dataset with access to literature search engines could be used to generate a semiautonomous, AI, data-to-paper manuscript. There are many questions that now arise for journals when we think about future reviewing and quality control. Does semi-supervised statistical testing in dataset research risk misleading readers (8)? Have ethical principles guiding AI use in health research been followed (9)? Is clinical science at threat (10)? Is there a risk of plagiarism (11)? How are the citations to references chosen (12)? So far, two critical care journals (Critical Care and Anaesthesia Critical Care & Pain Medicine) have published material about AI and academic medical writing (13–15). One perspective is “…consensus on how to regulate the use of chatbots in scientific writing will soon be required” (13). Such an agreement will take time. However, both critical care journals emphasize that readers should check citations to references, because of two problems. First, ChatGPT has limited access to literature databases, finding works only up to September 2021. (As an illustration, the discussion in the diabetes data-to-paper manuscript does not have literature references after 2019 [7]; find the up-to-date papers in PubMed.) Second, ChatGPT may generate random, false, or nonexistent references, called “AI hallucinations” (16). Taking all of the above into account, my conclusion is that authors will use ChatGPT and AI as an efficient way of gaining insight into a topic—but writing articles is a step too far until we understand more. I know that university or college students—even high schools—are embracing this approach. However, until ChatGPT plug-ins to databases such as PubMed are developed, there will remain a blind spot to new material, items without Abstracts, and articles that are not Open Access. In response to these possibilities, and problems, the editorial in Anaesthesia Critical Care & Pain Medicine said “…ChatGPT and other language models should make us, researchers, realize that only we can elevate scientific writing” (14). My paraphrase for PCCM is that we can use the opportunity enabled by ChatGPT and raise our standard of writing to a new level. PCCM IN THE NEW ERA: A FRAMEWORK FOR ENGAGING WITH REFERENCES This next item in the PCCM Notes, Methods, and Statistics series—already seven to date (1,2,17–21)—is therefore focused on a new framework for engaging with references in our Journal articles. The purpose is to elevate writing for PCCM, and in no way is it a criticism of past publications and our exemplary authors. Rather, we can use the opportunity of the ChatGPT-era to have a new approach that supports a reader-focused, structured engagement of cited references. This framework also adds to the 2023 initiative of the Society of Critical Care Medicine (SCCM) Reviewer Academy (22). There are five key questions about citations to references that should be thought about by authors submitting to PCCM (Table 3). These questions are an extension to our editorial principles (i.e., Read, Rigor, Relevant, Responsive, and Reach [23]) and the purpose is to help writers produce a more informative manuscript for readers, so that reference content is better appreciated. Briefly, PCCM will be asking authors to include details about the content of references that help readers to understand answers to questions such as: What is? What was? Why now? How? What are the appropriate comparisons and ongoing narrative? TABLE 3. - Questions to Ask When Including Citations to References in a Pediatric Critical Care Medicine Research Report Questions Citations to References No. 1: Does it describe “What is?” Introduction to prospective work: References about “what is?” are dated to before the start of cohort enrollment Introduction to retrospective work: Use any references about “what is?” dated before the start of analysis No. 2: If not “What is?,” then how about “What was?” Introduction to either prospective or retrospective work: References about rates and prevalence: Let “what is?” vs “what was?” be guided by epidemiology References about practice: Let “what is?” vs “what was?” be guided by timing of clinical practice guidelines or professional standards Other references: Let “what is?” vs “what was?” be guided by timing of cohort and not year of publication No. 3: Does it describe “Why now?” Introduction to retrospective work: Use references that describe the progression in ideas Refer to any previous or overlapping publications with the cohort No. 4: Describe “How?” methods apply to the cohort Methods in prospective work: References to a method, score, or definition predates the period of data collection Methods in retrospective work: References to a method, score, or definition must be available at the time of analysis. In essence, a “what if?” study in a convenience sample No. 5: What comparisons and narrative?” Discussion of comparison in prospective work: Include references to works since cohort enrollment Discussion of comparison in retrospective work: Ensure work in historical references are held to the appropriate standard Discussion on extending the narrative in either prospective or retrospective work: Check post 2021 (i.e., post-Chatbot Generative Pre-trained Transformer range) references in the pediatric critical care literature QUESTION NO. 1: DOES THE REPORT ADEQUATELY DESCRIBE “WHAT IS?” The Introduction to a clinical research or brief report in PCCM has two paragraphs (1,2). The first paragraph provides setting and context with enough information for readers to understand the reasoning presented in the second paragraph, which is about aims and hypotheses tested. Those citations to references are used to describe ideas in two types of study. Prospective Recruitment Consider, for example, an imaginary preplanned, prospective study carried out January 2020 to December 2022. Here, readers may need to be able to read references that explain the “what is?” of the clinical setting and context. In this example, the references should have been published in 2019, or before; references from 2021 to 2023, although more up to date, are misplaced in this Introduction because that information or ideas conveyed would not have been known at the time of study planning. Citations to 2021–2023 references should appear in the Discussion (see below, Question No. 5). Retrospective Cohort or Database Research Now consider a post hoc analysis of a dataset or single-center patient chart review. Here, the citations to references used to explain “what is?” can be from any time up to the timing of the analysis. That is, authors have had an idea about something important now, and it is being tested in a convenience sample. The first paragraph should establish how the idea arose. QUESTION NO. 2: IF NOT “WHAT IS?,” THEN HOW ABOUT “WHAT WAS?” When is “what was?” more accurate than “what is?” in the opening paragraph of the Introduction? Here, three possible explanations with added information from the references will help readers of both “prospective” and “retrospective” studies. First, if a reference is used to describe rate of admission, mortality or morbidity, or prevalence of some disease or complication, then year(s) of the cohort in the reference will be useful in the text. For example, consider the proportional use of PICU beds in the U.S. Pediatric Health Information System dataset of 43 children’s hospitals; admission rates increased during 2014–2019 (24). Now, we also know that U.S. school closures to mitigate the spread of COVID-19 in early 2020 were associated with significant decreases in standardized admission rates to 81 hospitals (in the U.S. Virtual Pediatric Systems LLC database; https://myvps.org) for bronchiolitis, asthma, and pneumonia when compared with 2018–2019 (25). So, when needing to use a citation about respiratory epidemiology, the “what is versus what was?” may now need to be defined in relation to before versus after COVID-19 by also providing for readers the chronology of the cohort in the cited reference. These same principles apply to other diseases or diagnoses. Second, whether a reference is describing the “what is versus what was?” consequence of a particular practice or management is best defined in relation to before versus after some widely accepted professional standard. For example, let readers know whether the reference about transfusion, or sedation, or pediatric acute respiratory distress syndrome (PARDS) practice in a cohort relate to a period before or after the introduction of the new guidance on transfusion (26), sedation (27), or PARDS (28), respectively. Third, if the reference is just used to describe some other type of cohort, the distinction between “what is versus what was?” may be down to the chronology of the cohort in the reference. The age of the cohort may be somewhere between 7 and 10 years, but it will depend on the subject area and the uniqueness of available data. Therefore, for readers, a dataset that closed recruitment 2013–2016 may not be important to understanding their PICU population managed in 2023/2024. Remember also that the year of publication of a reference does not determine whether its content can be categorized as “recent” or described as occurring “recently.” What matters are the dates of cohort enrollment. For example, a reference published in 2018, reporting a cohort from 2001 to 2010, is not about recent or contemporary practice. If that is the best information available, the solution for authors is to organize citations to references according to the timeline of the cohorts, rather than by year of publication. Help readers put this information into context by using the references to show how the chronology justifies the study plan, and it may even highlight the gap in what we know about “what is?” QUESTION NO. 3: CAN YOU DESCRIBE “WHY NOW?” Typically, the question of “why now?” is answered using text and citations to references that describe a progression in ideas. However, when reporting a “retrospective study” using a carefully curated and previously published clinical research dataset it is important to help readers with the evolution of the research plan, and how a published report has led to the current report. Consider an imaginary project in which a national acute kidney injury (AKI) dataset has been used to examine an association between fluid balance and the endpoint of, say, ventilator-free days, and the findings are published. Now, the same dataset is used in an unpublished report that examines an association and interaction between the diagnosis (or not) of infection or sepsis, fluid balance, and ventilator-free days. In the Introduction to the new report, our expert PCCM readers will not need a physiological account of sepsis, inflammation, and alveolar capillary membrane fluid leak research. Rather, our readers should be helped with an Introduction that describes findings in the published report and the reasoning that resulted in the work of the current report. Alternatively, an idea may have arisen from recently published research and the new submission is a test or validation of that idea in a different dataset. The citations to references in this case will cover both “Why now?” and “What was?” QUESTION NO. 4: CAN YOU PROVIDE DETAILS OF “HOW?” METHODS APPLY TO THE COHORT? The Methods section describes data collection and handling, methods of measurement, and the definitions of states, terms, and outcomes (1,2). Citations to references about methods used for measurement, and meaning ascribed to certain values, should also provide details of how they apply to the dataset in the report. For example, more explanation will be needed if a measurement has been validated only in older than 1-year-olds but that method is now being used in a PICU population of younger than 6-month-olds. Simply inserting the cited reference will not be helpful to readers who will want to understand potential limitations in the work. It may be the case that professionals working in the field use this methodology in all age groups even without validation; that is fine, readers just need to be told. Definitions also change. For example, during 2021–2023, PCCM provided updates to definitions for PARDS (28), AKI (29), and prolonged mechanical ventilation (30). Prospective Studies These studies, whether observational or interventional, have a defined period of recruitment. Citations to references about a method, scoring system, or definition used in the research must predate the period of data collection or cohort recruitment. Retrospective Studies Citations to references about a method, scoring system, or definition must have been available before the time of analyzing a retrospective cohort. Readers will also need to know that the work is a post hoc examination, or “what if?” study in a convenience sample. QUESTION NO. 5: WHAT ARE THE APPROPRIATE COMPARISONS AND ONGOING NARRATIVE? The Discussion provides an opportunity to present two or three main observations in the context of the “what is?” or “what was?” literature identified in the Introduction, as well as any newer reports that have been published. In this case, there are two writing formats that are helpful. Making Comparisons An observation is often generalized using a percentage. However, when making comparisons with percentages in cited references, it is helpful for readers to know that any potential differences being suggested have been checked with a statistical test of proportions. Invariably, there is no difference, which may suggest the new observation is no worse or no better than prior data. The other aspect of making comparisons is when considering the importance of some standard of care against which a dataset has been evaluated; it is useful to know that the cited references are appropriate and informative. As an example, consider the 2017 SCCM and American Society for Parenteral and Enteral Nutrition (ASPEN) guidelines for the provision and assessment of nutrition support therapy in the pediatric critically ill patient (31). The guidelines recommended a minimum protein intake of 1.5 g/kg/d (moderate quality of evidence, strong recommendation). Now, imagine a study cohort from 2018 or later; it is certainly appropriate to inform readers of an analysis comparing protein intake in this cohort with the 2017 SCCM/ASPEN standard (31). However, a comparison in protein intake between the new cohort and a reference cohort predating 2017 has no meaning. The authors of the pre-2017 work (or cohort) cannot be expected to have met a standard that did not exist at the time of publication or patient care. Extending the Narrative The overall goal of PCCM is to promote knowledge that accelerates change in our discipline and impacts and improves patients’ lives in our field (23). The articles in PCCM are published for clinical practitioners, for the SCCM Pediatric Section and the World Federation of Pediatric Intensive and Critical Care Societies, and for researchers. In 2020, PCCM published 30% of all original articles in pediatric critical care, and almost all the editorials and In this paragraphs 2 to of the Discussion should of extending the narrative in our literature (1). This search engines for references after (i.e., the current of It also that to substantiate the topicality of a particular question in the field, and importance to PCCM readers, references from the main pediatric critical care literature should be in the citations PCCM, in Journal of Pediatric Intensive Critical Care and Pediatric These are not to academic or but to help authors more whether PCCM has the readers for the work It may be that the work is better to a more Alternatively, the report may have been for now, it just to be for PCCM readers (1,2). In this ChatGPT-era of reference readers of PCCM can be that the Journal will to the to scientific writing” (14). new framework for engaging with citations to references is now an of and questions My three to authors to provide details from cited references that cover the essential questions about What is? What was? Why now? How? What are the appropriate comparisons and ongoing narrative? Second, to that PCCM will to out expert for reports who will help with the contemporary to use the editorial review to check the refe

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 10 citations exist upstream.

Papers that cite the same references — the relational fallback while embeddings are still backfilling.