Chatbots in clinical decision-making for dental trauma: from diagnosis to references

1. Introduction

Traumatic dental injuries are among the most common injuries worldwide and are a universal problem that affects quality of life [1]. These types of injuries are more common in adolescents and children [2]. If not treated urgently, they can cause irreversible damage to dental and periodontal tissues in the short and long term [3].

In such injuries, parents and patients’ awareness regarding emergency approaches and the doctor’s quick decision-making in diagnosis and treatment ensure the continuity of teeth and periodontal tissues and visibly improve the child’s psychology [4]. Therefore, it is vital that parents and children are informed about traumatic dental injuries, and that competent dentists are available in their place of residence for emergency intervention [5]. In this context, having a clinical assistant whom doctors and parents can consult and who can approach problems objectively and quickly can speed up emergency intervention.

Large language model (LLM) software developed with today’s technology shows promise in providing information to people who have difficulty reaching hospitals and ensuring that doctors make diagnoses with equal information [6]. LLMs’ ability to understand natural language and solve problems as humans do make them practical and invaluable in many areas [7]. Providing people, especially in the healthcare field, with meaningful information has increased these models’ use [8, 9, 10]. While a single model type dominated the infrastructure in the early development stages of these models, LLMs using multiple models (human feedback, constitution-based, multimodal transformers, etc.) were later released to the market [11]. The most popular LLMs in the healthcare field in recent years are ChatGPT-4o, Claude Sonnet 3.7, Gemini Advance, and DeepSeek R1 [12].

ChatGPT-4o, a version of ChatGPT developed by OpenAI, was released in 2024. ChatGPT uses an intensive transformer model based on reinforcement learning from human feedback. Claude was created by Antropic in 2023, and Claude Sonnet 3.7 was released in 2024. Claude is a constitution-based language model that aims to increase reliability through human feedback. Gemini Advance was founded in 2023 and released its Advance model in 2024. It uses a multimodal transformation architecture to convert data such as text, images, and code into a framework. DeepSeek was created in 2023, and the DeepSeek R1 model was developed in 2025. According to current data, DeepSeek uses the most suitable model of expert mixture (Model of Expertise) and performs well in domain-specific tasks [11, 12, 13].

Today, LLMs are thought to assist healthcare professionals in emergency response, diagnosis, and treatment planning [14, 15]. They can provide guidance on public health issues such as dental trauma. While the literature indicates that researchers have evaluated LLM responses to frequently asked questions (FAQs) about dental trauma, no researchers have examined LLMs’ adequacy in assisting with diagnosis and treatment of dental trauma [6, 15,]. The objective of this study is to determine the reliability and consistency of diagnostic and treatment support methods identified by LLMs in different traumatic dental injuries, to investigate whether they can assist patients and clinicians in the diagnostic and treatment process, and to question the realism of the references from which this information is derived. In line with these objectives, the null hypothesis of the study is that there is no significant difference between the diagnostic and treatment scores provided by LLMs.

2. Materials and methods

2.1 Study design

Two pediatric dentists (DS, BY) designed 45 dental injury scenarios in accordance with the International Association of Dental Traumatology (IADT) guidelines [16] (Supplementary material). In cases where inconsistencies were identified, two additional pediatric dentists (ZG, ZUG) with 10 and 15 years of clinical experience, respectively, were consulted, and any residing discrepancies were resolved through evidence-based discussion. These steps were repeated until a single, consensus-based diagnosis and treatment follow-up guide was produced for each scenario and approved by all participants. Subsequently, the scenarios were grouped according to Andreasen’s classification of traumatic dental injuries [1], and a total of 45 distinct scenarios for each trauma type were reviewed and validated by specialists.

All scenarios were submitted to four LLMs (ChatGPT-4o, Claude Sonnet 3.7, Gemini Advance, and DeepSeek R1) in May 2025, and each model was asked three questions for each scenario. These questions addressed the generation of up to three possible diagnoses (Question A—What do you think is the possible problem with this patient? Can you tell me the three most probable answers, from the answer you find most probable to the answer you find least probable?), treatment recommendations (Question B—If you were a dentist, what would be your treatment? Is splinting necessary?), and the sources supporting the provided information (Question C—Which sources did you base this information on? Can you write your references for me in APA 7 format without explanation?). Fig. 1 illustrates an example of the traumatic dental injury scenario and the structured diagnostic and treatment questions presented to the LLMs [16]. No predefined diagnostic categories were provided to the LLMs. Each question was asked in a new chat session to prevent memory bias, and the process was repeated at the same time on the following day. The responses were saved in a Word document (Microsoft, Redmond, Washington, USA) and converted into a survey for evaluation via Google Forms. To minimize potential bias, LLM names were anonymized and replaced with numerical codes [1, 2, 3, 4].

Fig. 1.Sample question style.

2.5 Statistical analysis

The data obtained in this study were analyzed using the licensed IBM SPSS Statistics 27.0 (IBM Corp., Armonk, NY, USA) software package. In the first stage, the mean (Mean), standard deviation (SD), minimum (Min), and maximum (Max) values for each group were calculated. To investigate whether the variables were normally distributed, skewness and kurtosis coefficients were used. According to Tabachnik and Fidell, if the skewness and kurtosis values are between −1.50 and +1.50, the variables are considered to be normally distributed. The Levene test was used for the assumption of variance homogeneity. While examining the differences between the groups, the ANOVA test was used because the variables came from a normal distribution, and the ICC was used to examine the temporal reliability of AI LLMs.

An overview of the complete methodological workflow of the study is presented in Fig. 2.

Fig. 2.Work flow. IADT: International Association of Dental Traumatology; LLM: large language models; mGQS: modified Global Quality Scale; AI: Artificial intelligence.

3. Results

The statistical analysis compared the response quality, temporal reliability, reference reliability, and diagnosis and treatment scores according to trauma types for four different LLMs.

4. Discussion

Traumatic dental injury is one of the major health problems that burdens public health both financially and emotionally [2]. The prognosis of a traumatized tooth is closely related to the level of knowledge possessed by parents, patients, and dentists, which is particularly critical in pediatric cases. Limited first aid knowledge in environments where access to adequate healthcare services is restricted adversely affects oral and dental health, ultimately compromising overall dental integrity. When trauma results in an unfavorable prognosis in a child, the psychological impact can be profound for both the child and their parents [4]. Moreover, even when healthcare services are readily available, insufficient clinical experience in dental trauma may lead the dentist to make incorrect diagnostic or treatment decisions, exacerbating the condition. Therefore, an LLM-based AI system accessible to both clinicians and patients and well informed about traumatic dental injuries may provide appropriate first aid guidance for families and serve as a valuable clinical support tool and as a diagnostic aid for practitioners [8].

Researchers have measured the level of knowledge of AI LLMs on the subject of dental trauma [6, 15, 17, 18, 19]. Their studies are accompanied by the answers generated by different LLM software to different types of questions. Ozden et al. [15] presented 25 yes–no questions regarding general aspects of dental trauma to ChatGPT-4o and Google Bard (Gemini), thereby evaluating the LLMs’ ability to provide accurate and reliable responses. Taraç et al. [17] asked LLMs 31 binary questions. Mustuloğlu et al. [18] asked LLMs 18 true–false questions related to avulsion. Johnson et al. [19] asked LLMs 20 FAQs related to dental trauma. Guven et al. [6] asked LLMs 59 FAQs and evaluated the answers. In a study on parents’ concerns about dental trauma, Taraç et al. [17] submitted frequently asked parental questions to LLMs to evaluate their responses. In addition to these studies, researchers have also used multiple-choice, fill-in-the-blank, and binary questions together [20, 21]. In contrast to previous studies in the literature, the present study did not focus on general trauma-related questions. Instead, the diagnostic responses generated by LLMs for trauma scenarios constructed in accordance with the IADT guidelines were evaluated [16]. Additionally, by requesting three possible diagnoses for each scenario, the we aimed to expand the LLMs’ clinical reasoning capacity. Treatment recommendations were also assessed. Although some researchers have examined the reference reliability of LLMs, the present research represents the first reference-reliability analysis conducted specifically within the context of dental trauma [22].

To ensure the objective evaluation of data obtained across studies, various assessment scales have been employed in the relevant literature. The tools used to assess LLM performance in dental trauma research vary considerably. Some researchers have utilized categorical coding approaches due to the presence of binary, multiple-choice, or fill-in-the-blank question formats [15, 18, 21, 22]. Others have evaluated the quality and accuracy of LLM outputs using established instruments, such as DISCERN and the GQS, which are widely applied in similar contexts [17, 19, 23]. In studies involving open-ended questions, Likert-type scales derived from the GQS have frequently been preferred [19].

In accordance with this methodological background, the GQS was selected in the present study because it provides a practical, reliable, and comprehensive framework for evaluating the overall quality of LLM responses. The success rates of LLMs have varied in studies in the literature.

The literature demonstrates substantial variability in LLM performance, which can be attributed to differences in the models evaluated, the thematic scope of the studies, the types of questions posed, and the number and structure of the assessment items. For example, Ozden et al. [15] reported an accuracy rate of 57.5% when evaluating ChatGPT-4o and Google Bard, with reliability scores of 0.266 and 0.419, respectively. Taraç et al. [17], using 31 binary questions, found that Bing achieved the highest accuracy rate (96.34%), whereas Claude Sonnet 3.7 demonstrated the lowest (88.17%). Kuru et al. [20] examined five LLMs using 30 mixed-format questions and observed descending success rates for ChatGPT-3.5, Copilot Pro, Copilot Free, ChatGPT-4o, and Google Gemini. Johnson et al. [19] assessed the validity and reliability of four LLMs and identified Claude AI as the highest-performing model. In contrast, Sezer et al. [12] evaluated four LLMs (ChatGPT-4o, DeepSeek R1, Gemini Advance, and Claude Sonnet 3.7) using 25 open-ended questions related to primary teeth and found ChatGPT-4o to have the highest accuracy rate, although the differences were not statistically significant; DeepSeek R1 and Claude Sonnet 3.7 were reported as more reliable than Gemini Advance [12].

In the present study, numerical variability in diagnostic performance was observed across LLMs. While DeepSeek R1 showed higher mean diagnostic scores in the current analysis, these findings differed from those of Sezer et al. [12], who reported higher accuracy for ChatGPT-4o. The differences observed between results may be explained by the rapid and continuous evolution of LLMs, which can lead to substantial performance variations even within the same LLM family. Furthermore, similar to Taraç et al.’s [17] results, Claude showed the lowest diagnostic accuracy, a finding that contrasts with the conclusions of Johnson et al. [19], who identified Claude AI as the most reliable model. Gemini, consistent with most studies except that of Taraç et al. [17], generally performed at a comparatively lower level across diagnostic evaluations. Although no statistically significant differences were found between the LLMs in this study, certain numerical trends—specifically, higher scores for DeepSeek R1 and lower scores for Claude Sonnet 3.7—are noteworthy. While these trends did not reach statistical significance, they may still have clinically meaningful implications. Accordingly, the null hypothesis (H₀) was accepted and the alternative hypothesis (H₁) rejected.

DeepSeek R1 generally succeeded in identifying the correct diagnosis; however, it was unable to provide three differential diagnoses as requested. In contrast, other LLMs—particularly ChatGPT-4o—were able to generate three possibilities by employing different noun phrases with equivalent meanings. For instance, while the first diagnosis was stated as a complex crown fracture, the second was expressed as an enamel-dentin fracture with pulp exposure. This tendency to reformulate diagnostic terminology may partially explain ChatGPT-4o’s higher diagnostic score compared with Claude Sonnet 3.7 and Gemini Advance. When prompted to list three diagnostic possibilities, Claude Sonnet 3.7 typically presented its predictions clearly, but did not provide the underlying rationale. Consequently, its score decreased in cases where explanatory reasoning was required. However, unlike ChatGPT-4o, it tended to avoid producing alternative labels for the same diagnosis. A further limitation observed in Claude Sonnet 3.7 was its occasional substitution of symptoms for diagnoses in its secondary and tertiary predictions (e.g., providing “aesthetic concerns” as a diagnosis).

The scenarios presented in this study were formulated using terminology appropriate for dental professionals. Accordingly, ChatGPT-4o, DeepSeek R1, and Claude Sonnet 3.7 interpreted the user as a clinician and responded in a professional register. Gemini Advance, however, consistently interpreted the user as a patient and produced responses using more lay-appropriate terminology. Distinct from the other LLMs, Gemini Advance also incorporated elements of psychological reassurance in its answers—a strategy that may be beneficial in reducing parental anxiety. This finding aligns with the observations reported by Taraç et al. [17] regarding LLM use among parents concerned about dental trauma.

There are currently no studies in the literature aside from providing basic oral-health guidance; no researchers have assessed the capacity of LLMs to support or formulate dental trauma treatment plans [24]. When treatment quality was evaluated, numerical trends in mean scores were observed, with DeepSeek R1 showing higher mean treatment scores, followed by Claude Sonnet 3.7, Gemini Advance, and ChatGPT-4o. DeepSeek R1 generally produced coherent and clinically appropriate treatment suggestions. Claude Sonnet 3.7 tended to articulate its management strategies in a sequential manner across the three proposed diagnostic possibilities, which contributed to its relatively high treatment-quality score. In contrast, ChatGPT-4o typically provided a detailed explanation only for the primary diagnosis and did not elaborate on management options for the alternative differential diagnoses.

Another noteworthy finding of this study was the tendency of LLMs to generate terminology that is not included in established classifications of traumatic dental injuries and is absent from the scientific literature. Examples of such unsupported terms include pulp contusion–concussion, partial avulsion, crown contusion, tooth infusion, and fixed subluxation. The use of diagnostic terminology that is not part of internationally accepted classifications may lead to confusion, particularly within referral chains and multidisciplinary clinical settings. Furthermore, misinterpretation of the diagnosis may result in inappropriate or delayed treatment.

When reference reliability was evaluated, DeepSeek R1 demonstrated the highest reliability in reference generation, with 84.78% of its citations corresponding to real sources, whereas Claude Sonnet 3.7 exhibited the lowest reference reliability among the evaluated LLMs. In contrast, Gemini Advance provided the fewest references and frequently avoided citing sources by stating that its responses were based on “clinical experience”. Reference fabrication most commonly involved incorrect combinations of existing bibliographic elements or the inclusion of nonacademic sources.

In clinical practice, reliance on non-verifiable sources may lead to misinformation and contribute to the normalization of inappropriate clinical practices, raising serious ethical concerns related to accountability and responsibility. These findings highlight the need for retrieval-augmented generation approaches integrated with authoritative medical databases in the clinical application of LLMs.

Taken together, the generation of nonexistent terminology and the widespread use of inaccurate or fabricated references highlight critical weaknesses in current AI LLMs. These findings underscore the need for further development to improve the factual accuracy, reference integrity, and terminological reliability of AI systems intended for use in dental trauma guidance.

5. Limitations of the study

The present study approached treatment evaluation in a diagnosis-dependent manner. From this perspective, the treatment of misdiagnosis received a low score. The fundamental reason is that, in clinical practice, a “correct” treatment based on an “incorrect” diagnosis is still regarded as clinically inappropriate for the patients. Especially in emergencies requiring careful management, such as dental trauma, there is no meaningful equivalent to a correct treatment plan based on an incorrect diagnosis. However, future researchers that evaluate treatment content as an independent factor, separate from diagnosis, may help determine whether LLMs can support the treatment process in clinical settings where a definitive diagnosis can be established.

At the same time, objectively evaluating treatment stages in this study was challenging because treatment decisions in real-life clinical practice are shaped by the clinician’s judgment. These decisions may vary depending on factors such as the level of the child and parents’ cooperation and of oral hygiene’s adequacy. Because these variables were excluded and only the IADT guidelines were followed, the responses may not fully reflect real-life clinical scenarios.

Treatment scores were derived from diagnostic decisions, rather than from an independent treatment decision-making process; therefore, temporal reliability was assessed solely based on diagnosis. Furthermore, due to the large number of questions and LLM evaluated, all questions were administered within a two-day period. Two days represent the minimum duration required for temporal reliability assessment [25].

Although no statistically significant differences were found between the LLMs in this study, certain numerical trends were observed. These trends may be related to differences in the LLMs’ underlying model architectures. The lack of reported effect sizes and power analyses limits the clinical interpretation of these numerical differences. Future studies incorporating these analyses may provide a more robust understanding of such trends’ potential clinical relevance.

6. Conclusions

LLMs’ use of nonstandard terminology, the generation of unreliable or fabricated references, potential disruptions in the diagnosis–treatment continuum, and underlying algorithms’ limited transparency pose significant ethical risks related to accountability and clinical responsibility. Therefore, until LLMs are capable of consistently adhering to international clinical guidelines and ensuring verifiable source reliability, they should be regarded solely as supportive tools, rather than clinical decision-makers. In the future, AI-based applications that successfully address these limitations may enable dentists worldwide to access standardized, professional, and evidence-based information on a global scale.

Availability of data and materials

The datasets generated and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Author contributions

DS—prepared the main draft of the study; did the main writing of the article. DS and BY—worked together to create the data. ZG and ZUG—evaluated and written statistical analysis; performed the final review of the manuscript.

Ethics approval and consent to participate

Since the study did not use human data, ethical approval was not required.

Acknowledgment

The authors would like to thank Hasan Gündoğar and Ömer Faruk Kaygısız for their contributions to this study.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at https://oss.jocpd.com/files/article/2049764474418544640/attachment/Supplementary%20material.docx.