Bridging the information gap in pediatric dentistry: a comparison of ChatGPT-4o, Google Gemini Advanced, and expert responses based on evaluations by parents and pediatric dentists

1. Introduction

Artificial Intelligence (AI) refers to computer systems capable of performing tasks that normally require human intelligence. Studies in this field aim to develop intelligent machines that can simulate human-like thinking and decision-making processes, combining disciplines such as computer science, mathematics and psychology. AI systems can analyze large data sets, recognize patterns, make predictions, and improve what they have learned over time. Additionally, through technologies such as natural language processing, speech recognition and computer vision, they also have the ability to interact with humans [1].

A subfield of AI, Large Language Models (LLMs), contributes to the provision of more efficient and accessible healthcare services in fields such as medicine and dentistry [2]. LLMs are neural network-based systems trained on large text datasets such as Wikipedia, digital books, scientific articles and web pages [3]. Using deep learning algorithms, they aim to generate meaningful, coherent, and human-like responses based on a given text [4]. Unlike traditional search engines, they save time by presenting information directly in text form, sparing users from browsing through various sources, and offering a user-friendly experience [5].

One of the most well-known models in this field, ChatGPT, was first publicly released by OpenAI at the end of 2022 (version 3.5, San Francisco, CA, USA) and quickly reached millions of users. Subsequently, ChatGPT-4 was introduced in 2023, followed by the GPT-4o model, which can also process visual inputs, in 2024 [6]. ChatGPT stands out for its easy accessibility, user-friendly interface and creative, fluent responses [7]. Similarly, Google introduced Bard based on Language Model for Dialogue Applications (LaMDA), followed by the Gemini model based on Pathways Language Model (PaLM) 2 in February 2024. Gemini is notable for its multimodal structure that can analyze text, audio and visual content simultaneously, and it has reached a broad user base due to its feature of providing free and unlimited interaction [8]. While Gemini stands out with its information density and comprehensive responses, ChatGPT models distinguish themselves with fluency and creative content generation [9]. Both models are developed through interactive learning methods based on user feedback [10].

The implementation of these large language models (LLMs), particularly in the form of chatbots, has also introduced various potential applications in specialized fields such as pediatric dentistry. In recent years, AI-powered chatbots have begun to be used in areas such as oral health education and counseling for children. For example, a chatbot named COSC (Chatbot for Oral Self-Care) developed in Taiwan aimed to improve the tooth brushing habits of children aged 6–12. In a pilot study, the usability score of this chatbot was reported as 79.91 out of 100, and user satisfaction was found to be quite high [11]. In another study conducted in the United States, the responses of ChatGPT and similar chatbots to 30 true/false questions related to pediatric dentistry were compared with those of expert dentists and students. In this study, pediatric dentists achieved an accuracy rate of 96.7%, while ChatGPT performed the best among the chatbots with a 78% accuracy rate. However, it was emphasized that chatbots cannot yet replace human clinicians in clinical decision-making processes [12]. In another study conducted in South Korea, the responses of ChatGPT-3.5 and Gemini chatbots to national exam questions related to pediatric dentistry were evaluated. Although both models had similar accuracy rates, they failed to reach the minimum score required to pass the exam. These results indicate that while chatbots may be useful in educational and counseling contexts, they still have significant limitations in clinical decision-making processes [13].

In order to better understand the potential contributions and current limitations of AI-supported chatbots in pediatric dentistry, this study aims to compare the information provided by two different AI-supported chatbots—ChatGPT-4o and Google Gemini Advanced—in response to frequently asked questions by parents in the field of pediatric dentistry, with the responses given by pediatric dentistry specialists. The main objective of the study is to evaluate the extent to which the information provided by these AI systems aligns with expert opinions in terms of accuracy and clinical adequacy, and to analyze the parents’ perception of the adequacy of these responses.

2. Material and methods

Before initiating the study, approval was obtained from the Afyonkarahisar Health Sciences University Scientific Research Ethics Committee (ID: 2025/1). The study was conducted in accordance with the principles outlined in the Declaration of Helsinki [14]. Written and verbal informed consent was obtained from the parents and pediatric dentists. Inclusion criteria required that parents had at least a high school education. Exclusion criteria included parents who were primary or secondary school graduates or illiterate, those whose children had not received treatment in our pediatric dentistry clinic, and those who did not volunteer to participate in the study. In the first stage, on 20 January 2025, a total of 57 questions were selected from the “Questions Frequently Asked by Parents” section, categorized by age groups, on the website of the International Association of Paediatric Dentistry (IAPD) (https://iapdworld.org/parents/). Twenty pediatric dentists working in our clinic independently reviewed these questions and were asked to identify the ones most frequently asked by parents based on their clinical experience. Based on the frequency of selections made by the experts, a list of the 32 most frequently chosen questions was created. To ensure both representativeness and impartiality, these 32 questions were sequentially numbered and randomly ordered using an online tool (https://www.random.org/lists/). The first 15 questions resulting from this process were selected for evaluation in the study (Supplementary material).

Three different response options were prepared for each selected question. For the first response option, the original answers available on the IAPD website were evaluated on 27 January 2025, by two pediatric dentists and one orthodontist, each with at least five years of clinical experience. The experts were asked to assess the accuracy and adequacy of these responses and to make revisions where necessary. The answers finalized through this evaluation process were accepted as expert responses.

For the second and third response options, the same questions were submitted without any modifications on 28 January 2025, to two different AI-based chatbot systems: ChatGPT-4o (https://chat.openai.com/) and Google Gemini Advanced (https://gemini.google.com/app?hl=tr). The three responses obtained for each question (provided by the experts, ChatGPT-4o and Google Gemini Advanced) were randomly assigned to the questions using the online randomization service offered by Random.org (Supplementary material).

Before the questions were included in the survey, the Flesch-Kincaid reading ease test was administered to assess the readability of all responses obtained from the three different sources and to investigate the impact of readability on the participants’ evaluations. This test was used to determine the understandability of the responses and how their complexity level affected the evaluation process. After the necessary adjustments were made, the survey was created on the Google Forms platform. The survey consisted of 45 scoring questions (15 questions, each with three different responses) and four demographic questions. Participants were asked to rate the adequacy of the three different responses provided for each question on a scale from 1 (less adequate) to 10 (very adequate). The survey and informed consent form were sent to participants via email and WhatsApp Messenger (v2.24.2.10, WhatsApp Inc, Menlo Park, CA, USA). The sample size was calculated using G*Power software (version 3.0.10, Franz Faul, Christian-Albrechts-Universität, Kiel, SH, Germany), which indicated that a minimum of 111 participants was required (effect size = 0.3; significance level = 0.05; power = 0.90).

All statistical procedures were performed using IBM SPSS software (version 27.0.1.0, Armonk, NY, USA). In the first stage, Cronbach’s alpha coefficient was calculated to assess the internal consistency of the survey, and reliability was tested. Next, the Jarque-Bera test was applied to determine whether the data followed a normal distribution. Descriptive statistics, including mean and standard deviation, were calculated for each variable. For normally distributed data, one-way analysis of variance (ANOVA) was used to assess differences between groups, followed by pairwise comparisons using the Bonferroni post-hoc test; corrected p-values were reported in this analysis. For non-normally distributed data, group comparisons were made using the Kruskal-Wallis test, with pairwise comparisons examined using the Tamhane post-hoc test. In pairwise comparisons, the Independent Samples T-Test was used for normally distributed data, while the Mann-Whitney U test was used for non-normally distributed data. A statistical significance level of p < 0.05 was set. Additionally, the Flesch-Kincaid readability test was applied to evaluate the readability level of the texts.

3. Results

In this study, the findings obtained from the survey, in which patient guardians and pediatric dentists evaluated the responses provided by the specialist, ChatGPT-4o and Google Gemini Advanced, are presented below. A total of 148 individuals participated in the study, consisting of 47 pediatric dentists and 101 parents. The mean age of the pediatric dentists was 30.14 ± 3.65 years, while the mean age of the parents was 26.88 ± 6.49 years. The Cronbach’s alpha reliability test yielded a value of 0.957, indicating excellent internal consistency. The results of the Flesch-Kincaid readability test were 57.11 ± 15.42 for expert responses, 53.26 ± 17.25 for ChatGPT-4o responses and 53.59 ± 18.81 for Google Gemini Advanced responses.

The results of the statistical analysis regarding the evaluation of pediatric dentists’ responses by specialists, ChatGPT-4o and Google Gemini Advanced are presented in Table 1. To provide a visual overview of the adequacy scores given by pediatric dentists, a bar chart comparing the mean ratings for expert, ChatGPT-4o and Google Gemini Advanced responses across all 15 questions is presented in Fig. 1. It was observed that pediatric dentists predominantly rated the specialist responses higher compared to the responses from ChatGPT-4o and Google Gemini Advanced. Specifically, for questions such as Q1 (7.64 ± 2.09, p < 0.001) and Q5 (8.81 ± 1.17, p = 0.001), specialist responses received significantly higher scores, while ChatGPT-4o responses (Q1: 5.72 ± 2.36, Q5: 7.39 ± 2.02) received the lowest ratings. Additionally, no significant differences were observed between the groups for Q3 and Q10.

Fig. 1.Pediatric dentists’ evaluation of responses.

The evaluations of patient guardians regarding the responses from the specialist, ChatGPT-4o and Google Gemini Advanced are presented in Table 2. To further illustrate how patient guardians evaluated the responses from each source, Fig. 2 presents a comparative bar chart showing the mean adequacy scores for the expert, ChatGPT-4o and Google Gemini Advanced responses across all 15 questions. According to the data, it was found that patient guardians generally preferred the expert responses. However, in some instances, particularly for certain questions, the answers from ChatGPT-4o and Google Gemini Advanced received scores that were either close to or even higher than those of the expert responses. For question Q2, the expert answers (6.06 ± 3.01) received the lowest score, while ChatGPT-4o (7.09 ± 2.89) and Google Gemini Advanced (6.91 ± 2.39) were rated higher (p = 0.021). Conversely, for questions such as Q6 (p = 0.008) and Q13 (p = 0.003), expert answers were rated higher than those from the artificial intelligence models.

Fig. 2.Patients parents’ evaluation of responses.

When comparing the scores given by pediatric dentists and patient guardians, significant differences were observed for some questions. As shown in Table 3, pediatric dentists rated the expert answers significantly higher than patient guardians, particularly for Q2 (7.50 ± 2.31 vs. 6.06 ± 3.01) and Q12 (9.03 ± 1.03 vs. 8.38 ± 1.90). On the other hand, in some cases, patient guardians provided more favorable evaluations than pediatric dentists, such as for the ChatGPT-4o answers to Q1 (7.07 ± 2.47) (p = 0.005). This finding suggests that while expert answers were generally rated highly by patient guardians, artificial intelligence responses provided acceptable information in certain instances and were rated more positively than expected by patient guardians. Tables 4 and 5 present the results for the evaluation of ChatGPT-4o and Google Gemini Advanced responses by pediatricians and patient guardians. Pediatric dentists rated ChatGPT-4o responses lower for some questions (Q1, Q9, Q10), whereas patient guardians gave these responses higher scores. Similarly, while pediatric dentists Google Gemini Advanced responses are more favorably rated for certain questions, patient guardians are rated Gemini Advanced responses higher for others (Q2, Q10). In conclusion, the study indicates that while pediatricians generally consider expert responses to be more competent, AI-based systems’ responses were also positively received by patient guardians for specific questions, such as Q2.

4. Discussion

In the digitalized world, people are increasingly trying to obtain information about health issues from the internet through written or video content [15]. Although this may seem appealing to patients, many researchers continue to investigate how much both textual and video content overlap with expert opinions [16, 17]. With the recent spread of artificial intelligence-supported chatbots, patients have begun to turn to these chatbots for questions about their health issues [11, 18]. For this reason, researchers from many branches and dentistry have begun to conduct studies measuring the adequacy levels of chatbots in health-related questions [19, 20]. However, to our knowledge, there is a limited number of studies investigating the adequacy of current chatbots in pediatric dentistry [12, 21]. Therefore, the aim of this study was to compare the answers given by two current chatbots, ChatGPT-4o and Google Gemini Advanced, to questions frequently asked by parents in pediatric dentistry with the answers given by specialist doctors to the same questions. The aim of this study is to fill a gap in the pediatric dentistry literature by investigating the adequacy and accuracy of artificial intelligence-supported chatbots in the field of pediatric dentistry.

The current study has highlighted both the potential and limitations of rapidly evolving AI-supported chatbot technology in specialized fields such as pediatric dentistry. In some cases, no significant difference was found between expert opinions and chatbot responses, indicating that chatbots have the potential to provide answers similar to those of experts for certain questions. However, this competence was not consistent across all questions. This finding underscores the limitation that chatbot responses may not always offer satisfactory answers for every query. We suggest that to improve the accuracy of these AI-based applications, it is essential for them to gain more clinical experience from seasoned physicians and to receive feedback from patients regarding the disease and treatments being addressed.

The findings of our study revealed that pediatric dentists consistently rated expert responses higher, whereas parents tended to evaluate the responses provided by AI-supported chatbots more positively. A possible explanation for this difference is that large language models such as ChatGPT-4o and Google Gemini Advanced are designed to present information in a simplified, clear, and accessible manner in order to optimize user experience [22]. The linguistically simplified and easily understandable nature of AI-generated responses may have led parents—who lack specialized knowledge—to assign higher scores. On the other hand, the lower ratings given by pediatric dentists to the responses provided by ChatGPT-4o and Google Gemini Advanced, compared to expert answers, may stem from perceived deficiencies in content accuracy. In a study conducted in Türkiye by Şişmanoğlu and colleagues, the answers given by ChatGPT-4.0 and Gemini Advanced to questions from the Dentistry Specialty Exam (DUS) were analyzed. Although both chatbots achieved scores sufficient to pass the exam, they were found to fall short, particularly in areas requiring clinical knowledge [23]. This suggests that participants with expert knowledge are more likely to identify information gaps in AI-generated responses and, therefore, adopt a more critical perspective in their evaluations.

Another notable finding is that the adequacy scores of the responses varied depending on the type of chatbot. For instance, ChatGPT-4o received higher scores for some questions, while Google Gemini Advanced scored higher for others. This highlights the unpredictability of which chatbot will provide a more adequate response to a given question. The variation in adequacy scores between the two chatbots may stem from differences in their algorithms for data processing and text generation. A study by Qi et al. [3] comparing ChatGPT-4V and Google Gemini, identified these differences and concluded that each model has its own unique strengths and specialized niches. In a similar study, Omar and colleagues reported comparable results [24]. On the other hand, Koç and Tiryaki found that the performances of Google Gemini Advanced and ChatGPT-4 were quite similar in their study. However, the researchers compared the two models specifically in the context of medical imaging. The authors attributed this similarity to the fact that ChatGPT is often more understandable and user-friendly in certain scenarios, such as data presentation, while Google Gemini is better equipped to analyze larger data sets and provide more detailed information [25].

The literature presents varied findings regarding the performance of AI-supported chatbots in healthcare applications. For instance, Schmidt et al. [26] reported that ChatGPT-4 can serve as an effective support tool for preoperative patient education, providing comprehensive and readable information to patients considering penile prosthesis implantation. Similarly, in the study conducted by Reyhan et al. [27], which evaluated the reliability of answers provided by five different chatbots to questions about keratoconus, the researchers found that while all models produced satisfactory results, Google Gemini and Microsoft Copilot offered more reliable and higher-quality answers.

Huo et al. [20] evaluated the performance of four different chatbots (ChatGPT-4, Copilot, Google Bard and Perplexity) in the context of surgical management for gastroesophageal reflux disease. Their findings indicated that Google Bard and ChatGPT-4 achieved high accuracy rates, while Microsoft Copilot and Perplexity showed low accuracy rates [20]. Similarly, Lim et al. [19] assessed the accuracy of ChatGPT-3.5, Claude, Gemini, and CoPilot in providing preoperative recommendations for abdominoplasty. They found that Claude and ChatGPT-3.5 demonstrated high reliability, while Microsoft Copilot and Google Gemini exhibited lower reliability [19].

In the field of dentistry, similar inconsistencies in research results are observed. For example, Avşar and Ertan reported that both ChatGPT-3.5 and Google Gemini had limited knowledge regarding prosthetic dental treatments, and both applications yielded similar results [28]. In a study by Guven et al. [21] which compared the competencies of three different chatbots (ChatGPT-3.5, ChatGPT-4.0 and Google Gemini) regarding traumatic dental injuries, it was found that ChatGPT-3.5 provided misleading and inaccurate answers, whereas ChatGPT-4.0 and Google Gemini offered more accurate and comprehensive responses. The authors attributed these differences to architectural variations in the chatbots’ algorithms and the scope and size of their training data. In contrast, a pilot study by Rokhshad et al. [12] compared the responses of nine different chatbots (Google Bard, ChatGPT-4, ChatGPT-3.5, Llama, Sage, Claude 2 100k, Claude-instant, Claude-instant-100k and Google Palm) in the context of pediatric dentistry, concluding that the use of chatbots in clinical pediatric dentistry may not be recommended. Previous studies focused mainly on expert evaluations, but the satisfaction of patients regarding the responses was not explored. This study, however, sought to provide a comprehensive analysis by including both expert evaluations and patient opinions.

When examining studies conducted in the fields of medicine and dentistry, it becomes evident that there is no clear consensus on the reliability of chatbots in healthcare. Different chatbots have demonstrated varying levels of success across different specialties. For instance, ChatGPT provided sufficiently reliable answers in the context of abdominoplasty but was found to be less competent when it came to the topic of dental prostheses [19, 28].

In the long term, chatbots have the potential to reach higher levels of accuracy as they continue to train and update themselves. This highlights their future capacity to provide more reliable answers in the healthcare field. Another potential solution for improving chatbot reliability and satisfaction could be the development of specialized chatbots designed for specific tasks, similar to how doctors specialize in particular fields of medicine. For example, Pupong and colleagues developed a chatbot named FunDee, which provides oral health care services. They demonstrated that this chatbot achieved high user satisfaction in oral health education [29].

Another important factor in studies evaluating chatbot performance is readability and understandability. The Flesch test was used to determine reading ease, while the Flesch-Kincaid test assessed reading level. The Flesch score ranges from 0 to 100, with higher scores indicating easier readability [30]. Nahir claimed that the readability of GPT responses in pediatric dentistry was low [31]. However, according to the current study, the readability levels of responses provided by chatbots were similar to those of expert opinions. This similarity suggests that the participants’ ability to understand the responses was comparable across the different sources. As a result, the participants seemed to be more influenced by the quality, accuracy, and persuasiveness of the content when evaluating the responses. On the other hand, Nahir’s study was based on ChatGPT-3.5. We believe that the discrepancy between the results can be attributed to the improved readability and understandability of responses in ChatGPT-4o compared to GPT-3.5. Additionally, AI-supported chatbots have the potential to enhance the clarity of their responses over time as they continue to evolve and improve.

In general, the results indicate that improving the performance and accuracy of chatbots requires optimizing the questions through prompt engineering methods. However, in real-world conditions, users typically ask their questions without any optimization. To better reflect real-world scenarios, no prompt optimization was applied in this study.

This study has several limitations. First, the evaluation was restricted to 15 questions frequently asked by parents, as listed on the IAPD website. While these questions reflect common parental concerns addressed to pediatric dentists, this narrow scope may limit the applicability of the findings to broader or more diverse dental care settings. Additionally, only two chatbot models—ChatGPT-4o and Google Gemini Advanced—were assessed, excluding other current or emerging AI-based chatbot systems that might perform differently under similar conditions. Another limitation lies in the absence of prompt optimization. Although this approach was deliberately chosen to replicate real-life usage, it may have hindered the chatbots from demonstrating their full potential. Future studies should aim to cover a wider range of dental topics, include larger and more diverse participant samples, evaluate various chatbot platforms, and adopt a mixed-methods approach combining both quantitative and qualitative data for a more comprehensive analysis.

5. Conclusions

The concordance between expert opinions and artificial intelligence (AI)-generated responses to questions related to pediatric dentistry and orthodontics remains unpredictable. While pediatric dentists generally find expert opinions to be more satisfactory, the level of satisfaction with AI-generated responses among parents varies depending on the specific question. The fact that the scores for all three response sources are relatively close suggests that AI has promising potential to provide satisfactory answers in the long term. There is a need for more advanced studies that include a larger number of participants, address a broader range of topics, and compare the outputs of multiple AI-based chatbots using a wider set of questions.

Availability of data and materials

Data and materials are available at the Pediatric Department in the Faculty of Dentistry, Afyonkarahisar Health Sciences University.

Author contributions

İHÇ and FS—conceived the study. İHÇ—designed the research process, collected and organized the data. FS—performed the statistical analysis and interpreted the results. HC—conducted the literature review and contributed to the manuscript editing. All three authors collaboratively wrote the manuscript, contributed to editorial revisions, and approved the final version of the manuscript.

Ethics approval and consent to participate

The research protocol was reviewed and approved by the Afyonkarahisar Health Sciences University Scientific Research Ethics Committee prior to the initiation of the study (Approval ID: 2025/1). Written and verbal informed consent was obtained from the parents and pediatric dentists.

Acknowledgment

The authors would like to thank the Pediatric Dentistry Department of the Faculty of Dentistry, Afyonkarahisar Health Sciences University, for their support in facilitating data access and clinical coordination throughout the study.

Funding

This research received no external funding.

Conflict of interest

The authors declare no conflict of interest.

Supplementary material

Supplementary material associated with this article can be found, in the online version, at https://oss.jocpd.com/files/article/2006258298729185280/attachment/Supplementary%20material.docx.