Commercial artificial intelligence lateral cephalometric analysis: part 2—effects of human examiners on artificial intelligence performance, a pilot study

1. Introduction

Automatic cephalometric landmark detection has attracted considerable research attention in dentistry [1]. With the introduction of artificial intelligence (AI) in this field in the past few years, the success rate of automated landmark identification is rapidly increasing [2, 3, 4, 5, 6, 7, 8, 9]. Moreover, commercial AI-supported automatic cephalometric services have recently been launched. Accordingly, the accuracy and efficiency of cephalometric analysis are expected to improve considerably [1].

The primary clinical efficacy expected from automatic landmark identification is a decrease in the landmarking error and the required time by eliminating human examiner intervention [10, 11]. However, as indicated in our previous study (Part 1), which assessed the performance of commercial AI cephalometric services, an inspection of each landmark position by a human examiner was an indispensable requisite for commercial AI services. In other words, a human examiner could not be excluded from these commercial AI-supported cephalometric analyses for now.

The question is what benefits can be gained from AI-supported cephalometric analysis when accompanied by a human examiner’s review. Whether time-saving and error-reduction effects can be expected even if less-experienced examiners, such as beginners in pediatric dentistry and orthodontic fields, review the AI results is unclear. Thus far, the mainstream of AI research has been to evaluate the performance of AI architectures [2, 3, 4, 5, 6, 7, 8, 9]. The effect of a human examiner’s expertise on the final results of the AI analysis has not yet been investigated. The effectiveness of these AI services under practical usage conditions is clinically relevant for clinicians because several commercial AI cephalometric services have already been released to the market with an increasing number of users.

This study aimed to evaluate the clinical efficacy of a commercial AI cephalometric analysis followed by a human examiner’s inspection. The time-saving effect, agreement and reliability of AI-assisted cephalometric analysis will be investigated among experienced and inexperienced human examiners, AI and combinations thereof.

2. Materials and methods

Eighty-four pretreatment cephalogram images were randomly selected from patients who visited the Department of Orthodontics and Pediatric Dentistry of Kyungpook National University Dental Hospital between 2012 and 2021 for the treatment of malocclusion. Patients with a history of orthodontic treatment or craniofacial malformations such as cleft lip and palate were excluded. All cephalogram images, taken using a CX-90SP (an X-ray scanner, Asahi, Kyoto, Japan), were in JPG format with a resolution of 150 DPI and a gray level of 24. Table 1 presents the characteristics of the sample.

Three human examiners participated in this study. Experts 1 (HKN) and 2 (SRB) are board-certified orthodontists with more than seven and five years of clinical experience, respectively, belonging to regular examiners. The beginner examiner (JSL) is a board-certified pediatric dentist with minimal expertise in tracing cephalograms, i.e., less than 40 cases over the seven years of his career. Thirteen commonly used cephalometric variables were evaluated after identifying 15 dental and skeletal landmarks (Tables 2 and 3).

The manual landmarking process was as follows: First, human examiners discussed and agreed upon the landmark definition using three cephalogram images that were not included in the study samples. Then, the human examiners initiated manual landmark identification using computer software (6.3 Sequential Tracing Mode, AudaxCeph, Ljubljana, Slovenia). Detection was performed independently, without any communication between examiners. There was no time limit for landmarking, and re-examining landmark positions was always possible until the examiners were satisfied. Expert 2 and the beginner examiner measured the time required from the first landmark identification to the final approval of the overall landmark position under this condition. Repeated measurements were performed by expert 1 after one month.

Commercial AI cephalometric service (WebCeph, 1.0.0, Assemblecircle, Gyeonggi-do, Korea) was used to obtain AI landmarking data. WebCeph automatically detected the landmark position and performed cephalometric analysis immediately after uploading the study samples. The landmark data from WebCeph were saved and delivered to expert 2 and the beginner examiner. Each examiner independently reviewed the positions of the 15 landmarks involved in this study. The review process was initiated 1 month after the previous manual detection. As in manual landmarking, each landmark position was examined and adjusted as necessary without any time limit. The examiners also measured the time required for the review process from the first landmark examination to the final approval.

The overall study design is shown in Fig. 1.

Fig. 1.Study flow chart. AI, artificial intelligence.

All statistical analyses were performed using the language R (4.3.1, R Foundation for Statistical Computing, Vienna, Austria) with a significance level of 0.05.

The intra-examiner reliability between the repeated measurements of expert 1 was evaluated with intraclass correlation coefficients (ICCs) using two-way mixed-effects, single rater and absolute agreement models [12]. The Dahlberg formula estimated method errors between repeated trials.

The time required for landmark identification was compared among expert 2, beginner, WebCeph and expert 2 and WebCeph and beginner. A one-way repeated-measures analysis of variance was used for comparison. Subsequent post hoc pairwise comparisons were performed using the Bonferroni correction.

The Bland-Altman analysis was used to evaluate measurement errors among examiners. Expert 1’s data were set as the reference. Then, according to the Bland-Altman protocol, data from other examiners were analyzed relative to this reference [13]. Specifically, the mean and difference between expert 1 and other examiners were calculated. Data normality was confirmed using the Shapiro-Wilk test, and the Bland-Altman statistics was evaluated. The bias, which is the mean difference between examiners, measures the systematic error. On the contrary, the limits of agreement (LoA) are the upper and lower bounds containing 95% of the measurement errors between examiners [14]. As an index of pure random error size between examiners, the maximum random error (MRE) was defined as the half-width of the upper and lower LoA, eliminating the effect of bias included in LoA [13, 15]. To help visually understand the magnitudes of systematic and random errors, the Bland-Altman plots were drawn.

The number of variables that satisfied the agreement criteria was counted for each examiner. We applied the same criteria for agreement determination based on the rationale described in our previous study (Part 1).

The inter-examiner reliability between expert 1 and other examiners was estimated with ICCs using the two-way random-effects and single-rater models [12]. We calculated both absolute agreement and consistency ICCs to assess the impact of systematic errors.

3. Results

The mean intra-examiner reliability of expert 1 was 0.97, and the mean method errors were 0.95° and 0.73 mm for angular and linear measurements, respectively (Table 3). Therefore, we took the mean of the first and second measurements and used these average values in the subsequent analysis.

Table 4 presents the time required for landmarking. The results of Mauchly’s test confirmed that the sphericity assumption was violated (p < 0.001). Hence, we report the test results adjusted by Greenhouse and Geisser (ϵ = 0.736). According to the analysis, at least one measurement time was significantly different: F (2.208, 218.592) = 229, p < 0.001. The subsequent post-hoc test showed significant differences in all comparison pairs (p < 0.001). Specifically, when comparing the required time between examiners according to the landmarking methods, expert 2 took significantly longer than the beginner for both manual landmarking and AI review. However, no consistency was observed when comparing each examiner’s required time between the landmarking methods. Expert 2 took approximately 30 s more for AI review than manual landmarking, whereas the beginner showed approximately 10 s shorter time in the same case.

The descriptive and Bland-Altman statistics of cephalometric variables measured by each examiner are presented in Tables 5 and 6. Statistically significant biases with confidence intervals not containing zero were common among examiners; however, their magnitudes were generally small. Exceptionally, the sella-nasion plane (SN plane)-associated variables (SNA, SNB, SN-MP, Björk-Jarabak sum, SN-U1 and SN-OcP) measured by the beginner showed relatively large systematic errors.

The magnitudes of random errors showed a clear pattern among examiners: expert 2 < WebCeph and expert 2 ≤ WebCeph and beginner < WebCeph < beginner (Fig. 2). The MRE of the beginner was 2.19 times larger than that of expert 2 on average. Similarly, WebCeph, WebCeph and beginner, and WebCeph and expert 2 showed 1.78, 1.35 and 1.18 times greater MREs than expert 2 (Table 6 and Fig. 2). As an illustrative example, the Bland-Altman plots of the SN-MP are shown in Fig. 3.

Fig. 2.Bar graphs of the magnitude of MREs. The horizontal solid line represents the acceptable clinical limit for random error.

Fig. 3.Bland-Altman plots for SN-MP. In each graph, the x-axis denotes the average of two examiners’ measurement outcomes, while the y-axis is the difference between the examiners. Green dotted lines represent the bias with a 95% confidence interval. Blue dotted lines stand for the upper and lower limit of agreement with a 95% confidence interval. (A) between expert 1 and expert 2; (B) between expert 1 and beginner; (C) between expert 1 and Webceph; (D) between expert 1 and Webceph which was revised by expert2; (E) between expert 1 and WebCeph which was revised by beginner. CI, 95% confidence interval; LoA, limit of agreement.

In this study, most of the examiners did not meet the agreement criteria (Table 7). Only WebCeph and expert 2 could satisfy the standard over three variables (Wits, IMPA and U1L1).

The mean inter-examiner reliability between expert 1 and expert 2 was 0.95 for absolute agreement and 0.96 for consistency (Table 8). Similar but slightly low ICC values were observed in WebCeph and expert 2 and WebCeph and beginner, i.e., 0.91–0.94, corresponding to excellent ICCs. By contrast, Webceph showed relatively compromised ICCs between 0.84 and 0.86. Finally, the beginner revealed the lowest ICC values of 0.78–0.82, with a large difference between absolute agreement and consistency.

4. Discussion

Time-saving may be one of the major clinical benefits that clinicians expect from using AI-supported cephalometric analysis [10, 11]. However, contrary to usual expectations, the statistical comparison of the elapsed time showed no consistent trend among examiners, i.e., that the beginner could save approximately 10 s on average, whereas expert 2 spent 30 s longer (Table 4). This is probably because the landmark position review process may negate the time saved by AI and can take more in some cases. In addition, regardless of whether it increased or decreased, the time difference between the measurement methods was <1 min, which was clinically insignificant. Therefore, contrary to the usual expectations, obtaining a clinically relevant time-saving effect from AI cephalometric analysis may be challenging as long as the examination by a human examiner is essential.

The overall systematic error was not clinically significant. However, the SN plane-associated variables measured by the beginner showed substantial systematic errors (Table 6). This is an example of a subjective error that has long been reported as a major error source for landmark detection [8, 10, 16]. Interestingly, this systematic error disappeared in WebCeph and beginner (Table 6, Fig. 3), supporting that subjective error reduction is achievable even after a human review.

In Part 1, we revealed that the random error size of WebCeph was clinically unacceptable. A human examiner’s review has been suggested as a countermeasure against the unreliable performance of AI [17, 18]. The study results showed that the MRE of WebCeph decreased when reviewed by a human examiner, regardless of their expertise level (Table 6, Fig. 2). These results may justify AI review by a human examiner as a method compensating for AI’s performance.

However, these advantages of AI review were not equally applicable to examiners with different expertise levels. Compared with manual landmarking, the average MRE decreased by 2.63 for beginners with AI aid, whereas expert 2 conversely increased by 0.57, albeit slightly (Table 6, Fig. 2). The number of variables meeting the agreement condition also illustrated the same points. The performance of expert 2 became worse with the help of Webceph (Table 7).

The ICC values could confirm the aforementioned study findings again. The inter-examiner reliability was calculated to evaluate the effect of the reduced measurement error on discriminating individuals in the study group (Table 8) [12]. As in the previous study, both absolute agreement and consistency ICC were measured. No considerable difference was found between the two ICCs, which further supported the previous decision that systematic error was not clinically relevant. Moreover, the beginner’s reliability could be improved to a level close to that of expert 2 with the help of AI. However, expert 2 showed a slight decrease in reliability when assisted by AI. These conflicting reliability results between the beginner and expert 2 may imply that AI-aided cephalometric analysis cannot be practical for all human examiners.

Overall, adding a human review to AI could significantly reduce measurement errors, which led to an apparent improvement in agreement and reliability compared with using AI alone in cephalometric analysis. However, in terms of superiority to manual landmarking, conflicting results were noted according to the examiner’s expertise. When using AI together, less-experienced examiners could obtain better results than manual alone, whereas regular examiners obtained slightly worse results than manual landmarking. Considering that the time-saving effect is not evident when using AI, AI-supported commercial cephalometric analysis may be more recommendable for less-experienced examiners such as general practitioners or a beginner in the pediatric dentistry or orthodontic field.

The reason for these conflicting results is unclear. As one hypothesis, expert 2 and the beginner might have made AI-dependent decisions during the landmark review. WebCeph’s performance is approximately halfway between the beginner and expert 2 (Fig. 2). Thus, WebCeph’s landmarking may have suggested a valuable guide for the beginner, whereas it may have hindered expert 2. For accurate verification, analysis at the landmark coordinate level may be required; however, this goes beyond the scope of this study. Further research may be needed.

As another limitation, the number of examiners participating was small due to its pilot nature. Consequently, various examiner factors may have affected the study results. If a sufficient number of qualified examiners had participated, these examiner-origin biases would have been controlled. The reference positions of the landmarks could have also set closer to the ground truth. Further studies with more examiners may be needed.

5. Conclusions

In this study, the clinical effectiveness of commercially available AI-supported cephalometric services was evaluated under the assumption of a mandatory human examiner review. Within the limitations of this study, the following conclusions were drawn:

1. Contrary to expectations, the time-saving effect was not evident when using AI services together with an examiner’s review.

2. AI measurement errors generally decreased when reviewed by a human examiner. However, the actual benefits were limited to beginners. The regular examiner’s error slightly increased more than manual landmarking when using AI.

3. Overall, commercial AI-supported cephalometric services are recommendable as an aid for less-experienced examiners or clinicians.

Abbreviations

AI, artificial intelligence; ICC, intraclass correlation coefficient; LoA, limit of agreement; MRE, the maximum random error.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author contributions

HKN, SRB and JSL—collected and analyzed experimental data and drafted the manuscript. HKN—designed the research and critically supervised the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This study was approved by the Institutional Review Board of Kyungpook National University Dental Hospital (KNUDH-2022-05-02-00). Patient informed consent was waived due to the study’s retrospective nature.

Acknowledgment

Thanks to Kyungpook National University Dental Hospital Dental Research Institute for support (2022).

Funding

This work was supported by Kyungpook National University Dental Hospital Institute for Dental Research (2022).

Conflict of interest

The authors declare no conflict of interest.