Commercial artificial intelligence lateral cephalometric analysis: part 1—the possibility of replacing manual landmarking with artificial intelligence service

1. Introduction

Lateral cephalometrics is a useful diagnostic tool for assessing growth in children and adolescents. Specifically, if a maxillofacial deformity is present, lateral cephalometric evaluation is essential before orthopedic treatments. Despite this clinical importance, accurate landmarking has been constantly raised as difficult, and it has become a great entry barrier, especially for beginners in pediatric dentistry or orthodontic fields. Computer-aided landmarking has been consistently attempted; however, no significant results in accuracy and reliability have been achieved [1].

Recently, deep learning-based artificial intelligence (AI) has been introduced for dental imaging. In many studies published after 2017, the Euclidean distance (point-point distance) between landmarks detected by AI and the human examiner was <2 mm in more than 80% of cases [2, 3, 4, 5, 6, 7]. Considering that the landmarking error between human examiners is approximately 1.5 mm [8], a dramatic improvement has undoubtedly been achieved in the accuracy of AI-assisted cephalometric landmark detection.

However, whether this technology is clinically applicable should be cautiously determined. For example, many studies have reported that the prediction performance of AI is degraded when evaluated with a test set whose source is different from that of the AI’s learning set. This phenomenon, known as the lack of generalizability, is due to the difference in image qualities between learning and test data [2, 9, 10, 11]. Given that the source of cephalogram images can vary in actual clinical settings, this can be a significant issue. Furthermore, despite AI’s improved performance, some landmarks showed different directional patterns in the scatter plot between AI and a human examiner [3, 5]. For instance, the landmark of the lower incisor edge was within a 2 mm error range for both human examiners and AI. However, while the human examiners showed an isotropic (circular) distribution, the AI showed an anisotropic (elliptic) pattern.

Considering the aforementioned issues, there is still a possibility that cephalometric variables measured by AI and human examiners may differ significantly. Nevertheless, various AI-assisted automatic cephalometric analysis services are commercially available [11]. Although some studies have already evaluated the performance of commercial AI, these studies have shown limitations such as insufficient sample size or evaluation of only a limited number of AIs [10, 12, 13]. The accurate and objective performance evaluation of AI cephalometric services that are commercially available is still an unresolved issue for clinicians.

Thus, this study aimed to assess the performance of four commercial AI-assisted automatic cephalometric services using cephalometric variables. The agreement and reliability of AI were compared with those of human examiners. From this, we determined whether commercial automatic cephalometric services can replace manual analysis by human experts. To the best of our knowledge, this is the first study comparing the performance of four commercial AIs at once with a sufficient sample size.

2. Materials and methods

The study samples were lateral cephalograms, which were randomly collected from the diagnostic records of patients who visited the Department of Orthodontics and Pediatric Dentistry of Kyungpook National University Dental Hospital for the treatment of malocclusion between 2012 and 2021. This study was conducted on patients after mixed dentition with erupted permanent first molars and incisors. Patients with craniofacial malformations such as cleft lip and palate, evident facial asymmetry observed in posteroanterior cephalogram, or missing molars or incisors were excluded from the study. A total of 84 cephalogram images were obtained. All images that were taken using CX-90SP (an X-ray scanner, Asahi, Kyoto, Japan) had a resolution of 150 DPI and gray level of 24 in JPG format. The image size was calibrated using a marker ruler in each cephalogram. The characteristics of the samples are presented in Table 1.

Fifteen skeletal and dental landmarks were selected to calculate the 13 commonly used cephalometric variables (Tables 2 and 3, Fig. 1). To estimate the performance of human examiners, two human examiners, expert 1 (HKN) and expert 2 (SRB), who are board-certified orthodontists with more than 7 and 5 years of clinical experience, respectively, participated in this study. First, the examiners discussed and agreed upon the definition of landmarks using three cephalograms that were not included in the study samples. Subsequent analysis was performed independently, without any communication between the examiners. The first set of measurement data was obtained by manual landmark identification on a monitor screen using computer software (6.3 Sequential Tracing Mode, AudaxCeph, Ljubljana, Slovenia). Measurements were repeated for all samples after 1 month.

This study used four commercially available AI-supported automatic cephalometric analysis services, namely, CellmatIQ (GmbH, Hamburg, Germany), CephX (ORCA Dental AI, Herzliya, Israel), AudaxCeph Automatic Tracing Mode (6.3, Audax, Ljubljana, Slovenia) and WebCeph (1.0.0, Assemblecircle, Gyeonggi-do, Korea). Initially, after uploading the anonymized cephalogram images, each AI automatically performed landmark detection. Then, to assess the pure performance of the AI architectures, the results of the AI analysis were obtained without the landmark adjustments of the human examiners. Among measurement variables, the values of Wits, FMA, FH-U1, SN-OcP and FH-OcP could not be obtained using CellmatIQ because this service does not support these variables.

The study design is illustrated in Fig. 2.

Fig. 1.Graphical representation of landmarks and variables. (A) Cephalometric landmarks; (B–D) Cephalometric variables. a: SN-U1; b: SNA; c: SNB; d: SN-MP; e: SN-OcP; f–h: Björk-Jarabak Sum; i: FH-U1; j: U1L1; k: IMPA; l: FMA; m: FH-OcP; n: Wit’s; S: sella; Na: nasion; Po: porion; Or: orbitale; Ar: articulare; Go: gonion; Me: menton; U1: upper central incisor; L1: lower central incisor.

Fig. 2.A flow chart summarizing the study design. AI: artificial intelligence.

All statistical analyses were performed using the language R (4.3.1, R Foundation for Statistical Computing, Vienna, Austria) with a significance level of 0.05. The intra-examiner reliabilities were evaluated between the first and second measurements of human examiners with intraclass correlation coefficients (ICCs) using two-way mixed-effects, single-rater and absolute agreement models [14]. Then, Dahlberg’s formula was used to calculate the method errors between the first and second attempts.

The Bland-Altman analysis was performed to estimate the measurement errors between conventional manual analysis and the newly developed AI technique [15, 16]. In the Bland-Altman protocol, measurements by expert 1 were set as the reference, and those of expert 2 and AIs were the evaluation targets. First, the means and differences between the two methods were calculated. The normality of the difference was confirmed using the Shapiro-Wilk test. The Bland-Altman statistics were then obtained. Bias measures the mean difference between the methods (systematic error), whereas the limit of agreement (LoA) represents the upper and lower limits that contain 95% of measurement errors (systematic and random errors) [17]. The maximum random error (MRE), defined as the half-width of the upper and lower LoAs, is an index of the magnitude of pure random errors between the methods [15, 16]. Finally, the Bland-Altman plots were drawn to visually understand the agreement.

The interchangeability between measurement methods can be judged by comparing the random error magnitude with the acceptable clinical limit, a priori criterion based on existing literature or widely accepted by most experts rather than statistically determined [15, 16, 17]. In particular, the new method can be interchangeably used with the previous method if the random error of a newly developed method is below or equal to the acceptable clinical limit [15, 16]. However, determining the acceptable clinical limit was challenging; there was neither an evident empirical standard nor previous studies explicitly reporting the allowable cephalometric error range. Therefore, the clinical criteria that coincided with the purpose of this study must be established. The study intended to determine whether the performance level of the AI matched that of a regular human examiner. Thus, the MREs between experts 1 and 2 were selected as the acceptable clinical limit for evaluating AI performance. Then, the MRE of the AI was compared to this clinical limit to determine interchangeability. The number of variables per AI that met this condition was counted.

The ICCs of the two-way random-effects and single-rater models were used to assess the inter-examiner reliability between the human reference (expert 1) and other examiners (expert 2 and AIs). The absolute agreement and consistency ICCs were calculated to evaluate the inter-examiner reliability with and without systematic error, respectively.

The classification reliability of the AIs was evaluated to illustrate their performance from a clinical perspective. The classification criteria are presented in Table 3. The classification by expert 1 was set as the true reference, whereas classifications by other examiners were predictions. Then, Cohen’s kappa coefficients were calculated.

3. Results

The mean intra-examiner reliability measures were 0.97 for expert 1 and 0.94 for expert 2, showing excellent reliability (Table 4). Dahlberg’s formula also estimated method errors of approximately 1 mm and 1° for linear and angular measurements, respectively. Therefore, the average of the first and second measurements were taken for the data of each human examiner, and these mean values were used in the subsequent analyses.

The descriptive and Bland-Altman statistics of cephalometric variables measured by each examiner are presented in Tables 5 and 6. The bias existed not only in expert 2 but also in the AIs. However, the magnitude of the bias was small. Exceptionally, some variables, such as SN-MP, FMA, Björk-Jarabak sum and IMPA, showed an unexpectedly large bias when measured by CephX. Meanwhile, the MREs of AI measurements were more extensive than that of human examiners, indicating more prominent random errors in AI measurements (Table 6). The magnitudes of the MREs among examiners are shown in Fig. 3. The MREs of expert 2 were always smaller than those of the AIs. Consequently, the number of variables for each AI that satisfied the agreement criteria was zero (Table 7). As an illustrative example, the Bland-Altman plots of the SN-MP are shown in Fig. 4.

Fig. 3.Bar graphs of the magnitude of MREs. The horizontal solid line represents the acceptable clinical limit for random error.

Fig. 4.Bland-Altman plots for SN-MP. In each graph, the x-axis denotes the average of two examiners’ measurement outcomes, while the y-axis is the difference between the examiners. Green dotted lines represent the bias with a 95% confidence interval. Blue dotted lines stand for the upper and lower limit of agreement with a 95% confidence interval. (A) between expert 1 and expert 2; (B) between expert 1 and CellmatIQ; (C) between expert 1 and CephX; (D) between expert 1 and AudaxCeph; (E) between expert 1 and WebCeph. CI: 95% confidence interval; LoA: limit of agreement.

The mean inter-examiner reliability measures were 0.96 and 0.96 for absolute agreement and consistency between the human examiners (Table 8), respectively. However, the mean ICCs for AIs were between 0.83 and 0.91, smaller than those of the human examiners. The difference between the absolute agreement and consistency, when measured using CephX, was evident in SN-MP, FMA, Björk-Jarabak sum and IMPA.

The overall classification reliability was estimated by Cohen’s kappa coefficients (Table 9). The reliability of expert 2 was 0.76, where the reliability of the AI was generally inferior to that of human examiners, ranging from 0.53 to 0.68. In particular, the coefficients for SN-MP, FMA, Björk-Jarabak sum and IMPA using CephX decreased to 0.2.

4. Discussion

Bias estimates the magnitude of the systematic error between the two methods [17]. The overall systematic errors between a human examiner and the AIs were clinically insignificant (Table 6). Exceptionally, only the mandibular plane-associated variables, i.e., SN-MP, FMA, Björk-Jarabak sum and IMPA, showed considerable bias when evaluated by CephX. Thus, bias correction may be necessary when evaluating these variables using CephX. Fortunately, bias correction is technically easy; i.e., the results can be quickly revised by subtracting the bias from the original measurement value [15, 16].

However, a random error correction is not available. Thus, the magnitude of the random error is the key criterion in determining the interchangeability of two methods. Accordingly, the overall performance of commercial AIs was unsatisfactory. The magnitude of the random errors of the AIs was generally more prominent than that of expert 2 (Table 6, Fig. 3). Consequently, the number of cephalometric variables that met the interchangeability criteria for each AI was nearly zero (Table 7). Thus, these currently available commercial AI services could not replace manual analysis by regular examiners.

This finding contradicts the conclusions of previous studies that reported the acceptable quality of AI cephalometrics. For instance, Kunz et al. [13] reported that CellmatIQ could analyze at the same level as human experts. Similarly, Jeon and Lee [12] concluded that CephX offers clinically acceptable results. Although these studies analyzed the measurement errors between humans and AIs using the Bland-Altman analysis, their results were interpreted without considering the acceptable clinical limit. To the best of our knowledge, this is the first attempt to determine the interchangeability between regular orthodontists and AIs according to the Bland-Altman protocol using a clinical limit. By adopting the MREs between regular, not highly experienced orthodontists as the allowable clinical limit for the random error magnitude of AIs, we could draw a different conclusion on the interchangeability between a human examiner and AI.

It might be argued that employing regular orthodontists as the standard for AI is an overly strict criterion. However, fully automated cephalometric analysis users may range from beginners to highly experienced experts. Therefore, the practical performance of AI may need to be superior to at least that of a beginner, which is why regular orthodontists were set as the evaluation criteria in this study.

While the random error sizes of the AIs were approximately twice those of Expert 2, the corresponding reliability index (ICCs) was reduced by approximately 0.1, showing values between 0.8–0.9 (Table 8). This relatively small reduction of ICCs can be understood by regarding the definition of the reliability index. The reliability index estimates the effect of the measurement error in differentiating the difference between samples and is calculated as the variance between subjects/total variance, where total variance = variance between subjects + measurement error [14, 15]. For instance, an ICC of 0.8 means that 80% of the total variance in the measurement outcomes is due to the sample’s unique variance, and 20% is owing to measurement error. The ICC formula implies that the magnitude of reliability can be less affected by the measurement error size if the variance between subjects is sufficiently large in the corresponding study group [15]. In fact, as shown in Table 1, the study samples include almost all anatomical features both anteroposteriorly and vertically, showing large variance. In summary, under a typical clinical situation that deals with various malocclusions, regular orthodontists can identify about 96% of group characteristics, while commercial AI can confirm about 86%.

In other words, if the same AI is applied to measure a group with a small variance, such as a class III or II high-angle group, the influence of the measurement error may increase, resulting in a smaller ICC value. This situation can arise especially when AI services are used for research. Bulatova et al. [10] suggested that AI-driven cephalometric analysis may facilitate research progress dealing with large samples. However, considering the relatively large measurement error and the dependence of reliability on the sample variance, caution may be needed to apply commercial AI in research.

According to the empirical standard, an ICC of 0.8–0.9 falls into the good to excellent range. However, it may be necessary to verify whether such measurement results are clinically acceptable. The dental/skeletal classification is an example of the application of cephalometric variables in actual clinical practice. Accordingly, classification reliability was evaluated by calculating Cohen’s kappa coefficients, an indicator of the classification agreement between two examiners (Table 9). Expert 2 showed the highest mean reliability index of 0.76. On the contrary, the mean reliability indices of AIs were 0.53–0.68. Depending on individual variables, there were even cases where the reliability index was <0.1 (Björk-Jarabak sum by CephX). Caution must be exercised when claiming that these relatively low-reliability indices are clinically acceptable. The classification results using commercial AI should be carefully interpreted.

We could estimate the effect of bias by comparing absolute agreement and consistency ICCs (Table 8). The difference between these two ICC types was generally between 0.01 and 0.07, supporting our previous interpretation that bias was clinically insignificant in most variables. Exceptionally, the mandibular plane-associated variables evaluated by CephX showed substantial changes from 0.6 (absolute agreement) to 0.9 (consistency). This observation indicates the importance of bias correction for these variables when using CephX.

Going through a review by a human examiner has been suggested as a countermeasure to compensate for the relatively low AI performance [12, 18]. However, in this case, the advantages of AI analysis may be limited. Reducing the tedious workload while eliminating subjective errors caused by human examiners may be the biggest advantage of AI cephalometric analysis [19, 20]. However, if a human examiner has to review and correct landmark positions individually, it is questionable whether time-saving and subjective error reduction can be obtained as expected. Some researchers argued that checking the landmarking by AI would be much easier than manual landmark detection; however, no concrete evidence supports this argument [18]. If the human examiner’s calibration is inevitable in applying AI technology, the human factor may affect the clinical effectiveness of AI cephalometrics in some way. We will address this topic in a subsequent study (Part 2).

This study has several limitations. The soft tissue measurement was not included in the study variable. In addition, the AI ​​comparison was limited to four commercial services. Furthermore, only two human examiners participated in setting the ground truth of the variables and evaluating inter-examiner errors. These limitations are related to the pilot nature of this study. The clinical significance of this study may be to provide relevant information to clinicians by quickly verifying the actual performance of commercially available AI cephalometric service in line with rapidly developing AI technology. Through this study, clinicians may get a brief clue about the current level of AI technology, but caution is needed when generalizing the results. Future studies are needed to supplement these issues.

5. Conclusions

In this study, the performance of commercially available AI cephalometric services was evaluated in comparison with the inter-examiner error between two regular human examiners. Within the limitations of this study, the following conclusions were drawn:

1. The systematic error in commercial AIs was clinically insignificant. However, the results of the mandibular plane-associated cephalometric variables of CephX must be interpreted cautiously.

2. The random error in commercial AI was significantly larger than that of human experts.

3. The reliability index of commercial AI was between 0.8 and 0.9, which corresponds to good to excellent levels according to empirical standard. However, caution may be needed when applied to a group with a small between-subject variance.

4. Cohen’s kappa coefficients, which measure the actual classification reliability of commercial AIs, were only 0.5–0.6, which were clinically unacceptable.

Therefore, commercial AIs still cannot replace manual landmarking by human experts. A human examiner’s landmark position review is mandatory when using commercial AIs.

Abbreviations

AI, artificial intelligence; ICC, intraclass correlation coefficient; LoA, limit of agreement; MRE, the maximum random error.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Author contributions

HKN and SRB—collected experimental data; HKN, SRB and JSL—analyzed the data and drafted the manuscript; HKN—designed the research and critically supervised the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This study was approved by the Institutional Review Board of Kyungpook National University Dental Hospital (KNUDH-2022-05-02-00). Patient informed consent was waived due to the study’s retrospective nature.

Acknowledgment

Thanks to Kyungpook National University Dental Hospital Dental Research Institute for support (2022).

Funding

This work was supported by Kyungpook National University Dental Hospital Institute for Dental Research (2022).

Conflict of interest

The authors declare no conflict of interest.