Article Data

  • Views 1545
  • Dowloads 191

Original Research

Open Access

Commercial artificial intelligence lateral cephalometric analysis: part 2—effects of human examiners on artificial intelligence performance, a pilot study

  • Jaesik Lee1,†
  • Seong-Ryeol Bae2,†
  • Hyung-Kyu Noh2,*,

1Department of Pediatric Dentistry, School of Dentistry, Kyungpook National University, 41940 Daegu, Republic of Korea

2Department of Orthodontics, School of Dentistry, Kyungpook National University, 41940 Daegu, Republic of Korea

DOI: 10.22514/jocpd.2023.087 Vol.47,Issue 6,November 2023 pp.130-141

Submitted: 10 April 2023 Accepted: 12 May 2023

Published: 03 November 2023

*Corresponding Author(s): Hyung-Kyu Noh E-mail:

† These authors contributed equally.


At the current technology level, a human examiner’s review must be accompanied to compensate for the insufficient commercial artificial intelligence (AI) performance. This study aimed to investigate the effects of the human examiner’s expertise on the efficacy of AI analysis, including time-saving and error reduction. Eighty-four pretreatment cephalograms were randomly selected for this study. First, human examiners (one beginner and two regular examiners) manually detected 15 cephalometric landmarks and measured the required time. Subsequently, commercial AI services automatically identified these landmarks. Finally, the human examiners reviewed the AI landmark determination and adjusted them as needed while measuring the time required for the review process. Then, the elapsed time was compared statistically. Systematic and random errors among examiners (human examiners, AI and their combinations) were assessed using the Bland-Altman analysis. Intraclass correlation coefficients were used to estimate the inter-examiner reliability. No clinically significant time difference was observed regardless of AI use. AI measurement error decreased substantially after the review of the human examiner. From the standpoint of the human examiner, beginners could obtain better results than manual landmarking. However, the AI review outcomes of the regular examiner were not as good as those of manual analysis, possibly due to AI-dependent landmark decisions. The reliability of AI analysis could also be improved by employing the human examiner’s review. Although the time-saving effect was not evident, commercial AI cephalometric services are currently recommendable for beginners.


Cephalometric; Artificial intelligence; Efficacy; Accuracy; Precision; Reliability

Cite and Share

Jaesik Lee,Seong-Ryeol Bae,Hyung-Kyu Noh. Commercial artificial intelligence lateral cephalometric analysis: part 2—effects of human examiners on artificial intelligence performance, a pilot study. Journal of Clinical Pediatric Dentistry. 2023. 47(6);130-141.


[1] Schwendicke F, Chaurasia A, Arsiwala L, Lee JH, Elhennawy K, Jost-Brinkmann PG, et al. Deep learning for cephalometric landmark detection: systematic review and meta-analysis. Clinical Oral Investigations. 2021; 25: 4299–4309.

[2] Kim H, Shim E, Park J, Kim YJ, Lee U, Kim Y. Web-based fully automated cephalometric analysis by deep learning. Computer Methods and Programs in Biomedicine. 2020; 194: 105513.

[3] Hwang HW, Moon JH, Kim MG, Donatelli RE, Lee SJ. Evaluation of automated cephalometric analysis based on the latest deep learning method. The Angle Orthodontist. 2021; 91: 329–335.

[4] Gil SM, Kim I, Cho JH, Hong M, Kim M, Kim SJ, et al. Accuracy of auto-identification of the posteroanterior cephalometric landmarks using cascade convolution neural network algorithm and cephalometric images of different quality from nationwide multiple centers. American Journal of Orthodontics and Dentofacial Orthopedics. 2022; 161: e361–e371.

[5] Yoon HJ, Kim DR, Gwon E, Kim N, Baek SH, Ahn HW, et al. Fully automated identification of cephalometric landmarks for upper airway assessment using cascaded convolutional neural networks. European Journal of Orthodontics. 2022; 44: 66–77.

[6] Park JH, Hwang HW, Moon JH, Yu Y, Kim H, Her SB, et al. Automated identification of cephalometric landmarks: part 1—comparisons between the latest deep-learning methods YOLOV3 and SSD. The Angle Orthodontist. 2019; 89: 903–909.

[7] Hwang HW, Park JH, Moon JH, Yu Y, Kim H, Her SB, et al. Automated identification of cephalometric landmarks: part 2—might it be better than human? The Angle Orthodontist. 2020; 90: 69–76.

[8] Choi YJ, Lee K. Possibilities of artificial intelligence use in orthodontic diagnosis and treatment planning: image recognition and three-dimensional VTO. Seminars in Orthodontics. 2021; 27: 121–129.

[9] Leonardi R, Giordano D, Maiorana F, Spampinato C. Automatic cephalometric analysis: a systematic review. The Angle Orthodontist. 2008; 78: 145–151.

[10] Rudolph DJ, Sinclair PM, Coggins JM. Automatic computerized radiographic identification of cephalometric landmarks. American Journal of Orthodontics and Dentofacial Orthopedics. 1998; 113: 173–179.

[11] Kazandjian S, Kiliaridis S, Mavropoulos A. Validity and reliability of a new edge-based computerized method for identification of cephalometric landmarks. The Angle Orthodontist. 2006; 76: 619–624.

[12] Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine. 2016; 15: 155–163.

[13] Haghayegh S, Kang HA, Khoshnevis S, Smolensky MH, Diller KR. A comprehensive guideline for Bland-Altman and intra class correlation calculations to properly compare two methods of measurement and interpret findings. Physiological Measurement. 2020; 41: 055012.

[14] van Stralen KJ, Jager KJ, Zoccali C, Dekker FW. Agreement between methods. Kidney International. 2008; 74: 1116–1120.

[15] Bland JM, Altman DG. Measuring agreement in method comparison studies. Statistical Methods in Medical Research. 1999; 8: 135–160.

[16] Lagravère MO, Low C, Flores-Mir C, Chung R, Carey JP, Heo G, et al. Intraexaminer and interexaminer reliabilities of landmark identification on digitized lateral cephalograms and formatted 3-dimensional cone-beam computerized tomography images. American Journal of Orthodontics and Dentofacial Orthopedics. 2010; 137: 598–604.

[17] Tanikawa C, Lee C, Lim J, Oka A, Yamashiro T. Clinical applicability of automated cephalometric landmark identification: Part I—patient-related identification errors. Orthodontics & Craniofacial Research. 2021; 24: 43–52.

[18] Jeon S, Lee KC. Comparison of cephalometric measurements between conventional and automatic cephalometric analysis using convolutional neural network. Progress in Orthodontics. 2021; 22: 14.

Abstracted / indexed in

Science Citation Index Expanded (SciSearch) Created as SCI in 1964, Science Citation Index Expanded now indexes over 9,500 of the world’s most impactful journals across 178 scientific disciplines. More than 53 million records and 1.18 billion cited references date back from 1900 to present.

Biological Abstracts Easily discover critical journal coverage of the life sciences with Biological Abstracts, produced by the Web of Science Group, with topics ranging from botany to microbiology to pharmacology. Including BIOSIS indexing and MeSH terms, specialized indexing in Biological Abstracts helps you to discover more accurate, context-sensitive results.

Google Scholar Google Scholar is a freely accessible web search engine that indexes the full text or metadata of scholarly literature across an array of publishing formats and disciplines.

JournalSeek Genamics JournalSeek is the largest completely categorized database of freely available journal information available on the internet. The database presently contains 39226 titles. Journal information includes the description (aims and scope), journal abbreviation, journal homepage link, subject category and ISSN.

Current Contents - Clinical Medicine Current Contents - Clinical Medicine provides easy access to complete tables of contents, abstracts, bibliographic information and all other significant items in recently published issues from over 1,000 leading journals in clinical medicine.

BIOSIS Previews BIOSIS Previews is an English-language, bibliographic database service, with abstracts and citation indexing. It is part of Clarivate Analytics Web of Science suite. BIOSIS Previews indexes data from 1926 to the present.

Journal Citation Reports/Science Edition Journal Citation Reports/Science Edition aims to evaluate a journal’s value from multiple perspectives including the journal impact factor, descriptive data about a journal’s open access content as well as contributing authors, and provide readers a transparent and publisher-neutral data & statistics information about the journal.

Scopus: CiteScore 2.0 (2022) Scopus is Elsevier's abstract and citation database launched in 2004. Scopus covers nearly 36,377 titles (22,794 active titles and 13,583 Inactive titles) from approximately 11,678 publishers, of which 34,346 are peer-reviewed journals in top-level subject fields: life sciences, social sciences, physical sciences and health sciences.

Submission Turnaround Time