Transfer Deep Learning for Dental and Maxillofacial Imaging Modality Classification: A Preliminary Study

INTRODUCTION

In dental practice, various modalities of X-rays are used. Almost all patients need a specific X-ray or series of images. The most common of them are intraoral periapical and bitewing X-rays or extraoral panoramic and cephalometric X-rays. Medical images can be taken in the clinic of the attending dentist or by referral to radiological services. It is important to note that change of the place of treatment can occasionally occur. In addition, referrals to a dental specialist, or to a specialist in other medical fields, may necessitate performance of additional images of different modality. It follows that dentists receive a wide variety of X-rays and diverse types of medical imaging, some of which are not familiar for them, with possible difficulties not only in interpretation, but even in identification of the type of image. The lack of an integrative environment with a wide range of specialists in different fields of medicine and dentistry precludes a fast and convenient consultation form many dentists. Therefore, an uncertain type of medical imaging modality might seriously complicate the understanding of the patient’s status and negatively affect the diagnosis and appropriate treatment, as demonstrated by the flow chart of a possible decision-making algorithm in the presence of an unidentified X-ray image (Fig.1).

Figure 1: A flow-chart of a possible decision making algorithm in the presence of an unidentified X-ray image

The presence and possibility of retrieving certain types of medical images are also important factors in the training of staff and students, as well as in scientific research [1]. The number of different medical images is constantly growing and extracting information from big database collections is of increasing importance. This is especially noticeable in dentistry because of the lack of proper labeling and highly decentralized data. Consequently, the classification of images is of great importance in clinical practice as well as in training and research. The possibility of automatic classification using computer technology could solve this challenge.

Recently, there has been a serious progress in various algorithms related to the development of artificial intelligence. Research and innovation in this field are based mainly on machine learning and in particular on methods of deep learning. Definitely, the methods of machine learning have changed and even revolutionized the functioning of entire areas, including, machine language translation, information retrieval, and analysis of large databases. In the medical field many applications are being developed on the basis of deep learning, such as solving issues of clinical diagnostics and the choice of treatment algorithms [2]. Special consideration has been given to the application of these technologies in radiology [3]. In this line, much attention is being given to the application of computer-based diagnostics based on machine learning in radiology for classification of medical images, anatomical formations and detect pathologies. There is a certain progression in this kind of research, as can be seen on examples of analysis of medical images form liver [4], lung [5, 6], breast [7] and brain [8, 9].

A number of studies have been performed in the field of oral and maxillofacial radiology, from the detection of caries [10] and evaluation of periodontal tissues [11, 12] to the detection of cysts and tumors [13, 14]. It should be emphasized that most research in the field of deep learning currently requires the use of very large sets of image datasets. This might be problematic in dentistry, as the data is decentralized to a large number of different clinics and X-ray services. The present research, as in our previous studies on this topic [15, 16], we attempted to use a small X-ray dataset with a proper neural network architecture for this task.

Neural networks in radiology are applied using classification, object detection and segmentation algorithms. Among existing algorithms, the task of classification of images is basic and one of the most successful. The classification algorithm is a prediction of what kind of class a certain element belongs to, in this case the element being the radiologic image. The classifier can be binary or multiclass and classify an element into one of several categories. Classification is one of the most common methods of machine learning and is commonly used to solve such problems of image recognition [17]. Furthermore, deep learning, based on the use of neural networks, provides the best results in the classification of images [18].

In medicine, modality is one of the most important tools for the retrieval of medical images, including X-rays and other types of images. As it has already been noted, the accelerated growth in the number of medical images, associated with a subsequent increase in the number and size of databases, makes image retrieval very difficult [19]. Automatic modality classification may resolve this problem, as has been shown by The Evaluation Forum (Image CLEF) [19] that organized the classification of medical images using various methods, including textual and visual. Currently, the best performance in predicting the modality of medical images is being attributed to the convolutional neural network, which is a state of the art in the classification of all types of images. The use of transfer deep learning, in which neural networks do not start training from scratch, but undergo preliminary training on other images, significantly improves the final result of automatic classification [20].

We have not found any peer-review study that verified the concept of classification of modality and sub-modality separately and specifically in the field of dentistry.

Therefore, the purpose of this study was to apply a technique of transfer deep learning based on neural networks on a relatively small data set of the most available and familiar types of X-ray images in dentistry, in order to determine the possibility of their automatic classification.

MATERIALS AND METHOD

Dataset

In this research, we used in-house dataset created within the School of Dental Medicine, Tel Aviv University. The study was approved by the ethical committee of Tel Aviv University. The dataset contained randomly chosen and anonymized 620 digital Panoramic and Cephalometric X-ray images for orthodontic examinations from CS 8100 Digital Panoramic System (Carestream Dental LLC, Atlanta, USA) and Planmeca ProMax 2D (Planmeca, Helsinki, Finland) units. These types of images were chosen as the most familiar and used extraoral images, the classification of which does not cause any problems in the ordinary practice. In addition, these images are the most accessible for creating a database and increasing its size if necessary. Selections were made by specialists in oral medicine with special training in oral and maxillofacial radiology.

As there was a small amount of available training data, we used data augmentation by randomly flipping and rotating images to multiple angles, width and height shifting and zoom changing (Fig. 2). This allowed the network to learn from larger variability of examples and prevent overfitting.

Figure 2: Examples of data augmentation by randomly flipping and rotating images to multiple angles, width and height shifting and zoom changing

Training Details

The input images were used to train the networks in the minibatch manner, when batch size was set to 32. Gradient descent computation and updates were carried out by Adam optimizer with a fixed learning rate of 0.001. It used 10 epochs with 40 steps per epoch. Initially, the number of epochs was taken as 100, but the maximum result was achieved much earlier and the use of more epochs was not significant. For the implementation, we used Keras with Tensorflow backend. In this way, 70% of the data were randomly chosen for training, 10% for validation, and accuracy was tested on 20%. The test dataset contained Panoramic and Cephalometric X-ray images from 2 different devices: CS 8100 Digital Panoramic System (as in training and validation) and Planmeca ProMax 2D. The model was trained using NVIDIA GeForce GTX 1080 Ti GPU.

RESULTS

The simplified structure of the final modality classification convolutional neural network model is illustrated in Fig. 3. This modality has completed the transfer deep learning based on the network, pre-trained on ImageNet, and has used panoramic or cephalometric images as input data and classification labels as the outputs. Unexpectedly, accuracy on all neural network architectures (VGG16, NASNetLarge, Xception) was 100%. The X-ray images in the test database were not subject to any preliminary processing. Paradoxically, the maximum result in the training and validation process was achieved after only a few epochs, which is illustrated by the example of VGG16 training process (Fig.4). The use of poor quality images did not lead to any changes in the result. For neural network training the X-rays from only one CS 8100 Digital Panoramic System unit were used, however for the purposes of testing the images obtained on Planmeca ProMax X-ray unit were also applied, which absolutely had no negative impact on the result. Statistical analysis in this study were not relevant, as all results were true positive.

Figure 3: Simplified structure of the final modality classification convolutional neural network (CNN) model, which has completed the transfer deep learning on the basis of the network, pre-trained on ImageNet, and which uses panoramic or cephalometric images as input data and classification labels as the outputs.

Figure 4: Example of VGG16 training process

DISCUSSION

Currently, there are active developments related to the use of machine learning, and specifically deep learning algorithms, in radiology, in general, and in oral and maxillofacial radiology, in particular [28, 29]. As already mentioned, the main tasks in computer vision based on neural networks are classification, detection and segmentation. Classification is the baseline and most successful function. It is necessary to mention that after the process of segmentation or detection there is a subsequent stage of classification. There has been incredible success in non-medical image classification, and in the Large Scale Visual Recognition Challenge (LSVRC) based on ImageNet task, image classification with neural networks has reached an error rate of only 3.57%, exceeding human accuracy [30]. Encouraging results have also been reported in maxillofacial radiology. For example, the use of deep learning technology to detect and classify teeth on periapical radiographs showed high precision and recall results of 95.8% and 96.1% in total [31]. In another interesting study, it has been shown that the accuracy, sensitivity, and specificity of deep learning were 96.0%, 100%, and 92.0%, respectively. The corresponding values of experienced radiologists were 98.3%, 99.3%, and 97.3%, while those of inexperienced radiologists were 83.5%, 77.9%, and 89.2% [32]. It may be noted, that the results of the neural network were very similar to those of experienced radiologists, but much better than those of inexperienced radiologists.

In regard to the classification of medical image modalities, the leading method is also deep learning by neural networks. For instance, The Evaluation Forum (ImageCLEF) [19] 2016 achieved 87.37% accuracy based on transfer deep learning [33]. In one of the most successful studies using the VGG16 architecture, an accuracy of 98% was achieved in the classification of seven different modalities of medical images [34].

In this study, the maximum accuracy of detection of 100% was reached, which can be explained by several factors. Initially, as has already been demonstrated, there is an incredible success in automatic classification, exceeding human classification capabilities. In medical research there are also very high results. It should be considered that the presented results are preliminary and only two modalities of images were used in the study, although there were many types of images. Currently, different intraoral imaging methods such as occlusal, periapical and bitewing projections are used in dentistry. Extraoral images are even more varied and include two-dimensional imaging as cephalometric, panoramic, Waters, and submentovertex projections, as well as three-dimensional imaging as cone beam computed tomography, multidetector computed tomographic imaging, magnetic resonance imaging and other modalities. In addition, three-dimensional images have a variety of submodalities. Furthermore, the use of bimodality provides the best, almost the highest possible result. Most probably, as can be seen from additional studies, the multiclass character of the image decreases the success of automatic classification. Therefore, it is necessary to carry out research in order to use the automatic classification of most modalities. Creating this kind of database in dentistry is very difficult for a number of reasons, as already mentioned. Certain types of modality and submodality in medicine are identical or similar to those used in dentistry, therefore, similar high results are expected in both fields.

In the dental practice, a large number of different units, located mostly in private clinics, are being used for intraoral and extraoral imaging. It is difficult, and almost impossible, to obtain examples of images from most of the available units in order to create a complete database in terms of variability. In this study, it was decided to use images from one device for training, and for testing to use images from an additional device. Although the image quality was different, there were no changes in the prediction, and the model kept working with maximum accuracy. Furthermore, testing was provided with poor quality images, which on clinical grounds are not diagnostically effective. These modifications did not change the automatic classification when using the obtained models. Therefore, the high stability of the models under different modifications has been demonstrated. We could assume that the use of images performed on other devices would have produced poorer results. Likewise, the use of a large number of test datasets would also increase the error rate.

Traditional machine learning, including deep learning, is used to solve a particular problem that requires a specific dataset. Learning begins from scratch, using no “prior knowledge”. Thus, in computer vision, solving each specific task necessitates training of the neural network from scratch, which usually requires very large datasets. However, a significant number of studies in dentistry in this field are performed using small-sized datasets [35]. In transfer learning, the network is already trained on other visual data, and transferred to the new training in a more “experienced state”, based on the already acquired “knowledge”. Furthermore, this allows to speed up the learning process, to increase the quality of the resulting model, and crucially especially in dentistry, to decrease the size of the dataset [33].

CONCLUSION

In the current dental practice, a large number of various X-rays and other methods of medical imaging are used. Although there are permanent and familiar imaging methods for dentists, some images might be difficult to categorize or sub-categorize accurately. This might result in loss of important medical information, which in some cases can seriously complicate the process of reaching the right diagnosis and offering the best treatment plan. In addition, the increasing number of databases, that are most often not categorized in dentistry, makes it difficult to retrieve specific types of images. In this study, outstanding results have been obtained for the automatic classification of different modalities of X-ray images used in dentistry. The most promising direction for the development of this kind of applications is transfer deep learning, which in this case allowed to achieve maximum results using a limited number of X-ray images database. Despite the use of images familiar with dentists, the possibility of successful automatic categorization of such images has been clearly established. Further studies on the automatic categorization of modalities, as well as sub-modalities, can entirely avoid the occasional difficulties that might be encountered in the daily practice of dentists and, ultimately, could improve the quality of diagnosis and treatment.

Conflict of interest

Lazar Kats, MarilenaVered, Johnny Kharouba and Sigalit Blumer declare that they have no conflict of interest.