2.1 Network architecture

There are several issues with the current methods for improving Mask R-CNN. First, for the extracted regions of interest (RoI), the mask branch uses full convolutional operations for semantic segmentation. While fully convolutional operations have good sensitivity to local semantic information, they neglect contextual information. Second, Mask R-CNN utilizes a two-stage strategy involving detection followed by segmentation, with the segmentation results constrained by the detection outcomes. Finally, while the model performs relatively well for well-conditioned teeth, it is prone to errors when teeth are missing and lacks consideration of shape and position information. To address the aforementioned challenges, our proposed solution is a collaborative learning network model known as the Co-Mask R-CNN, as illustrated in Fig. 1.

Fig. 1.Co-Mask R-CNN Network Structure. FPN: Feature Pyramid Network; RPN: Region Proposal Network.

The Co-Mask R-CNN method adopts an encoder-decoder structure consisting of three main modules: an image enhancement module, a collaborative learning module, and a feature fusion module. The image enhancement module aims to improve the contrast and accentuate the textural details present in the original image, thereby aiding in detecting tooth edge lines. The original and enhanced images are fed into the collaborative learning module, where the encoder extracts complementary features from the tooth-related images to obtain complementary information. The feature fusion module utilizes the attention mechanism proposed by Fu et al. [28] to dynamically fuse the feature maps extracted by the collaborative learning module. This attention mechanism can incorporate contextual information and compensate for the lack of deep features in Mask R-CNN. Additionally, the weight distributions of the two input images in the model can be adaptively adjusted to achieve optimal weight ratios. Ultimately, the prediction branch produces the ultimate segmentation map. In the following sections, we explain the implementation details for each of these components.

2.1.1 Image enhancement

To reduce brightness, inconsistent exposure, and blurred edges. These factors significantly impact the effectiveness of segmentation networks, image enhancement is crucial in X-ray images. The present investigation introduces a data augmentation technique to address these challenges inspired by the methodology established by Zhou et al. [29]. To alleviate the issue of indistinct dental boundaries, we employ the Sobel operator to convolve the image and enhance the prominence of edge information, resulting in edge image generation. This approach reduces noise, enhances image contrast and sharpness, and standardizes image quality to sharpen textural details. Edge images enhance the edge-related information within the original image, emphasizing the outlines and forms of the teeth. In contrast, the original image presented a more comprehensive depiction of dental information, encompassing contextual background details as well.

In the enhancement process, the contrast-enhanced image I_ce(x,y;δ) is obtained through the following formula:

$I_{ce}(x,y;\delta )=\eta I(x,y)+\theta G(x,y;\delta )\ast I(x,y)+S(x,y)\ast I(x,y)+\beta$ (1)

Where I(x,y) is the raw bitewing radiograph, * denotes the convolution operator, G(x,y;δ) represents the Gaussian filter with standard deviation δ, and S(x,y) denotes the Sobel filter with a 3 × 3 kernel. The contrast-enhanced images are used as inputs for one branch of the collaborative learning network model. Fig. 2 shows the effect of applying image enhancement, where (1) is the original image, (2) is the image enhancement by Zhou et al. [29], and (3) is the edge image. As shown in the figure, the edge image provides higher contrast and brightness and sharper edges than does the original and Zhou’s methods.

Fig. 2.Image enhancement results.

2.1.3 Complementary feature fusion

This study incorporates a dual attention mechanism comprising a Channel Attention Module (CAM) and a Position Attention Module (PAM) to leverage the extracted features from both branches more effectively [32]. These modules dynamically assign weights to the two branches, effectively incorporate contextual information, and achieve complementary feature aggregation. The outputs of the attention modules are transformed using convolutional layers and then combined through weighted summation to achieve feature fusion, as depicted in Fig. 3.

Fig. 3.Dual attention network structure.

2.1.3.1 Position attention module

The PAM explores the spatial relationships within the feature maps obtained from the two branches. This is accomplished by updating the features at specific positions and assigning weights based on the similarities between corresponding features at those positions. As a result, positions with similar features can mutually enhance each other, irrespective of their spatial separation. By encoding extensive contextual information into the original features, the PAM enhances the representativeness significantly.

As illustrated in Fig. 4, P_enhancement and P_original represent the feature maps of the edge image extracted after the backbone network and the feature map of the local image, respectively. The feature map P_enhancement ϵ R^C×H×W acquired from the enhancement branch is initially fed into a convolutional layer, resulting in the generation of two novel feature maps, denoted as G and F, {G,F} ϵ R^C×H×W. Subsequently, the G, F matrix is transformed into an R^C×N matrix, where N = H × W repres matrients the overall pixel count. Through the multiplication of the transposed F and G matrices, the spatial attention map M ϵ R^N×N is derived. To ensure normalization, a softmax function is applied to this map, producing a normalized output.

$M_{yx}=\frac{\exp (G_x\cdot F_y)}{\sum _{i=1}^N\exp (G_x\cdot F_y)}$ (2)

Fig. 4.Position attention module.

Where M_yx ensures the influence of the x^th position on the y^th position, and the correlation between the feature representations of two positions increases when their corresponding M_yx values increase. Concurrently, feature P_enhancement undergoes a convolutional layer, generating a fresh feature map denoted as O ϵ R^C×H×W. This map is then reshaped into R^C×N. Subsequently, the dot product of O and the transposed M matrix is computed, resulting in a reshaped output R^C×H×W. This output Q ϵ R^C×H×W is subsequently scaled by the parameter λ and added elementwise to the original features P_original ϵ R^C×H×W of the input branch, as shown below:

$Q_y=\lambda \sum _{x=1}^N\left(M_{yx}{D}_x\right)+{P\mathrm{\_}original}_y$ (3)

By initializing the weight parameter λ to 0 and progressively adapting it during the training phase, Eqn. 3 provides valuable insights. This approach implies that the resultant feature Q at each position is obtained by combining features from all positions and the original feature via weighted summation. This mechanism facilitates the integration of global contextual information and the selective incorporation of context guided by the spatial attention map. As a consequence, similar semantic features are mutually reinforced, leading to improved intraclass compactness and enhanced semantic consistency.

2.1.3.2 Channel attention module

The CAM treats each channel of a feature map as a distinctive response corresponding to a specific class, thus revealing the interrelated nature of diverse semantic responses. By delving into the interdependencies among channel maps, CAM has the ability to highlight interdependence between feature channels and enhance the specificity of semantics.

Fig. 5 illustrates the architecture of the CAM, where P_enhancement and P_original represent the feature maps of the edge image extracted after the backbone network and the feature map of the local image, respectively. Unlike the PAM, feature P_enhancement is directly reshaped into R^C×N and then subjected to matrix multiplication via transpose. The resulting feature maps are multiplied and passed through a softmax function, generating the channel feature map T ϵ R^C×C, as shown below:

$T_{yx}=\frac{\exp ({P\mathrm{\_}enhancement}_x\cdot {P\mathrm{\_}enhancement}_y)}{\sum _{x=1}^N\exp ({P\mathrm{\_}enhancement}_x\cdot {P\mathrm{\_}enhancement}_y)}$ (4)

Fig. 5.Channel attention module.

Where T_tx indicates the impact of the x^th channel on the y^th channel. Furthermore, matrix multiplication between T and the transpose of P_enhancement is performed, resulting in the reshaped feature T ϵ R^C×H×W. This outcome is then scaled by the parameter λ and added elementwise to the original feature P_original from the input branch, yielding the final output Q ϵ R^C×H×W, as shown below:

$Q_y=\lambda \sum _{x=1}^N\left(T_{yx}{P\mathrm{\_}enhancement}_x\right)+{P\mathrm{\_}original}_y$ (5)

Where the weight parameter λ is initialized to 0 and gradually assigned higher significance during the training phase. The equation illustrates that the ultimate feature representation of each channel is a combination of the features extracted from all channels, including the original features, with each channel’s contribution weighted accordingly. By performing a weighted summation of all channels, the final feature representation can capture long-range semantic dependencies between feature maps, improving feature distinctiveness. To leverage the benefits of distant contextual information more effectively, the proposed approach combines the features extracted by two attention modules.

2.1.4 Prediction branch

The prediction branch integrates information by leveraging the network architecture of Mask R-CNN and employing the region proposal network (RPN) and ROI modules to extract and consolidate relevant features from the feature maps, as depicted in Fig. 6. Subsequently, the feature vectors are individually fed into two separate prediction network branches. The upper branch performs classification prediction and bounding box regression, while the lower branch is responsible for generating masks that correspond to the detected objects.

Fig. 6.Prediction branch.

2.2 Datasets

We collected and curated the experimental dataset from Peking University School and Hospital of Stomatology. It encompasses a collection of 815 bitewing radiographs focusing on children’s teeth. Each image was resized to 512 × 512, and the label data were annotated by a professional dentist with extensive clinical experience. After annotation, the researchers used LabelMe software to pixel-level annotate the tooth area drawn by the dentist on the bitewing radiographs. The generated annotation data were saved in JSON files, including the contour coordinates of each tooth. These annotated data were divided into training and testing sets for validation of the tooth instance segmentation network.

2.3 Metrics for statistical analysis

To evaluate the performance of Co-Mask R-CNN on tooth instance segmentation tasks, we employ metrics such as AP (Average Precision), AR (Average Recall), and IOU (Intersection Over Union). Precision serves as a metric for assessing the accuracy of a model when making positive predictions and represents the percentage of correctly identified positive instances. Recall quantifies the proportion of true positives in the testing dataset that the model correctly detects, reflecting the comprehensiveness of positive identification. The evaluation is conducted by employing IOU thresholds of 50 and 75, along with a range of IOU thresholds from 50 to 95 with increments of 5. The IOU = (50:95) signifies the mAP (mean Average Precision) calculated across various IOU thresholds, thereby offering a comprehensive measure of the model’s performance across a spectrum of IOU values. The inclusion of IOU increment averaging aims to ensure that the model performs well not only at IOU = 50 but also at higher IOU thresholds. By calculating the AP of the model at various IOU values and taking the average, we obtain a comprehensive assessment of the model’s accuracy.

2.4 Implementation details

In the experimental process, the dataset was divided into a training set and a test set at a ratio of 9:1. The training set was subsequently divided into five parts for fivefold cross-validation, after which the average value was taken as the final result. The hyperparameters, such as the initial learning rate, learning rate decay strategy and iteration number, were dynamically adjusted, the learning rate decay method used was the cosine function, and the initial learning rate was set to 0.0001 at the beginning of the training process. The same loss function as that of Mask R-CNN was used, and the cross-entropy function was used as the classification loss. Similarly, the SmoothL1 loss function was used as the bounding box. Similarly, the loss function as Mask R-CNN was used. Similarly, the cross-entropy function was used as the classification loss, the SmoothL1 loss function was used as the bounding box loss, the binary cross-entropy function was used as the mask loss, and the sum of each loss was obtained as the final loss value to optimize the model performance. A total of 100 iterations were carried out in the training phase of this experiment, and as shown in Fig. 7, the decline rate of the loss function tends to level off when the epoch is greater than 80, at which time the final model in the training phase is determined.

Fig. 7.Loss curves.

4.1 Comparisons with state-of-the-art models

To assess the superior performance of Co-Mask R-CNN, we performed comparative experimental analysis with popular instance segmentation networks such as SSD (Single Shot MultiBox Detector) [32], YOLOv3 (You Only Look Once v3) [33], YOLOv7 (You Only Look Once v7) [34], and Mask R-CNN [15] and evaluated the performance based on their recall rate and accuracy at various thresholds. Notably, all the models were trained without utilizing pretrained weights. Table 3 provides a comprehensive performance comparison of the tooth image dataset. Our Co-Mask R-CNN outperforms other classical models in terms of both precision and recall.

As shown in Table 3, at an IOU threshold of 50, the various models perform similarly in terms of precision. However, as the IOU threshold increases, Co-Mask R-CNN has more advantages, achieving a much higher precision rate than other classical models at IOU = (50:95), with a 5.3% improvement compared to Mask R-CNN. For recall, Co-Mask R-CNN still obtains the best results at high IOU thresholds, indicating that it has higher accuracy in detecting objects, and the generated detection boxes are closest to the true labels, resulting in more precise segmentation boundaries.

These experimental findings affirm the potential effectiveness of the proposed Co-Mask R-CNN as a viable solution for tooth instance segmentation tasks.

The experimental results of three images are visualized in Table 4, including a normal clear image, an image with metal artefact noise, and an image with uneven exposure. As shown in the first row of the normal clear image, SSD missed some teeth, while YOLOv3 and YOLOv7 made correct detections; however, the detection scores for each tooth box were much lower than those of the Co-Mask R-CNN. In the second row of the image with metal artefact noise, SSD and YOLOv7 missed some teeth, while Co-Mask R-CNN had higher detection scores than YOLOv3 and Mask R-CNN. In the third row, only YOLOv3 correctly detected teeth with uneven exposure in the image, while SSD and YOLOv7 missed some teeth. Although both Mask R-CNN and Co-Mask R-CNN exhibited instances of false detections, we demonstrated superior performance in terms of segmentation boundaries. The segmentation boundaries produced by the Co-Mask R-CNN were smoother and more accurate than those produced by the Mask R-CNN, resulting in a larger area of correct segmentation. Especially in scenarios where the object region tends to be larger than the background, the Co-Mask R-CNN excels and closely approximates the true label values. Its exceptional performance in accurately delineating tooth regions further establishes its efficacy in this specific segmentation task.

Table 4.Tooth instance segmentation results.

	SSD [32]	YOLOv3 [33]	YOLOv7 [34]	Mask R-CNN [15]	Co-Mask R-CNN
(a)
(b)
(c)

4.2 Ablation study

To validate the effectiveness of each module of the Co-Mask R-CNN, we performed four ablation experiments using the original image dataset and the enhanced image dataset. The first experiment involved training Mask R-CNN solely on the original image dataset, denoted as M_s1. The second experiment trained Mask R-CNN on the augmented image dataset, referred to as M_s2. The M_s3 network model is a two-branch Mask R-CNN but does not use a feature fusion module. Simple feature fusion is performed by superposition of the two-branch feature map sampling channels, and the datasets are the original image and the enhanced image. Finally, the Co-Mask R-CNN model was trained on both the original and augmented image datasets. The instance segmentation results obtained from these experiments are presented in Table 5. In this study, the segmentation and classification results are separately discussed to facilitate comparison with the results of other methods.

Table 5.Ablation study.

	Input	Ms1	Mss2	Mss3	Co-Mask R-CNN	Ground Truth
(a)
(b)
(c)

4.3 Comparison of tooth segmentation results

A comparison of the results of the four sets of experiments is shown in Table 6. The quantitative evaluation shows that the collaborative learning model proposed in this paper improves the precision rate and recall by 5.3% and 3.8%, respectively, compared with the original Mask R-CNN model of M_s1 at IOU = (50:95); however, M_s2 has a decrease in the precision rate and recall by 2.1% and 2.7%, respectively, compared with M_s1. This is because the dataset used for M_s2 is an enhanced image, and although the enhanced image makes the edge lines more obvious, the enhancement causes the image to lose more detail, worsening the segmentation results. Furthermore, M_s3 has some performance improvement compared to M_s1, but the performance decreases compared to the model proposed in this experiment, thus proving the effectiveness of the proposed feature fusion module.

As shown in Table 5, we present the instance segmentation results of the ablation study on three representative dental images: a clear normal image (a), an image with metal artefacts (b), and an image with tooth loss (c). In image (a), M_s1 exhibited poor performance, with two missed teeth and unclear boundaries. In contrast, M_s1, M_s3 and Co-Mask R-CNN accurately segmented each tooth, with Co-Mask R-CNN producing smoother and clearer boundaries. For image (b), M_s2 also missed teeth and produced unclear boundaries, while Co-Mask R-CNN produced smoother and more accurate segmentation boundaries, indicating its robustness to metal artefacts. For image (c), all four experiments resulted in oversegmentation, highlighting the need for improved models for images with exposure interference and missing teeth. Compared with the original Mask R-CNN, the Co-Mask R-CNN shows an improvement in tooth segmentation accuracy, demonstrating the potential of attention to detail extraction for enhancing the segmentation performance of neural networks. Nonetheless, further research is required to address the challenges posed by dental images with complex backgrounds and missing or damaged teeth.

4.4 Comparison of tooth detection results

Table 7 shows the detection results of the four experiments, which demonstrate that the proposed collaborative instance segmentation model achieves the best performance in accurately identifying and detecting teeth. Compared to the Mask R-CNN M_s1, the Co-Mask R-CNN achieves 4.3% and 2.9% improvements in precision and recall, respectively, at IOU = (50:95), and the Co-Mask R-CNN also obtains better detection results than does M_s3. However, M_s2 has the worst detection, which is in line with the segmentation results. This also proves the effectiveness of the proposed module in this study.

In Table 5, the object detection labels and corresponding scores are presented above the bounding boxes. For image (a), Co-Mask R-CNN demonstrated the best detection performance, without any missed or false detections, with the detection boxes almost perfectly overlapping with the ground truth and with the highest detection score. M_s1 and M_s3 exhibited instances of multiple detections, while M_s2 had the poorest detection results with missed and false detections. For image (b), Co-Mask R-CNN still demonstrated the best detection performance, with accurate detection boxes. M_s1 detected the same number of boxes as did the Co-Mask R-CNN but with lower detection scores. M_s2 exhibited the poorest detection results. For the image (c) with missing teeth, M_s2 had the poorest detection results; M_s1 and M_s3 exhibited instances of false detections, detecting six teeth; and Co-Mask R-CNN also exhibited instances of false detections, detecting five teeth. Nevertheless, Co-Mask R-CNN still demonstrated the best detection performance, with higher detection scores for the correctly detected boxes than did M_s1. Therefore, the proposed instance segmentation model provides more accurate tooth detection, with higher recall, than does its original network.

Upon analysing the segmentation and detection results, we observed that the performance of the Co-Mask R-CNN surpassed that of the Mask R-CNN model (M_s1). Furthermore, as the IOU threshold increased, the differences among the four experiments became more prominent. The performance of the Co-Mask R-CNN model demonstrates its ability to produce results that closely align with the ground truth labels, providing further evidence of the effectiveness of the dual-branch fusion strategy proposed in this paper. Specifically, the attention-based feature fusion module prioritizes detailed information extraction, thereby integrating diverse multiscale contextual features. By employing an attention mechanism to fuse the weights of the original and enhanced images, the issues of inadequate global information extraction and potential segmentation errors caused by localization in the conventional Mask R-CNN were successfully resolved. This approach significantly enhances the model’s generalization ability and results in more precise tooth instance segmentation. The fused model exhibits superior stability and surpasses the unfused results.