Deep Learning-Based Image Annotation for Leukocyte Segmentation and Classification

Deep Learning-Based Image Annotation for Leukocyte Segmentation and Classification of Blood Cell Morphology

This research focuses on the segmentation and classification of leukocytes using a deep Convolutional Neural Network (CNN) model. Leukocytes are crucial for diagnosing various diseases through medical image analysis.
The dataset comprises four classes of leukocyte images: monocytes, lymphocytes, eosinophils, and neutrophils.
Leukocyte segmentation involves image processing techniques like background subtraction, noise removal, and contouring.
Steps for isolating leukocytes include:
- Background mask creation
- Erythrocytes mask creation
- Leukocytes mask creation
Data augmentation techniques (brightness/contrast adjustment, flipping, random shearing) enhances the CNN model's generalizability.
The deep CNN model has four convolutional blocks with:
- Eleven convolutional layers
- Eight batch normalization layers
- Eight Rectified Linear Unit (ReLU) layers
- Four dropout layers to capture complex patterns
A publicly available Kaggle dataset of 12,444 images of the four leukocyte types was used.
The framework achieved:
- Accuracy: 97.98%
- Precision: 97.97%
This combined CNN architecture and pre-processing establishes a foundation for advancement in medical image analysis.
Keywords: Leukocytes, Diseases, Leukemia, Deep learning, White blood cells, Segmentation

Introduction

Examining blood cell morphology is fundamental in restorative imaging and diagnostics for understanding and diagnosing illnesses.
Leukocytes are essential in the immune system's defense against toxicities and infections.
However, analyzing leukocyte morphology poses challenges to traditional diagnostic strategies.
Leukocyte division and classification are critical in medical image investigation, especially in hematology.
Leukocytes (white blood cells) play an essential part in guarding the body against infections and maladies as basic components of the resistant framework.
Precise identification and classification of leukocytes contribute substantially to diagnostic processes and the early detection of various wellbeing conditions.
The process of leukocyte division involves confining individual white blood cells within complex microscopic images.
This task is challenging due to the variety of cell shapes, sizes, and potential overlap in thick image regions.
The coming of profound learning, particularly CNNs, has led to a noteworthy jump within the exactness of leukocyte division.
Manual WBC counting is time-consuming and requires medical experts. A framework is needed to automatically analyze white blood cells through division and classification.
Division of leukocytes from blood cell images extracts vital highlights for processing.
This research explores the division and classification of leukocytes leveraging profound learning inside the space of restorative image examination.
Traditional methods for classifying leukocytes in blood cell images face challenges:
- Inconsistencies in human interpretation due to observer experience and training.
- Labor-intensive manual classification requiring trained personnel, making it time-consuming and costly.
- Limited scalability of manual classification to large datasets.
- Artifacts or overlapping cells in blood cell images.
Deep learning offers a solution by automatically learning features from large datasets, reducing reliance on human interpretation and manual feature extraction.
The research focuses on image processing-based segmentation and deep learning-based classification of leukocytes.
Major research offerings:
1. Leukocyte segmentation is achieved through image processing techniques, including background subtraction, noise removal, and contouring.
2. Bounding boxes are created around the segmented leukocyte region and are further cropped without taking background region and erythrocyte region in consideration. Isolated leukocytes are then subjected to data augmentation, including brightness and contrast adjustment, flipping, and random shearing, to improve the generalizability of the CNN model.
3. A deep convolution neural network based model having four convolutional blocks consisting of eleven convolutional layers, eight batch normalization lay- ers, eight Rectified Linear Unit (ReLU) layers, and four dropout layers has been proposed to capture complex patterns for classification of leukocytes into its four different classes such as monocytes, lympho- cytes, eosinophils, and neutrophils.
Rest of the paper structure:
- Literature review
- Proposed work
- Results and discussion
- Conclusion and future work

Literature review

Significant study is underway to adopt a novel way for automatically segmenting WBCs.
Leukocyte segmentation is time-consuming due to the large range of cell shapes and imaging conditions.
Leukocytes can be studied using several automated technologies, as indicated by the literature.
Researchers are still trying to develop a system that can automatically segment leukocytes with higher accuracy in the shortest amount of time.
Stepwise averaging method using interval-valued fuzzy sets was used to separate the nucleus from the cytoplasm in blood smear images.
Authors attained more than 95% accuracy in the separation of the nucleus and the cytoplasm by employing random forest classification.
Leukemia diagnosis is predicted and diagnosed using classification algorithms based on leukocyte segmentation obtained from two separate blood smears utilizing the CMYK color space.
- Classification accuracy of 86.67% was achieved using support vector machines.
- The Bayes classifier-based method produced an overall accuracy of 80.88%.
Application of CNN classifiers was examined for identification of lymphocyte image cells.
SVM and deep learning are used to classify abnormal blood cell images.
A new technique based on the thresholding segmentation technique for the segmentation of leukocytes from blood cell images was presented.
Complete end-to-end system for CNN to recognize the various WBCs classes was provided.
Two transfer learning techniques incorporate this mostly for the identification of WBCs.
An excellent approach for distinguishing normal WBCs was developed.
- Linear Discriminant Analysis (LDA) was used to identify the textural characteristics and reduce the dimensionality of the dataset.
Segmentation of leukocytes with higher accuracy in the shortest amount of time is necessary.
Therefore, a precise model is required for the segmentation of leukocytes so that manual counting of leukocyte can be replaced with an intellectual mechanism.
Here, in this study, an innovative deep learning-based model with advanced accurateness is proposed for cell segmentation, feature extraction, and classification of leukocyte that can replace traditional ways of counting leukocyte.
Although literature review methods can be used to generate good classification engines, they still have some drawbacks.
Traditional machine learning methods need to extract features manually.
The acquisition of features mainly depends on the designer’s prior knowledge.
This feature extraction method is difficult to make full use of the information contained in the image, and will increase the designer’s workload.
The deep learning algorithm effectively solves this problem.
It can automatically learn the effective features of the image.
Deep learning algorithms such as deep residual network also have good performance in image classification tasks.
However, these neural network classification algorithms cannot fully utilize some features of the image that have a long-term dependency relationship with image labels, and thus these classification methods cannot classify cell images like people with memory.
Moreover, it is extremely hard to obtain a sufficient number of annotated and labelled images that can be used to train deep models in a given biomedical domain.
There may be slight differences between two given images from a biomedical or medical area, and this could mean that the two images may indicate two different types of diseases.
Here, an image processing based technique is introduced for segmentation and fuse it with a convolutional neural network to perform the task of blood cell image classification.

Proposed work

In this pipeline of the proposed work, leukocyte isolation by image segmentation is performed using various steps that are discussed in the following sections.
The task of automatically classifying leukocytes in blood images is challenging due to the variability in their appearance.
The figure breaks down leukocyte classification into several stages.
First, the input dataset is preprocessed.
- The pre-processing workflow consists of three pivotal stages: the creation of Background Masks, Erythrocytes Masks, and Leukocytes Masks.
- Leukocyte segmentation is accomplished using image processing methods, encompass- ing procedures such as background subtraction, noise elimination, and contouring.
- Leukocytes undergo data augmentation techniques, which involve adjustments in brightness and contrast, flipping, and random shearing.
- These augmentation processes aim to enhance the overall adaptability and robustness of the CNN model.
Next, the preprocessed data is fed into the convo- lutional neural network.
- CNNs are a type of artificial neural network that are well-suited for image recogni- tion tasks.
- The CNN in the figure consists of four con- volutional blocks, each containing convolutional layers, a max pooling layer, and a ReLU layer.
  - Convolutional layers extract features from the data, max pooling layers reduce the dimensionality of the data, and ReLU layers intro- duce non-linearity.
  - The stride value for convolution and pooling layers is zero and no padding is applied in these layers.
- After each convolutional block, there is a batch normalization layer, which helps to improve the training speed and stability of the network.
Following the convolutional blocks, there are several fully connected layers.
- These layers take the output from the convolutional blocks and map it to the final class labels (neutrophil, eosinophil, lymphocyte, monocyte).
Finally, the model is evaluated using metrics such as pre- cision, recall, F1 score, and accuracy.
Overall, the convolutional neural network architecture in the figure provides a comprehensive approach to clas- sifying leukocytes in blood images.
By segmenting the leukocytes, preprocessing the data, and using a CNN with appropriate layers, the model can achieve high accu- racy in classifying the different types of leukocytes.
The proposed model is analysed using the Google Colab plat- form with Python.

Input dataset

To validate the proposed model, blood samples have been collected from an online source.
Blood sample images are collected from a publicly available dataset from Kaggle Mooney et al. [20].
The dataset contains four leukocyte types named Neutrophil (NE), Eosinophil (EO), Lympho- cyte (LM), and Monocyte (MN)
It consists of a total of 12,444 blood sample images out of which 3144 images belong to EO class, 3139 to LM, 3132 to MN and 3171 to NE class.
The splitting ratio of 80:20 is used for training and testing of the model according to which the total count of training images is 9955 and the count of testing images is 2489.
The size of the input image is 256 * 192.

Leukocyte isolation by image segmentation

A blood cell image consists of three parts.
First, the image undergoes background subtrac- tion to remove erythrocytes and other non-leukocyte elements.
Then, a noise removal step cleans up the image.
Next, the leukocytes are segmented from the background using a mask creation process.
Finally, the design corners are removed, leaving behind the iso- lated leukocytes.
Here in the blood cell image, gray part is the background whereas, the leukocyte is shown in blue color and the dark brown part is the erythrocyte.
For the segmentation of images and for isolating leuko- cytes, it is necessary to know the position of leukocytes in the training samples.
For the detection of leukocytes, the processing is done by creating a mask for the back- ground detection.
After that, a mask is created for the detection of erythrocytes.
Simple masks are created for filtering out the background area to extract leuko- cytes.

Background mask creation

In blood cell images, the background area does not con- tain erythrocytes as well as leukocytes and it is gray in color.
The gray pixel in the image consists of the red, blue, and green components in the majority.
For creating back- ground mask, threshold ( $threshold_1$ ) is calculated using Otsu thresholding method.
So, the background mask is created using the following Eq. (1)
Background_mask = (img[:, :, 0] > threshold1)&(img[:, :, 1] > threshold1)&(img[:, :, 2] > threshold_1)
In this equation, red component i.e. img[:, :, 0], green component i.e. img[:, :, 1] and blue component i.e. img[:, :, 2 all are having pixel value greater than threshold_1 whose value comes out to be 182 here.
All background pixels satisfying the above equation are high- lighted in blue color.

Erythrocytes mask creation

In blood cell image erythrocyte is shown in light brown color.
So, to extract the erythro- cyte area from the blood cell images, the erythrocyte mask is created with a red component where the red pixel value is greater than the blue pixel value or the blue pixel value should be less than threshold value which comes out to be 150 calculated using Otsu thresholding method.
This function will detect the erythrocytes. So, the erythrocyte mask is created using the following Eq. (2)
$Erythrocyte_mask = (img[:, :, 2] < threshold_2)|(img[:, :, 0] > img[:, :, 2])$
In this equation, blue component i.e. img[:, :, 2] will always be less than red component i.e. img[:, :, 2].
Here, blue component i.e. img[:, :, 2] is less than 150 or red component i.e img[:, :, 2] is greater than blue component i.e. img[:, :, 2].
All erythrocyte pixels satisfying the above equation are highlighted in red color.

Leukocytes mask creation

The leukocyte mask is created that does not include a background mask and erythrocyte mask.
So, the leuko- cyte mask is created using the following Eq. (3)
Leukocyte_mask =∼ is_background(img)& ∼ is_erytrocyte(img)
From Fig. 7 (a), (b), and (c) it can be seen that the leukocyte mask is shown in white color whereas, the background of the blood sample image is black.

Noise removal in leukocyte mask

To remove the noise from the leukocyte mask morphologi- cal operations are used.
From Fig. 7 shown in the last sec- tion, it can be seen that noise is present in leukocyte mask images.
To get rid of the little scraps and make the masks rounder, opening morphological operation is used i.e. ero- sion followed by dilation.
The equations of dilation, ero- sion, and opening are given in Eqs. 4, 5, and 6 respectively.
The equation of dilation operation is $P ⊕ Q = {R|[(Q∧_z ) ∩ P]ǫP}$
- Here, P is the image and Q is the structuring element.
- (Q ∧ _z) it means to take a reflection of Q about its origin and shift it by R.
- Therefore, dilation of P with Q is a set of all displacements, R such that (Q ∧ _z) and P overlap by at least one element.
The equation of erosion operation is $P ⊖ Q = {R|(Q ∧ _z)ǫP}$
- Here, the erosion of P by Q is a set of all points that Q, shifted by R is a subset of P that is Q is entirely contained within P.
- Erosion reduces the number of pixels from the object boundary.
The equation of opening operation is $P ◦ Q = OPEN(P, Q) = (P ⊖ Q) ⊕ Q$
- Morphological opening of an image is erosion followed by dilation.

Bounding boxes of the selected leukocytes

In this step, the mask having the highest area in the image sample is marked as leukocyte with the bounding boxes and the outline color is made red.

Image cropping to get isolated leukocytes

In this step, the selected leukocyte image is cropped to an image size of 128 * 128.

Data augmentation

Overfitting is a challenge for large neural networks in biological applications since only a relatively small num- ber of datasets are available.
The dataset is therefore expanded to achieve greater performance.
Various operations such as rotation, flipping, contrast, brightness, and random shear are applied to each image input during data augmentation.
Thereby increasing the overall training image count is a good thing.
Images are then divided into two sets: one for training and another for testing reasons.
The CNN model is used to train the model.

Deep convolution neural network architecture for segmented leukocyte

The convolutional neural network (CNN) architecture is a four-block deep learning model designed to classify leukocytes in blood smear images.
Convolutional layer applies filters (learned kernels) that slide across the input image, extracting features like edges and textures.
The number of filters increases progressively through the blocks, allowing the network to study progressively complex features.
ReLU activation layer introduces non-linearity into the network, enabling it to learn complex relationships between features.
Batch normalization layer reduces internal covariate shift, sta- bilizing the training process and improving the model’s generalizability.
Following these convolutional blocks, a global average pooling layer captures spatial information from the fea- ture maps, producing a fixed-size output that’s suitable for feeding into the final densely connected output layer.
This final layer has four output neurons, corresponding to the four leukocyte classes i.e. neutrophil, eosinophil, lymphocyte, and monocyte.
The first convolutional block consists of 3 convolution layers with filter size of 3 * 3 and a total of 32 filters, 2 batch normalization layers, 1 dropout, and 1 max pool layer with 2 * 2 filter size with 2 ReLU layers.
The second convolutional block contains 3 convolution layers with filter size of 3 * 3 and a total of 64 filters, 2 batch normalization layers, 1 dropout, and 2 ReLU lay- ers.
The third convolutional block contains two convo- lution layers with filter size of 3 * 3 and 128 total filters, two batch normalization layers and two ReLU layers.
The fourth convolutional block consists of three convo- lution layers, two batch normalization layers, two ReLU layers and one dropout layer.
After the 4 convolutional blocks, the global average pooling layer is attached with two dense layers.
In the last step, the blood sample image is classified into the four classes named as Neu- trophil, Eosinophil, Monocyte and Lymphocyte.
Overall, this CNN architecture employs a step-wise approach, progressively extracting higher-level features from the input image to ultimately achieve accurate leukocyte classification.
The rationale behind using a deep Convolutional Neu- ral Network (CNN) for leukocyte classification lies in its ability to automatically learn hierarchical features from raw data.
Deep CNN has more convolution layers to cap- ture more complex features in the leukocyte images to learn deep abstract information of the data, potentially leading to better performance.
CNNs are well-suited for this task because they can capture spatial hierarchies of features in images through their convolutional layers, which apply filters to detect patterns at different spatial scales.
The proposed approach has total parameters as 134,853, with 134,085 as Trainable parameters and 768 as Non-trainable parameters.

Results and discussion

This segment shows the results attained using the pro- posed deep CNN model integrated with image process- ing methods.
To analyse the presentation of proposed model, it is simulated in three ways.
In the first case, model is simulated in such a way that neither segmenta- tion of leukocyte is performed nor augmentation is done.
In the second case, both segmentation as well as augmen- tation are performed in the proposed model.
Thereafter, the classification results of these two cases are compared to analyse the best case.
The best case is further com- pared with the state-of-art models.
The segmentation result analysis cannot be performed for the proposed model because the ground truth for segmentation mask is not provided for this dataset.
Hence, the performance of model is analysed in terms of classification accuracy, not in terms of segmentation accuracy.
The model has been analyzed based on train loss, train accuracy, validation loss, validation accuracy, precision, sensitivity, F1-score, and accuracy.
The fine-tuning of the model is performed using diverse hyperparameters like Adam optimizer, batch size value 32, and epochs.

Results of proposed model without segmentation and augmentation

Here, the proposed model is applied directly on the origi- nal images without data pre-processing techniques.
The leukocyte region is not segmented and cropped from the original images.
Moreover, no data augmentation technique is applied on the original or cropped images in this case.
An analysis is performed based on confusion matrix parameters, Cohen’s Kappa score, training, and validation accuracy and loss curves.
the model is simulated using the 32 batch size value and a total of 12 epochs.
Figure 13 shows the training and validation loss and accuracy curves.
Figure 13 (a) shows the accuracy curves and it can be analyzed from the figure that the value of training accuracy is approximately 80% and the value of validation accuracy is approximately 55%.
Similarly, Fig. 13 (b) shows the loss curves and it is observed that the value of validation loss is between 0-2.
Figure 14 (a) shows the confusion matrix of the pro- posed model on batch size 32 and Adam optimizer.
In a matrix, the number of images classed by a given model can be determined by the diagonal values of the matrices.
Figure 14 (b) shows the Cohen’s Kappa score on the test set.
Cohen’s Kappa statistic is utilized to determine how well two raters or judges agree on the classification of an item into two distinct groups.
The Cohen’s Kappa can be calculated as: $K = (PO − PE)/(1 − PE)$
- Here, PO is the Relative observed agreement among raters and PE is the Hypothetical probability of chance agreement.
- The Cohen’s kappa score is a statistical meas- ure used to assess the agreement between two raters or classifiers.
- It considers the agreement that would be expected by chance and then normalizes the observed agreement by this value.
- This normalization accounts for the possibility of random agreement, providing a more robust measure of agreement.
- The kappa score ranges from -1 to 1, where 1 indicates perfect agreement, 0 indi- cates agreement equivalent to chance, and values less than 0 indicate agreement worse than chance.
- It is com- monly used in the evaluation of classification models, particularly in cases where the classes are imbalanced.
- The value of Cohen Kappa score is 0.51 on the test set.
From the confusion matrix, various considerations such as sensitivity, precision, F1-score and accuracy are cal- culated and shown in Table 1.
The overall accuracy value of the model without segmentation and augmentation obtained is 79%.
In case of precision, the model is perform- ing best for lymphocyte type with the value as 83%, whereas in case of sensitivity and F1-score, the model is showing best value for neutrophil as 90% and 83% respectively.

Results of the proposed model with segmentation and augmentation

Here, the proposed model is applied on the segmented images.
The leukocyte region is segmented and cropped from the original images.
Also, data augmentation tech- nique is applied on the segmented or cropped images in this case.
An analysis is performed based on confusion matrix parameters, training, and validation accuracy and loss curves, Cohen’s Kappa score.
Figure 15 displays the validation and training loss, accuracy curves for a total of 12 epochs.
Figure 15 (a) shows the accuracy curves and it can be analyzed from the figure that the value of training accuracy is near 92%.
Similarly, Fig. 15 (b) shows the loss curves and the value of the loss is constant after the 2nd epoch.
The loss value is declin- ing with the increase in the epoch value.
Figure 16 (a) shows the confusion matrix of the proposed model on batch size 32 and Adam optimizer.
Figure 16 (b) shows the Cohen’s Kappa score on the test set.
The value of Cohen’s Kappa score is 0.625 that is ranging between 0-1.
From the confusion matrix, various considerations such as sensitivity, precision, F1-score and accuracy are considered and shown in Table 2.
The overall accuracy value of the model with segmentation and augmentation obtained is 91.18%.

Ablation analysis for the proposed model

The performance comparison of proposed model is simulated in two ways.
In the first way, the proposed model is applied directly on the original images without data pre-processing techniques.
The leukocyte region is not segmented and cropped from the original images.
Moreover, no data augmentation technique is applied on the original or cropped images in this case.
In this case the proposed model is simulated on the various perfor- mance factors such as accuracy, precision, sensitivity and F1 score.
In the second way, the proposed model is applied on the segmented images.
The leukocyte region is segmented and cropped from the original images.
Also, data augmentation technique is applied on the seg- mented or cropped images in this case.
Figure 17 shows the ablation analysis of the proposed approach.
Without segmentation and augmentation, the model achieved moderate precision ranging from 65% to 83% for different leukocyte types.
However, with the inclusion of segmentation and augmentation, there was a substantial improvement in precision, with values rang- ing from 97.91% to 98.03%.
This significant increase demonstrates the effectiveness of segmentation and augmentation in enhancing the model’s performance, particularly in accurately classifying neutrophils, eosin- ophils, monocytes, and lymphocytes in human blood images.
The Fig. 18 showcases the sensitivity values of a pro- posed model for leukocyte classification, comparing out- comes without segmentation and augmentation to those with these techniques.
Without segmentation and aug- mentation, the model exhibited varying sensitivities for different leukocyte types, ranging from 40% to 90%.
How- ever, with segmentation and augmentation, there was a notable improvement in sensitivity across all classes, with values ranging from 97.87% to 98.18%.
This enhance- ment highlights the effectiveness of segmentation and augmentation in improving the model’s ability to cor- rectly identify neutrophils, eosinophils, monocytes, and lymphocytes in human blood images, particularly evident in the substantial increase in sensitivity for eosinophils and monocytes.
The Fig. 19 displays the F1-scores of a proposed model for leukocyte classification, comparing results without segmentation and augmentation to those with these techniques.
Without segmentation and augmen- tation, the model achieved moderate F1-scores ranging from 50% to 83% for different leukocyte types.
How- ever, with segmentation and augmentation, there was a significant improvement in F1-scores across all classes, with values ranging from 97.93% to 98.05%.
This substantial increase demonstrates the effectiveness of segmentation and augmentation in enhancing the mod- el’s ability to balance precision and recall, particularly evident in the remarkable improvement in F1-scores for eosinophils, monocytes, and lymphocytes.

Comparison with state-of-art

The evaluation of the proposed model with state-of-art methods is shown in Table 3.
It illustrates that the pro- posed framework has been successfully utilizing seg- mentation and classification both, whereas the current state-of-the-art techniques has worked on classification only.

Conclusion and future work

Leukocyte (WBCs) are in charge of the immune system in human blood that protects the human body from diseases, parasites and bacteria.
The manual count of WBCs and its classification into its four types such as lymphocytes, monocytes, eosinophils, and neutro- phils is common practice among hematopathologists to identify leukemia.
But this is a time consuming and laborious task that necessitate the assistance of medical professionals.
Moreover, due to the intensity variance and imaging conditions of blood images, WBC segmen- tation is challenging task.
Thus, there is necessity for a computer-aided system that can segment and classify leukocytes automatically to analyze human blood.
In this work, a deep learning based model has been pro- posed for segmentation and classification of leukocytes into its four different types.
The proposed model is esti- mated independently on each of the four types of leu- kocyte images, and it shows good precision, sensitivity, accuracy, and F1 score.
Overall, this method achieves a higher classification accuracy of 97.98% than any other state-of-art techniques.
The proposed deep learning model for leukocyte seg- mentation and classification has significant practical implications for medical diagnostics and patient care.
By automating the laborious and time-consuming task of manual leukocyte counting and classification, the model can improve the efficiency of hematopathologists and reduce diagnostic errors.
This could lead to faster and more accurate diagnoses of leukemia and other blood- related disorders, ultimately improving patient outcomes.
In future work, the proposed model’s performance can be enhanced by incorporating more advanced deep learning architectures and algorithms, such as atten- tion mechanisms or graph convolutional networks, to improve feature extraction and classification accuracy.
Furthermore, the model’s capabilities can be extended to handle additional leukocyte subtypes or abnormalities, thereby increasing its applicability in a broader range of hematological analyses.
Finally, we intend to validate the proposed model on a larger and more diverse dataset to ensure its robustness and generalizability.