Masked-Attention Transformers for Surgical Instrument Segmentation

MATIS: MASKED-ATTENTION TRANSFORMERS FOR SURGICAL INSTRUMENT SEGMENTATION

Authors

  • Nicolas Ayobi
  • Alejandra Pérez-Rondón
  • Santiago Rodríguez
  • Pablo Arbelaez
  • Affiliation: Center for Research and Formation in Artificial Intelligence, Universidad de los Andes, Colombia

Abstract

  • MATIS introduces a two-stage, fully transformer-based method designed for surgical instrument segmentation.
  • Employs a modern pixel-wise attention mechanism to enhance segmentation accuracy.
  • Utilizes a masked attention module to generate and classify fine instrument region proposals.
  • Incorporates long-term video-level information using video transformers to improve temporal consistency and mask classification.
  • Validation conducted on public benchmarks: Endovis 2017 and 2018.
  • Experimental results indicate that MATIS outperforms state-of-the-art methods and enhancing temporal consistency leads to further performance boosts.
  • Source code is available on GitHub: https://github.com/BCV-Uniandes/MATIS.

Index Terms

  • Instrument Segmentation
  • Robot-Assisted Surgery
  • Computer-Assisted Interventions
  • Transformers
  • Deep Learning

1. Introduction

  • Instrument segmentation is necessary for understanding surgical scenes, which supports the development of computer-assisted surgical systems for:
    • Instrument tracking [1, 2]
    • Pose estimation [3]
    • Surgical phase estimation [4]
  • Major challenges:
    1. Accurate identification and segmentation of surgical instruments.
    • High similarity between different instrument types.
    • Class imbalance in datasets.
    1. Incorporating temporal information across video frames for consistent recognition and enhanced overall understanding of surgical procedures.

2. Background

Previous Methods

  • Convolutional Neural Networks (CNNs) are predominantly used for surgical instrument segmentation.
    • Initial segmentation efforts used Fully Convolutional Networks (FCN) [5, 6].
    • Following the Endovis challenges, methods adapted FCN models for semantic segmentation of different instruments across frames [7, 8, 9, 10].
    • Utilization of additional data sources like:
    • Optical and motion flow [11, 12]
    • Stereoscopic information [13]
    • Saliency maps [14]
    • Emerging methodologies included:
    • Weak supervision [11, 15]
    • Domain adaptation [16]
    • Pose estimation [3]
    • Kinematic data [17, 18]
    • Image generation [19]
  • Many methods rely on per-pixel classification, potentially hindering spatial consistency and neglecting the multi-instance nature of segmentation.
    • ISINet [20] proposed a region classification style for instrument segmentation.

Advances in Transformers

  • Vision Transformers (ViTs) have emerged as leading architectures for various computer vision tasks [23, 25, 26].
  • First transformer integration for instrument segmentation involved CNNs with Swin transformers as a backbone for Mask-RCNN.
  • New architectures such as DETR [25], MaskFormer [26], Deformable DETR [29], and Mask2Former [30] have demonstrated superior segmentation capabilities with efficient training procedures.
  • Video transformers have also outperformed traditional CNN methods but need more focused applications for surgical instrument segmentation.

Challenges in Past Methods

  • Existing CNN strategies often used optical flow instead of leveraging comprehensive video-level understanding.
  • TraSeTr [28] lacks long-term tracking capabilities while STSwinCL [33] employs a pixel-classification modality, leading to identical classifications for varying instances.

3. MATIS Architecture

Overview

  • Proposed architecture: Masked-Attention Transformers for Instrument Segmentation (MATIS).
  • Key Features:
    • A two-stage mask classification scheme.
    • Hierarchical processing to generate and classify region proposals for each instrument instance in every frame.

MatIS Components

3.1 Masked Attention Baseline
  • Utilizes Mask2Former [30], which is designed to produce a fixed-size set of class probability-binary mask pairs using learnable queries.
  • Extraction of region proposals and corresponding segments from Mask2Former is optimized for instrument segmentation.
  • Custom priors address class frequency imbalance, enhancing selection for the top-k scoring regions/instances per class.
3.2 Temporal Consistency Module
  • Adopts a Multi-Scale Vision Transformer (MViT) [32] for enhanced video analysis, which computes global spatio-temporal features.
  • Modified classification head utilizes per-segment embeddings to maintain localized features.
  • Pooled temporal features are processed through a multilayer perceptron (MLP) before concatenation and classification.
  • Additional supervision implemented using a MLP to predict instrument presence, applying binary cross-entropy loss alongside the classification loss.
3.3 Implementation Details
  • Pretraining performed using Mask2Former’s available instances from MS-COCO [35].
  • Trained for 100 epochs on NVIDIA GPUs using an ADAMW optimizer.
  • Swin Small (SwinS) [23] provides the architecture backbone, balancing performance and parameters.
  • Temporal module trained for 20 epochs, validating the effectiveness of window size and strides.

4. Experimental Results

4.1 Datasets and Evaluation Metrics

  • Evaluated on publicly available datasets: Endovis 2017 [7] and Endovis 2018 [8].
  • Metrics used for assessment include:
    • Mean Intersection over Union (mIoU)
    • Intersection over Union (IoU)
    • Mean Class Intersection over Union (mcIoU)

4.2 Validation of Performance

  • MATIS Frame: Baseline without temporal consistency, demonstrating superior performance across all metrics compared to historical methods.
  • The individual class metrics especially highlighted performance gains in well-represented training categories, outperforming previous models by significant margins.

4.3 Upper Bound Assessments

  • The upper bounds, including inferred and total matching annotations, indicate MATIS's potential for high segmentation accuracy.
  • Identified that inaccuracies emerged primarily from classification, not mask generation.

4.4 Temporal Module Enhancements

  • The addition of the temporal consistency module positively influences evaluation metrics, particularly for the Endovis datasets.
  • Addressed common misclassifications, although some classes may experience variability due to complexity in recognizing instances over time.

5. Conclusions

  • MATIS offers an innovative architecture for surgical instrument segmentation, effectively using global temporal information to improve segmentation quality.
  • Main contributions:
    • Employs masked attention for fine instance segmentation.
    • Combines localized features with long-term temporal data to enhance classification.
    • Sets new performance benchmarks in surgical scene understanding.
  • Ethical compliance confirmed, using revised publicly available datasets without additional approvals.

Acknowledgments

  • Acknowledgements were made for the scholarship support given to authors from UniAndes-DeepMind for the years 2022 and 2023.

References

  • Provided extensive reference list [1-36] with academic papers discussing various relevant contributions to surgical instrument segmentation, methodology, and advancements in computer vision techniques.