Study Notes: Sensitivity Analysis of Vertical Split Learning on Medical Datasets

Core Research Framework and Objectives

• Vertical Split Learning ($VSL$) enables multiple organizations holding complementary feature sets to collaboratively train shared models without the exchange of raw data. This study investigates the robustness of $VSL$ outside idealized conditions, focusing on four primary deployment axes:
  • Participating client count ($C$).
  • Feature partitioning strategies across clients.
  • Additive Gaussian measurement noise ($\sigma$).
  • Missing Completely at Random ($MCAR$) feature dropout.
• The research objective is to provide evidence-based deployment guidance to determine which factors drive performance variation, utilizing a systematic statistical lens including linear mixed-effects models ($LMMs$) and effect-size estimation.

Technical Foundations of Vertical Split Learning

• $VSL$ is a variant of Vertical Federated Learning ($VFL$) where different parties own different feature subsets for the same samples.
• The learning task is distributed between client-side sub-networks and a central server-side model.
• Clients process only their local features and transmit intermediate representations to the server, preserving data privacy while enabling joint training.
• Deployment challenges include flux in institutional partnerships, variable informativeness of held features, sensor/entry noise, and incomplete records (missing fields).

Experimental Setup and Configuration Grid

• The study utilized thirty-one experimental configurations ($E1$–$E29$, plus supplementary points $E4b$ and $E12b$).
• Baseline Configuration ($E3$):
  • Number of clients: $5$.
  • Partitioning: Balanced.
  • Noise level ($\sigma$): $0.00$.
  • $MCAR$ missing rate: $0.00$.
• Datasets:
  • Cervical Cancer: Target is Normal/Abnormal; Batch size $35$; Learning rate $0.008320$; Total neurons $[10, 65, 22]$.
  • Heart (TenYearCHD): Target is Presence/Absence; Batch size $127$; Learning rate $0.001182$; Total neurons $[71, 126]$.
  • Diabetes: Target is Diabetic/Non-Diabetic; Batch size $60$; Learning rate $0.006554$; Total neurons $[15, 50]$.
• Preprocessing Pipeline:
  • Mean imputation (where required).
  • One-hot encoding of categorical variables.
  • Feature normalization.
  • Class-imbalance correction via $SMOTE$.
• Factor Levels and Ranges:
  • Clients ($C$): $\{2, 3, 5, 8, 10\}$.
  • Partitioning Strategies: Balanced, Random Balanced, Moderately Imbalanced, Highly Imbalanced.
  • Gaussian Noise ($\sigma$): $\{0.00, 0.05, 0.10, 0.20, 0.30\}$.
  • $MCAR$ Missingness: $\{0.00, 0.10, 0.20, 0.30\}$.

Statistical Analysis Pipeline

• Each configuration was executed five times to mitigate the influence of random initialization and stochastic optimization.
• Metrics recorded: Accuracy, $F1$-score, Area Under the ROC Curve ($AUC$), communication cost (approximate bytes exchanged), and training time ($s$).
• Statistical Tools:
  • Linear Mixed-Effects Models ($LMMs$).
  • Likelihood-Ratio Tests ($LRTs$).
  • Non-parametric Wilcoxon signed-rank pairwise comparisons.
  • Rank-biserial effect-size estimation ($r$).
  • Benjamini–Hochberg ($BH-FDR$) correction for multiple comparisons.

Sensitivity Analysis Results: Structural Factors

• Structural perturbations (client count and feature distribution) dominate performance variance compared to data-quality perturbations (noise and missingness).

Impact of Client Count ($C$)

• Fragmentation of the feature space affects local representation quality and communication volume.
• Average Effect Size: $f^2 = 0.5222$ ($95\%\,CI: 0.25$–$0.80$), interpreted as Large.
• Cervical Dataset: Non-monotonic behavior. Performance peaked at $C = 3$ (Accuracy $0.814$, $F1\,0.259$, $AUC\,0.800$). Performance dropped significantly at $C = 8$ (Accuracy $0.419$).
• Heart Dataset: Optimal at $C = 2$ (Accuracy $0.653$, $F1\,0.322$, $AUC\,0.647$). Increasing to $C = 5$ (baseline) reduced metrics significantly ($AUC\,p = 5.50 \times 10^{-4}$, rank-biserial $r = -1.0$).
• Diabetes Dataset: Comparative indifference; $C = 2$ reached the best $F1\,(0.784)$ and $AUC\,(0.842)$, but accuracy remained stable until $C = 8$ (Accuracy $0.627$).
• Computational Cost: Training time increases with $C$; for Cervical, time rose from $3.14\,s$ (baseline) to $6.66\,s$ at $C = 10$.

Impact of Feature-Distribution Strategy

• Average Effect Size: $f^2 = 0.2872$ ($95\%\,CI: 0.14$–$0.43$), interpreted as Medium.
• Finding: Balanced partitions are not a universally safe default. Clustering informative features can lead to sharper local embeddings.
• Heart Dataset: Imbalanced partitions significantly outperformed balance. Moderately Imbalanced yielded Accuracy $0.593$ and $AUC\,0.642$ ($p = 3.95 \times 10^{-4}$ vs. Balanced).
• Diabetes Dataset: Maximum gains occurred here. Highly Imbalanced achieved Accuracy $0.850$ and $AUC\,0.909$ compared to baseline accuracy of $0.735$.
• Cervical Dataset: Highly Imbalanced partitioning improved accuracy but caused $F1$ to collapse ($0.075$), indicating biased models against minority classes when features are too concentrated.

Sensitivity Analysis Results: Data Quality Factors

Impact of Additive Gaussian Noise ($\sigma$)

• Average Effect Size: $f^2 = 0.0244$ ($95\%\,CI: -0.01$–$0.05$), interpreted as Small.
• Heart Dataset: Extreme stability in accuracy ($0.461$ at baseline vs. $0.459$ at $\sigma = 0.30$). Although $AUC\,LRT$ was significant ($p = 5.00 \times 10^{-6}$), the practical effect was negligible ($f^2 = 0.0009$).
• Diabetes Dataset: Negligible sensitivity ($f^2 < 0.01$) across the range tested.
• Cervical Dataset: Showed non-monotonic behavior likely matching optimization variance rather than a true noise effect.

Impact of $MCAR$ Missing Features

• Average Effect Size: $f^2 = 0.0163$ ($95\%\,CI: 0.01$–$0.03$), interpreted as Negligible.
• Heart Dataset: Exhibited a monotonic and statistically significant accuracy decline: from $0.461$ ($0\%$) to $0.427$ ($30\%$). Significant decline started from $20\%$ missingness ($p = 5.28 \times 10^{-4}$).
• Diabetes Dataset: Effectively flat performance up to $30\%$ missingness ($f^2 < 0.02$).

Interaction Effects and Operational Considerations

• Client Count $\times$ Noise interaction: Final models did not support significant interaction effects (Cervical Accuracy $p = 0.968$; Diabetes Accuracy $p = 0.575$). Perturbations are additive rather than synergistic.
• Combined Stress ($E29$): Crossing $\sigma = 0.20$ with $20\%$ $MCAR$ did not produce multiplicative collapse. Resulting accuracies were intermediate between single-factor perturbations.
• Communication Efficiency:
  • On Heart, $C = 2$ is the most efficient high-performing setting ($9.74 \times 10^{5}\,\text{bytes}$).
  • On Diabetes, Highly Imbalanced partitioning improves performance but increases communication cost to $1.93 \times 10^{5}\,\text{bytes}$ ($1.86\times$ baseline).

Comparative Sensitivity Ranking and Final Conclusions

• Final Hierarchy of Factors (by $f^2$):
  1. Clients ($0.5222$)
  2. Distribution ($0.2872$)
  3. Gaussian Noise ($0.0244$)
  4. $MCAR$ Missingness ($0.0163$)
• The ratio between structural factors and data-quality factors regarding performance influence is approximately $32:1$.
• Practical Takeaways:
  • Selection of client count and partition strategy are first-order design decisions requiring empirical validation.
  • Denoising and imputation are secondary concerns for moderate corruption levels ($\sigma \le 0.30, MCAR \le 30\%$"Title": "Study Notes: Sensitivity Analysis of Vertical Split Learning on Medical Datasets"}