FRESHNet: Study Notes – 3-Axis Stem-Aware 3D Apple Detection

Abstract & Citation Details

• Paper title: “FRESH: Fusion-Based 3D Apple Recognition via Estimating Stem Direction Heading”

• Authors: Geonhwa Son, Seunghyeon Lee, Yukyung Choi

• Affiliation: Sejong University, Republic of Korea (AI & Robotics / Convergence Engineering for Intelligent Drone)

• Journal: MDPI Agriculture, 2024, Vol. 14, Article 2161

• DOI link: https://doi.org/10.3390/agriculture14122161

• Editorial timeline:

  • Received: 25 Oct 2024

  • Revised: 22 Nov 2024

  • Accepted: 24 Nov 2024

  • Published: 27 Nov 2024

• License: Creative Commons CC-BY 4.0 (open-access – unrestricted use with attribution)

Motivation & Background

• Apples are among the most consumed fruits; quality control is labor-intensive across pruning, monitoring, harvesting.

• Automated systems must not only locate fruit but also know stem direction to avoid damaging fruit during the standard “roll-and-twist” harvest motion.

• Prior computer-vision work:

  • 2D detection/segmentation boosts orchard tasks but ignores 3D stem orientation.

  • Existing 3D apple datasets (Fuji-SfM, PApple_RGB-D-Size) lack 3-axis rotation labels.

  • Prior rotation work [26] handled only planar (2D) rotation → inadequate for true 3-axis twisting.

• Practical implication: Robots need full 3-axis pose to align grippers, cut stems, prune branches.

Key Contributions

• Re-processed PApple_RGB-D-Size to add 3-axis stem-direction labels (roll, pitch, yaw) → first orchard dataset with full orientation.

• Proposed FRESHNet: a real-time multimodal 3D detector that fuses RGB semantics + point-cloud geometry and directly regresses $(x,y,z,w,h,l,\text{roll},\text{pitch},\text{yaw})$.

• Introduced novel “Stem-Direction Loss” that optimises the dot-product between predicted & GT stem vectors for rotation accuracy.

• Achieved SOTA on new dataset: $\text{AP@0.25}=89.56\%$, $\text{AP@0.5}=78.29\%$, beating axis-aligned methods and previous fusion baselines.

Dataset Reprocessing

Source & Rationale

• Base dataset: PApple_RGB-D-Size (RGB + depth, orchard scenes 2018/2020)

• Needs intrinsic calibration; uses SfM to recover camera intrinsics → back-projects depth to dense point clouds.

• Added annotations:

  1. Generate initial 3D box by using depth minimum insidep modal mask, radius from diameter.

  2. Manually adjust box position in labelCloud v1.1.1.

  3. Rotate box so local $x$-axis aligns with visible stem → defines stem vector.

  4. Generate tight 2D box by projecting non-rotated 3D box (avoids loose fit, cf. Fig 3).

Statistics

• Total image–cloud pairs: $3381$

• Split:

  • Train $1995$ (west orchard) – apples $4976$

  • Val $701$ – apples $2052$

  • Test $685$ – apples $2016$

• Average diameter: $\mu=68.82\,\text{mm},\;\sigma^2=0.206\;\text{mm}^2$

• KDE of stem vectors shows dense “upright” cluster but wide angular spread → automation can’t assume vertical stems.

Related 3D Detection Landscape

• Outdoor AV: BEV compresses height – unsuitable for fruit on branches.

• Indoor robotics: full 3D boxes; two categories:

  1. AABB (no rotation)

  2. OBB (1-axis yaw only)

• Lack of 3-axis datasets has stalled rotation-aware algorithms → FRESH work fills gap.

FRESHNet Architecture

• Backbone:

  • 3D: MinkResNet (sparse 3D convolutions) – efficient voxel processing.

  • 2D: ResNet-50 + FPN (pre-trained) – multi-scale RGB features.

• Multimodal Fusion (Section 2.3.1):

  1. Project each point to image plane using camera intrinsics.

  2. Sample FPN feature at that pixel; build sparse tensor $T_{I3D}$.

  3. Transform via $F<em>{I3D}=\text{ReLU}(\text{BN}(\text{Conv}(T</em>{I3D})))$ … (Eq 1).

  4. Fuse with first 3D feature map $F<em>{B3D}$ via element-wise add: $F</em>{\text{fusion}} = F<em>{B3D} + F</em>{I3D}$ (Eq 2).

• Neck: transposed sparse convs to upsample & densify points.

• Detection head: parallel sparse convs output class prob + box + rotation.

Loss Functions

1. Bounding-box DIoU loss (Eq 3):
   $L_{\text{bbox}} = 1-\text{IoU} + \frac{\rho^2(b,\hat{b})}{c^2}$
   • $\rho$ = centre distance, $c$ = diagonal of smallest enclosing box.

2. Stem-direction loss (novel):

   • Convert Euler triple $A=(\text{roll},\text{pitch},\text{yaw})$ to rotation matrix $R$.

   • Stem unit $u_{x}=[1,0,0]$.

   • Stem vectors: $v=u<em>x R\,,\;\hat{v}=u</em>x \hat{R}$.

   • Loss: $L_{\text{stem}} = 1 - (v \cdot \hat{v})\in[0,2]$ (Eq 5).

3. Total:
   $L = \frac{1}{n<em>{\text{pos}}}\bigl(L</em>{\text{cls}} + L<em>{\text{bbox}} + L</em>{\text{stem}}\bigr)$ (Eq 6) – where $L_{cls}$ = focal loss.

Training & Implementation

• Framework: MMDetection3D + Minkowski Engine.

• Hardware: NVIDIA A100 40 GB, Ubuntu 18.04.

• Optimiser: Adam, $\text{lr}=10^{-4}$.

• End-to-end training from scratch.

Evaluation Metrics

• 3D IoU thresholds: $0.25$ & $0.5$.

• Precision $P = \frac{TP}{TP+FP}$ (Eq 7), Recall $R = \frac{TP}{TP+FN}$ (Eq 8).

• AP = area under PR curve (Eq 9); AR = mean recall (Eq 10).

• Rotation error: quaternion distance $\lVert q-\hat{q}\rVert$ (Eq 11) + per-axis angular error.

Experimental Results

Main Comparison (Table 2)

• FRESHNet (image + point, 3-axis):

  • $\text{AP@0.25}=89.56\%$, $\text{AP@0.5}=78.29\%$

  • $\text{AR@0.25}=99.16\%$

  • Inference speed: $13.17\,\text{FPS}$ → fastest among fusion methods.

• Gains over axis-aligned TR3D+FF: +$20.36\%$ AP@0.5; demonstrates rotation awareness matters.

• PR curves: axis-aligned precision drops sharply beyond recall ≈$0.7$; 3-axis methods remain stable.

Ablation on Rotation Loss (Tables 3 & 4)

• Comparing loss variants:

  1. Euler + MSE

  2. Quaternion + MSE

  3. Stem-direction (ours)

• Stem-direction achieves best AP and smallest quaternion distance $\approx0.0071$ (vs $\approx0.013$ for baselines).

• Roll error shrinks from $\sim60^\circ$ → $\sim1^\circ$; pitch/yaw differences minimal (≈$2^\circ$) but overall quaternion superior.

• Qualitative figs (Fig 7): ours aligns box edge colour (stem axis) closely with GT.

Discussion & Implications

• Rotation-aware boxes give tighter fits → higher precision object localisation, critical for robotic grippers.

• Real-time capability (≥$10$ FPS) suits mobile orchard robots.

• Limitation: severe occlusions by leaves/branches still problematic (Fig 9). Future work: occlusion-aware modules, leveraging dataset visibility meta-labels.

• Ethical/operational view:

  • Open-access CC-BY license promotes reproducibility.

  • Potential labour displacement balanced by reduced injury & quality loss; requires inclusive deployment strategies.

Conclusion & Future Directions

• First orchard dataset & detector to couple 3-axis stem direction with 3D apple detection.

• FRESHNet’s fusion + stem-direction loss outperforms SOTA while maintaining speed.

• Envisions fully autonomous pruning, pollination, harvesting robots.

• Next steps: robustness under heavy occlusion, unseen fruit varieties, domain adaptation to other crops (strawberries, tomatoes, melons).

Key Equations (Quick Reference)

• $L_{\text{bbox}} = 1-\text{IoU} + \dfrac{\rho^2}{c^2}$

• $F<em>{I3D}=\text{ReLU}(\text{BN}(\text{Conv}(T</em>{I3D})))$

• $F<em>{\text{fusion}} = F</em>{B3D} + F_{I3D}$

• $L_{\text{stem}} = 1 - (v \cdot \hat{v})$

• $L = \frac{1}{n<em>{pos}}(L</em>{cls}+L<em>{bbox}+L</em>{stem})$

• $P=\frac{TP}{TP+FP},\;R=\frac{TP}{TP+FN}$

• $\text{QuaternionDistance}=\lVert q-\hat{q}\rVert$

Real-World Connections

• Aligns with broader smart-ag initiatives (IoT sensing, 6G networking, fuzzy logic irrigation) listed in refs [4–6].

• Complements deep-learning detection advances (YOLOv5, YOLOv8, Mask R-CNN) by adding 3D & orientation.

• Similar orientation-driven tasks exist for strawberries, tomatoes, melons – FRESHNet principles transferable.