Advances in HCI: Mixed Reality for Immersive Experiences

Presenter and Topic

• Dr Fotios Spyridonis with material from Dr Nadine Aburumman

• Topic: Advances in HCI - Mixed Reality for Immersive Experiences

• Course: CS3001- CS3606 Advanced Topics in Computer Science and Business Computing

Announcements

• A couple of announcements were made, specifics not provided.

Overview of Topic

Definition of Mixed Reality (MR)

• Mixed Reality is presented as a class of immersive systems that merge virtual and physical worlds.

• Covers core technical approaches:

  • Marker-based Augmented Reality (AR)

  • Marker-less AR and Visual Simultaneous Localization and Mapping (Visual SLAM)

• Focuses on understanding the systems rather than just using tools to interact with them.

Relevance

• This topic builds on previous knowledge in Human-Computer Interaction (HCI) and Computer Science (CS) concepts.

• It provides foundational key knowledge for:

  • Extended Reality (XR) development projects

  • Final-year projects involving AR/MR

  • Advanced topics in HCI and spatial computing

Understanding Immersive Technologies

Types of Immersive Systems

• Augmented Reality (AR)

• Virtual Environment

• Mixed Reality (MR)

• Virtual Reality (VR)

Mixed Reality (MR) Explained

The Reality–Virtuality Continuum

• Visualizes the relationship between MR, AR, and VR as a continuum, where:

  • MR combines real-world and virtual elements

  • AR overlays digital content onto the real world

  • VR represents fully immersive digital environments.

Importance of MR

• Addressing the challenge: aligning digital and physical worlds

• Key Distinction:

  • VR is considered a controlled environment with defined limitations.

  • MR involves uncontrolled physical environments, requiring adaptability to dynamic elements such as lighting and motion.

Core Technical Approaches

Marker-based Augmented Reality

• Functionality of Marker-based AR pipeline:

  1. Video input: A video stream is captured via a camera.

  2. Image Processing: The video stream is converted to a binary image to identify a black marker.

  3. Marker Detection:

  • Easy detection is facilitated through edge and corner detection, considering surface color discontinuity and illumination discontinuity.

  1. 3D Transformation: Compute position and orientation of the marker relative to the camera represented as $T = {P, R}$, where P indicates position and R signifies orientation.

  2. Rendering: Virtual objects are positioned and oriented based on the detected marker, rendered within the video stream.

  3. Augmentation: Overlaying the virtual object onto the real-world video frame based on calculated transformations.

Limitations of Marker-based AR

• Dependency on strong visual contrast of markers.

• Non-functionality with reflected light and occlusion.

• If the camera moves too far from the marker, virtual content disappears.

Image-based Augmented Reality

• Continuous tracking of features within each frame is critical for maintaining stability.

• Challenges in keeping a continuous track include tracking stability and outlier detection due to significant frame-to-frame changes.

Marker-less Augmented Reality

Categories

• Prepared environment: Known features and markers; examples include predefined reference points.

• Unprepared environment: Works in real-world settings without prior knowledge of the space (Adopts Optical Tracking and SLAM).

SLAM (Simultaneous Localization and Mapping)

• Described as a method where a system constructs a map of an unknown environment while tracking the camera's position using visual data.

• Example: Early SLAM dates back to 1986, evolved through the integration of computer vision and sensor data.

Visual SLAM Pipeline

1. Track feature points across camera frames.

2. Use these tracks to calculate triangulated 3D structure and position.

3. Simultaneously use estimated point location to determine camera pose.

Advantages of Visual SLAM

• Operates without markers.

• Applicable in unknown environments and adaptive to larger spaces.

• Essential foundation for modern AR systems (e.g., ARCore, ARKit).

Mixed Reality Hardware Components

Types of Hardware in MR

• See-through displays

• Multiple cameras: depths camera, various image sensors, IR illuminators, Inertial Measurement Unit (IMU).

• Hardware aspects include:

  • Aspect ratio: 3:2

  • Resolution: 2K

  • Display rates of 120 - 240Hz

Calibration in MR Systems

Intrinsic Properties

• Optical Center, scaling represented by intrinsic parameters.

Extrinsic Properties

• Incorporates camera rotation and translation defined through extrinsic matrices:
  $T = \begin{bmatrix} r<em>{11} &amp; r</em>{12} &amp; r<em>{13} &amp; t</em>1 \ r<em>{21} &amp; r</em>{22} &amp; r<em>{23} &amp; t</em>2 \ r<em>{31} &amp; r</em>{32} &amp; r<em>{33} &amp; t</em>3 \ 0 &amp; 0 &amp; 0 &amp; 1 \ \end{bmatrix}$

Spatial Mapping in Mixed Reality

Definition of Spatial Mapping

• The process through which MR devices map the real-world environment, creating a digital understanding of the space.

• Mesh creation: a representation achieving a fishing-net-like overlay covering real surroundings.

Applications of Spatial Mapping

• Facilitates visualization and navigation of virtual objects relative to the real world.

• Enables physics simulations where virtual objects interact with their physical equivalents.

Interaction Models in Mixed Reality

Interaction Methods

• Varies by recognized structures and user engagement:

  • Hands are recognized as skeletal models (left and right)

  • Additional features include colliders on fingertips to ensure interaction accuracy.

• Various forms of interaction include:

  • Direct interaction with collidable fingertips

  • 3D object manipulation techniques using bounding boxes.

  • Proximity shading for depth perception in engaging with 3D objects

Future Directions

• Gaze and head tracking as well as voice-based interaction methods are also identified for future developments in MR.

Lecture Summary and Key Takeaways

• The evolution of immersive systems: VR, AR, MR presented as significant technological advancements.

• Progression highlighted: from marker-based tracking to Visual SLAM and further to spatial mapping.

• Core conclusion: MR extends beyond simple rendering; it requires comprehensive interaction with and understanding of real-world environments.

Additional Information and Tasks

• Key readings available in Topic Guide on Brightspace.

• Students are encouraged to participate in discussion and quizzes to enhance understanding.

• Upcoming events include a guest speaker and a problem-solving task in Howell Theatre.

Contact Information

• CS3001-CS3606 Advanced Topics in Computer Science and Business Computing

• Questions can be directed to Dr Fotios Spyridonis via email: fotios.spyridonis@brunel.ac.uk

References

• Rokhsaritalemi, Somaiieh, Abolghasem Sadeghi-Niaraki, and Soo-Mi Choi. "A review on mixed reality: Current trends, challenges and prospects." Applied Sciences 10.2 (2020): 636.

• Speicher, Maximilian, Brian D. Hall, and Michael Nebeling. "What is mixed reality?" Proceedings of the 2019 CHI conference on human factors in computing systems. 2019.

• Kruijff, Ernst, J. Edward Swan, and Steven Feiner. "Perceptual issues in augmented reality revisited." 2010 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 2010.