Hardware & Software (Lecture 2)

VR

• creates end-to-end mechanisms that replaced our natural real world environments w/immersive simulation

• uses combination hardware & software

to create an immersive & realistic VR system needs a combination of

• infrared LEDs

• motion sensory

• cameras

• screens

  • allows a headset to gather relevant info & present it to human eye

HMD VR

Immersion

• is the ability of the virtual reality system of actually tricking you in feeling that you’re somewhere else

• perfect immersion in VR = same sensorial info of the real world

Presence

• is how you’re really engaged and feel yourself inside the virtual world

• if story is compelling, they will be completely absorbed by it

  • indicates how much the user feels engaged with the virtual reality experience.

what technical elements make an immersive experience in VR

• tracking

• rendering

• display

tracking

• is the process of measuring body movements of the user.

• Experts state that of the 3 parameters that comprise a VR experience, tracking is by far the most important.

• It is pertinent to note that the tracking of eye movements is more

Rendering

• the process of taking a symbolic 3D model that consists of mathematical information & instantiating appropriate sights, sounds, touch and sometimes even smell for the new location

• user may change their position rapidly → inecessary to render this info real fast & almost in real time.

Display

• the manner in which the physical senses are replaced w/digital info

• once the sights/sounds for new location are processed & rendered they need to be delivered to the user

Process of VR immersion

1. Display & optics generate 2 separate views for each eye

2. graphics processor generated the graphics

3. input controllers captures input commands by the user

4. sensorys infer the pistion & movement of input

Stereoscopic lenses

• positioned between screen & eye to distort images & make them 3d

• headset passes 2 images through the lenses

  • one for each eye

Infrared cameras

• within headset

• adjust the light to the user’s need

• trackers allow the device to shift the content on a screen as we move our heads

• some can track eye movement 

VR headsets

• high-resolution displays which present the content of the virtual world

• use a special type of lens called Fresnel lens

• lens focuses on the content for the user

• Most smartphones can work as VR headsets by adding optical elements e.g google cardboard

Latency

• time delay between a user’smovement & corresponding visuals

  • align virtual information with the real world

minimal latency

• is necessary to ensure that what we see changes simultaneously when we move around, turn our heads, or look in a specific direction

• generally considered to be below 20 milliseconds

high latency

• can break immersion & induce motion sickness

high-resolution VR headsets

• perceived to have better view quality & higher immersion.

• Higher display resolution creates more processing load on the device

Field of View as an important characteristic in VR headsets

• Modern-day VR headsets support at least a 110- degree horizontal view

Frame rate

• determines how immersive the experience is and how likely we are to encounter VR sickness.

• human eyes can see up to 1,000 frames per second, the brain only interprets frame rates up to 150 FPS.

• Generally, the frame rate of a movie in a theater is only around 24 FPS, but these films aren’t intended to simulate reality.

Frame rate & VR sickness

• VR developers find anything less than a rate of 60 FPS causes nausea & headaches.

Refresh rate

• is the number of frames that the display can output per second

• High refresh rates (75Hz or higher) alleviate motion sickness & reduce flicker-induced fatigue

  • minimises blur & maintains immersion

• higher refresh rates require powerful processors to deliver content in a timely manner.

  • Thus cost of the VR headset increases w/increasing resolution, refresh rate, quality of optical elements and the FoV.

Spatial audio

• has become increasingly important in any VR experience in recent years.

• Sound & where it comes from influence our perception of a 3D space

  • if we’re walking through a meeting room in VR & someone calls our name from “behind us” in the setting, we hear that sound coming from behind us.

Graphics processing

• is a resource-heavy task and the two main classes of VR devices, tethered and standalone, approach this differently

Tethered devices & graphic processing

• offload graphic processing to a dedicated computer w/powerful GPU through USB 3.0 & HDMI

  • e.g Oculus Rift S & HTC Vive

• BUT tether cables can be unwieldy/limit the user’s movements.

Standalone devices & graphic processing

• use onboard mobile GPU for graphics processing

  • Oculus Go & Google Cardboard

• mobile GPU is not as powerful as a PC-based GPU.

• unit has to rely on inbuilt battery power which is also limited

Head, movement & position tracking

• use a combination of sensors, gyroscopes, & AI to influence what we see as we move around.

• Basic headsets initially used 3DoF systems

  • only allowed us to look left, right, and up and down

• most advanced headsets (Varjo XR-4) use 6DoF to ensure we can look around in a 360-degree format,

  • like we would in real life

tracking sensors

• go beyond capabilities offered by input controllers

• tracking-based input communicates the orientation & position of a tracked entity to the VR headset

• can be hand-held controllers, user’s fingers or hands, user’s body or even the VR headset itself within a room

• Tracking enables user to perform more natural-feeling actions to interact with virtual world

types of tracking

• tracking sensors can be

  • room-scale & external (e.g. Oculus Rift)

  • internal on the device (inside-out) (e.g. Oculus Quest).

• Inside-out tracking is very useful for the portability of untethered devices

Virtual reality controllers

• many headsets still feature connected controllers

• hardware components that allow users to take action & do things in a virtual environment

• allow us to submit information about what we want to do to VR software directly

• support a combination of buttons, trackpads & joystick controls bundled into a hand-held unit that communicates wirelessly w/headset

interacting within the VR world

• To interact w/world, the device has to capture inputs provided by the user

• certain devices support the user’s gaze, head-pose and simple hand movements as inputs.

• Eye-facing infrared cameras, gyroscopes and accelerometers sense the inputs.

• The input controller units are self-contained and support 3DoF or even 6DoF

challenges facing VR

• Varjo, HTC VIVE, & Microsoft trying to address “VR sickness” & the discomfort felt when exposed to VR for long periods

• Intro of “metaverse” environments & growing adoption of VR unveils potential issues → ethics, cyber security

• more affordable BUT creating VR tools & software is still expensive/time consuming

• Creating intuitive VR environments requires significant investment & may impact the environment due to high-level computing requirements

• All these issues need to be addressed to create a world where everyone can discover the benefits of virtual reality

best VR accessories for deeper immersion

• Virtual Reality Cameras

• Innovative VR Controllers

• Accessories for Spatial Sound

• VR Trackers, Sensors, and Base Stations

• Haptic Gloves & Suits

• Accessories for New Sense Stimulation

• Chairs & Treadmills

• Accessories for User Comfort

• Custom Prescription Lenses

• Virtual Reality Accessories for Battery Power

Accessories for Spatial Sound

• The Quest 3, for instance, comes with spatial sound capabilities and speakers up to 40% louder than those on previous devices. Meta has partnered with Razer to deliver high-performance audio experiences to users who want more privacy in VR landscapes. Elsewhere, companies like Rokid are developing comprehensive spatial computing platforms, which can be enhanced with powerful cinematic speakers.

Haptic Gloves & Suits

• Companies like SenseGlove are now introducing more advanced haptic gloves that can offer haptic feedback based on impact and vibrations.

• Some companies, like Meta, are even investing in combining haptic solutions with electromyography hardware capable of detecting minute muscle movements.

• Innovators like Tesla are also exploring opportunities to create full haptic suits that offer sensory feedback in various ways.

• These suits allow users to experience touch sensations, simulations of weight, and even different temperatures when navigating virtual reality spaces.

Chairs & treadmills

• Companies like Roto VR, for instance, are producing chairs attached to a motor system that allows users to move around more freely when seated while also connecting them with in-app experiences.

Accessories for New Sense Stimulation

• For instance, the sense of smell has yet to be fully implemented into the virtual reality space. In 2019, FeelReal made waves with its Kickstarter program for a “Multisensory mask.”

Innovative VR Controllers

• “Tap” creating knuckle straps that allow users to interact w/virtual keyboards w/natural finger movements

Mobile VR headsets

• Mobile headsets are shells w/lenses 

• Lenses separate the screen into 2 images for your eyes

  • turns a smartphone into a VR device

• Relatively inexpensive

• Not tethered

• Phones aren’t designed specifically for VR,

  • can’t offer best visual experiences

  • underpowered compared w/PC or game console-based VR

• no positional tracking with mobile VR. Y

  • you can’t look around objects.

• e.g Merge Vr

Standalone VR headsets

• An all-in-one VR headset, or standalone headset, puts everything in the headset needed to convince you that you’re in another world.

• It is a single integrated piece of hardware → wireless

• e.g Meta Quest series

PC / Console connected VR headsets

• provide a more immersive experience at a higher price point

• Tethered

  • cables from the headset to an external piece of hardware to power the headset

• E.g Vive or PlayStation VR

3 main types of movement and positioning tailored for each play area size

• Roomscale VR

  • set a boundary or play area to move freely around in the game

  • With these games, can physically move around your space to interact w/environment

• Seated and/or Standing

  • user stays roughly in the same place & uses the controller to move instead of physically moving through a space.

Motion Sickness/VR sickness

• the sense of nausea & unease many experience after being exposed to an immersive experience for an extended period

• some people are more susceptible to this issue than others

  • women & people 50+ are generally more likely to feel unwell after using a VR headset

how can VR sickness be reduced

• Improved spatial tracking

  • Sensors capable of tracking movement can significantly reduce symptoms

• Enhanced user interfaces

  • Handheld controllers cause a disconnect between what we’re physically doing and environment we’re seeing

  • creates sensory conflict leading to disorientation & discomfort

• Reduced latency Latency

  • more time it taken to register in-app movements more confusing for the brain

  • investing new technologies to help minimize latency

Psychological issues of VR

• “uncanny valley” effect when interacting due to realistic avatars

  • anxiety and stress among user 

• VR used in training sessions might lead to an increased level of stress when exposed to worrying situations & simulations

  • Surgery simulations 

• More time users spend in a virtual world, the more likely they feel disconnected from the real world

  • suggested that p w/mental health issues should avoid VR headsets

Software

• to create content for VR, creator need to develop/deploy software ecosystems customised for device

• 2 leading software ecosystems,

  • SteamVR and Oculus (PC or Mobile)

  • Google Daydream supports VR on smartphones

• complex & involves a steep learning curve

• browser-based webXR has emergence as an alterantive VR tool

types of software

• Unreal and Unity engines provide SDKs (Software Development Kits) and integration for most platforms (e.g PS VR)

Unity vs. Unreal Engines

• Unreal Engine & Unity are 2 of the most popular game engines available to create VR experiences

• Unreal Engine has a higher quality output, at the expense of ease of use e.g Batman Arkham VR

gLTF/GLB

• a neutral, open source format.

• The Khronos Group created this format for 3D web, AR, VR, Games and 3D advertising

• based on JSON → stores some data in external files like textures (JPEG or PNG), shaders (GLSL), or geometry & animation data (BIN)

FBX

• is a proprietary 3D file format

• Originally developed by Kardara

• used in the film & video game industry

• It supports geometry, appearance & animations (skeletal & morphs)

• most popular for animation

• used as an exchange format between different programs like Maya, 3DSMax, AutoCAD, Roman’s CAD, and others.

OBJ

• It is a neutral 3D format when used as an ASCII variant.

  • when used as a binary variant, it is proprietary.

• 3D printing, graphics, & 3D scanning all use this file format

  • ability to store geometry, colour & texture information.

• does not support animations, but is one of the most popular interchange formats for 3D graphics.

Metaverse

• refers to collective virtual shared space that is created by the convergence of virtually enhanced physical reality & physically persistent virtual spaces

• envisioned as an evolution of the internet where users can work, play, socialise in a fully immersive/interconnected enviroment

• Mystakidis (2022) says it is heavily reliant on XR technologies to create seamless experiences

• aims to integrate words more completely 

issues regarding metaverse

• issues of privacy

• data security

• digital divide

• cost

realisation of the metaverse

• requires a comprehensive intergration of

  • hardware

  • software

  • content

• creates immersive experience

Hardware

• central to creating immersive/interactive experiences

• provide visual & auditory experiences

• HMDs

  • crucial for visual immersion by tracking user’s head movements

• input tool to enhance user’s interaction w/VW

  • handset tools for tactical interaction

  • eyetracking

  • voice recognition

  • body trackers/treadmills

implementation of the metaverse involves several stages

1. design

2. model training

3. operation,

4. evaluation

in the design phase of the metaverse what is considered

• goals

• constraints

• user scenarios

model training phase of the metaverse involves

• data analysis

• user modelling

• iterative learning to finetune the system

operation phase of the metaverse involves

• considerations for system operation, simulations, and network environments

evaluation phase of the metaverse involves

• assesses content fidelity, interaction authenticity, and implementation feasibility

Current landscape of VR

• characterised by rapid advancements in hardware & software

• investing billions into developing infrastructure 

why are there growing concerns about privacy, security & ethical use

• XR devices collect vast amounts of sensitive data

  • including biometric information, pose significant risks if not properly managed

• Metaverse promises an interconnected virtual world

  • involving even greater amounts of data & more complex interactions between users and digital environments.

applications of the metaverse

• simulation is most prominent

  • gaming

  • education

  • marketing

  • social research

• provide immersive & interactive experiences that mimic real-world scenarios, offering valuable insights and training opportunities

challenges of metaverse

• cyber motion sickness

  • caused by the disconnect between physical movement and visual feedback.

• demand for real-time data processing & high-speed rendering in a 360-degree field of view poses technical challenges that require continuous innovation

• Metaverse is crucial for maintaining user engagement.

• interdisciplinary collaboration, technological advancements, and user experience innovation will be essential to overcoming these challenges

density of pixels within HMD

• high pixel density is essential to avoid visible graininess & ensure sharp imagery

Experimental headsets → retinal projection technology

• projects images directly onto the retina using micro-projectors & mirrors

• Advocates suggest that this approach may reduce eye strain,

• Unable to match the full immersive capacity of LCD & OLED systems

Field of view

• average human sees the world around them in a 200 to 220-degree arc around the head

• Our vision from our left and right eyes overlap at an angle, which allows us to see in 3d

optics & lenses

• lenses are used to magnify/shape the image so that it fills a users field of view

• horizontal field of view of around 90 to 100 degrees is necessary to convincingly simulate a natural environment

• quality of lenses determines the clarity of the image

  • design is as important as the display itself

Head tracking

• allows the headset to translate movements of the user’s head into changes in perspective within the virtual environment

  • achieved through combinations of accelerometers, gyroscopes, and external tracking cameras

• Advanced HMD do directional & postopal tracking

  • enabiling users to lean/move slightly for enhanced realism

eye tracking

• monitors the direction of gaze allowing adjustment of depth of field

• enable gaze-based interaction & enhance realism by allowing virtual characters to respond naturally to user attention.

motion tracking

• fundamental to the VR experience

• level of motion tracking depends

  • cockpit requite head & hand

  • VR world require full body

motion tracking must account for 6DoF

• translational movement

  • x, y, and z axes,

• rotational movement

  • pitch, yaw, and roll)

types of motion tracking

• optical

• non-optical

optical motion tracking

• relies on cameras/imaging devices like light emitting

• may wear full body suits

• infrared depth sensors

• superior in capturing precise full-body motion

non-optical motion tracking

• microelectromechanical sensorys

  • accelerometer, gyroscopes & magnetometers

• developed fro automotive & mobile now compact

• allows controllers/HMD to capture motion data at low latency 

• indispensable for portable headsets

• often combined with optical methods to increase accuracy and stability

• VR gloves

• treadmills

future for motion tracking

• may be transformed by brain-computer interfaces

  • control robotic limbs through direct neural signals

• muscle reinnervation have demonstrated that rerouted nerve signals can control artificial limbs in a manner closely resembling natural movement

software in VR

• requires specialised tools & processes that bring together 3D modelling, real time rendering & advanced user interaction mechanics

3D Modeling and Asset Creation

• requires specialised 3D modelling software e.g blender

• animate objects in virtual space

• textures applied to 3d models need to be detailed and optimised to prevent lag in rendering

Real-Time Rendering 

• Determines how the virtual environment is visually presented to the user

• must render scenes at high frame rates, typically 90 frames per second (fps) or higher, to maintain a smooth and immersive experience

• requires powerful hardware & optimised software that can handle the intense computational load

scenes

• collects all renderable + audible (geometry, lights, audio, cameras) and their spatial hierarchy

• At runtime, it’s rendered from the user’s viewpoint to one (mono) or two+ (stereo/multi-projector) images