Augmented and Virtual Reality Notes

History of AR/VR

  • 1838 - Stereoscope: Charles Wheatstone discovered that the brain processes different two-dimensional images from each eye.
  • 1849 - Lenticular Stereoscope: David Brewster created a smaller, hand-held stereoscope.
  • 1939 - Viewmaster: William Gruber made this stereoscope focused on virtual tourism.
  • 1950 - Sensorama: Morton Heilig created an arcade-style cabinet simulating all senses.
  • 1960 - Telesphere mask: Morton Heilig's telesphere mask was the prototype of HMD.
  • 1961 - Headsight: Comeau and Bryan developed the headsight, projecting a screen for each eye.
  • 1968 - Sword of Damocles: Ivan Sutherland, with Bob Sproull, created the first VR/AR head-mounted display, which was primitive and heavy.
  • 1974 - Videoplace: Myron Krueger established Videoplace, where users could interact with artificial reality without special tools.
  • 1987 - Virtual Reality: Jaron Lanier coined the term "Virtual Reality".
  • 1990 - Augmented Reality: Tom Caudell coined the term "Augmented Reality".
  • 1991 - Virtuality Group Arcade Machines: Virtual Reality became available to the public through Virtuality group Arcade Machines.
  • 1992 - Virtual Fixer: It uses the AR technology to aid human performance in both direct and remotely controlled tasks.
  • 1993 - KARMA: The first system incorporated knowledge-based AR system.
  • 1996 - Collaborative AR System: The first collaborative AR system, called the Studierstube.
  • 1997 - Outdoor AR System: Developed by Feiner et al., called Touring Machine, included a see-through HMD with GPS.
  • 1999 - BARS Research Begins: BARS, an abbreviation for Battlefield Augmented Reality System, began its research.
  • 2000 - AR ToolKit: A software that uses video tracking to overlay computer graphics on a video camera.
  • 2003 - Handheld AR System: First handheld AR system that runs autonomously on smartphones.
  • 2014 - Google Glasses: Google released Google Glasses, a wearable AR device.
  • 2016 - Pokemon GO: Mobile game, a wildly successful application using AR to create digital models of Pokemon on top of the view of the real world.

What is AR/VR/MR?

  • Augmented Reality (AR):
    • Technology that layers computer-generated enhancements over an existing reality to make it more meaningful through interaction.
    • Expands our physical world by adding layers of digital information onto it.
    • AR overlay is executed immediately with the input received from a camera or another input device like smart glasses, bringing virtual objects into the surrounding environment.
    • Combines real and computer-based spectacle and pictures to transfer a unified but enhanced view of the world.
    • Encompasses computerized techniques and simulations like animation, text to speech recognition and handheld devices or powered display environments to overlay virtual objects over real-world environments.
    • Examples: Snapchat lenses, Pokemon Go.
  • Virtual Reality (VR):
    • A realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by the movement of the body.
    • An immersive, interactive experience generated by a computer.
    • VR uses computer technology to create a simulated environment, placing the user inside an experience instead of viewing a screen, immersing users in 3D worlds.
  • Mixed Reality (MR):
    • In MR environments, users navigate through both real and virtual spaces simultaneously, making virtual interactions appear real.
    • Virtual objects are integrated into the user’s real world and augment their real environment.
    • When both real and artificial worlds merge, a new environment and visualization become possible where physical and digital objects co-exist and interact in real-time.
    • Microsoft's Hololens is a self-contained holographic computer that enables users to interact with digital content around them, visualizing and working with digital content as part of their real world.
    • MR brings the best of both worlds by combining Virtual and Augmented Reality.

Comparison 1: AR vs. VR

  • AR:
    • Adds to your field of vision and only show up on your smartphone or tablet screen.
    • Delivers virtual elements as an overlay to the real world.
    • Uses your phone's camera to watch a view of the real world in front of you, then put a layer of information, including text and/or images, on top of that view.
    • Enhances both the virtual and real world.
    • Users can control their presence in the real world.
    • Accessed with a smartphone or tablet.
  • VR:
    • Offers a digital recreation of a real-life setting.
    • Completely covers and replaces your field of vision (completely virtual).
    • With VR you can move around and look in every direction -- up, down, sideways and behind you, as if you were physically there.
    • Only enhances a fictional reality.
    • Users are controlled by the system.
    • Requires a VR headset device.

Comparison 2: AR vs. MR

  • AR:
    • The visible real environment is overlaid with a layer of digital content; just displaying simple images.
  • MR:
    • Strives to put fully digital objects that are traceable and intractable in the user’s environment.
    • Virtual objects are integrated into and responsive to the real world and can be manipulated by the user.
    • An enhanced form of AR that takes the best qualities of AR and VR.
    • Rendering demands more processing power than AR.
    • Virtual objects are integrated into and responsive to the real world and can be manipulated by the user.

Statistics About AR/VR

  • The video games sector is the highest, followed by the Education & Healthcare sector.

Applications of AR/VR

  • Medicine: Used in medical studies to enable students to know the human body structure and in scientific research laboratories.
  • Training Applications: Used in military training for battlefield familiarization and in driving schools for real-look road and traffic simulations.
  • Education: Used as Apps for kids. Children can use an iOS or Android device to scan the globe and view animations and educational content such as animals, monuments, and food.
  • Development Work (e.g., City Planning): Urban planners could use AR to place bridges, view traffic intersections up for redevelopment or get visuals on how wind may impact building development and pedestrian movement on a street.

Case Study: Human Body

  • James Tromans, a primary school teacher in the UK, used VR technology for a science lesson about the human body.
  • Children used VR headsets, and the teacher used the human Anatomy Playlist to explain different sections of the body.
  • The experience allowed children to explore the human body, positively impacting their understanding.

How Does VR Technology Work?

  • VR systems work by tricking the brain into perceiving the virtual environment as reality.
  • The process combines hardware and software to create immersive experiences that “fool” the eye and brain by simulating as many senses as possible.
  • Hardware supports sensory stimulation, while software creates the rendered virtual environment.
  • Hardware includes:
    • A personal computer: Processes inputs and outputs, renders high-quality graphics using GPUs (Graphics Processing Units).
    • Input devices: Keyboards, mouse, VR controllers, balls or tracking balls, controller wands, data gloves, trackpads, on-device control buttons, motion trackers, bodysuits, which help users navigate the 3D worlds.
    • Output devices: Haptic displays and head-mounted displays (HMD) which simulate a sense organ and present the VR content or environment to the users to generate a feeling, Headphones with spatial audio which provide an audio landscape that matches the visuals from HMD.
      • Modern VR headset displays are based on technology developed for smartphones, including:
        • Motion sensors for tracking head, hand, and body positions.
        • Small HD screens for stereoscopic displays.
        • Small, lightweight and fast processors.
  • VR Software:
    • Creates the content, develops 3D objects, and manages VR input/output devices.
    • Inputs to the VR software must be on time, and the output response should be prompt (real-time).
    • Old Methods: The Virtual Reality Modelling Language (VRML), without dependency on headsets. The Web3D consortium standards for web-based 3D graphics. The consortium subsequently developed X3D from the VRML framework as an archival, open-source standard for web-based distribution of VR content. Web VR is a JavaScript application programming interface (API) that makes it possible to experience VR in your browser.
    • New Methods:
      • VR Software Development Kits (SDKs).
      • VR Game Engines.
      • VR Content Creation.
      • VR Visualization Software.
      • VR Training Simulators.

How Does AR/MR Technology Work?

  • An AR system captures a part of the environment using a camera on a smartphone, tablet, or head-mounted display (HMD).
  • It scans the captured piece of the environment to identify a location where to overlay additional information using markers or trackers like GPS or sensors.
  • Once the additional information is generated, the AR system forms a complete image consisting of the real-world background and overlaid AR graphical data.
  • Hardware and software are needed to accomplish these steps.

AR Hardware

  • Sensors: Collect data from the physical environment and transmit it to an AR app (software).
  • Modern mobile computing devices like smartphones and tablets contain a camera, GPS and MEMS (Micro Electro Mechanical Systems) sensors such as accelerometer, gyroscope and solid state compass, making them suitable AR platforms.
  • AR Smart Glasses: Wearable computer-capable glasses that add extra information, ideally 3D images, animations, and videos, to the user’s real-world scenes by overlaying the computer-generated or digital information on the user’s real-world.
    • Can retrieve information from computers, smartphones, or other devices and can support Wi-Fi, Bluetooth, and GPS.
    • Example: Google Glass Explorer Edition and Microsoft HoloLens.

AR Software

  • Developers of AR Apps usually use Apple ARKit and Google ARCore.
  • These SDKs allow developers to quickly and easily create AR apps for iOS- or Android-based smartphone and tablet devices accordingly.

Advantages of AR/VR

  • In Training: VR provide ways of modeling complex task- performance behaviors, many of which carry life-or- death risks in real-world learning.
  • In Education: VR makes learning easier, interesting and more comfortable.
  • In Entertainment and Gaming: Goggles and headsets can insert participants into imagined worlds, turning a watching screen into living an experience.
  • In Help & Healing:
    • VR can assist in treating phobias, especially those that involve handling or being near specific animals, environments or objects.
    • VR holds promise in physical rehabilitation, providing patients with opportunities to refine ambulatory or other skills in a clinic setting before moving on to the real-life equivalent.
    • Simulating traumatic events can help military service members work through some of the effects of post traumatic stress disorder that result from combat.
  • In Architecture & Planning:
    • Applying VR technology to architectural design and urban planning helps decision-makers visualize the outcomes of proposed development and renewal.
    • Understand space management in a better way.
    • Using VR combined computer-aided design with geographic information systems to produce a virtual world in a Web browser, saving communication time between designer and client, and providing a personalized experience.
    • Best for training purposes for new Architects.

Disadvantages of AR/VR

  • Loss of Spatial Awareness:
    • "Cybersickness" , digital motion sickness, is the opposite of traditional motion sickness.
    • You see movement that you do not feel, causing sensory conflict and leading to nausea, sweating, dizziness, and headache.
    • People prone to motion sickness and unsteadiness are not going to be on a good foundation going into the world of VR
  • Eye Soreness: Short-term eye strain is normal in VR.
  • Seizures: People susceptible to seizures should avoid VR due to potential responses.
  • Nausea, Dizziness and Disorientation.
  • It is not good either for the elderly people, pregnant women’s and young children. There is No child safety settings.

Challenges of AR/VR

  • Price is Too High: The majority of the engaged audience can’t afford buying VR gear, and industry players can’t afford lowering the price.
  • Content is Lacking: The industry is still waiting for its killer app.
  • Lack of Viable Business Models: There are no viable cost-effective business models for VR-oriented companies.
  • Health Effects: Uncertainty about possible effects on users' health and barely explored long term effect.
  • VR Perceived as Gimmick: VR lacks acceptance from anyone beyond tech fans and early adopters and the majority of the public perceives it as an expensive gimmick designed for video games.
  • Ethical Issues: VR developers should spend more time navigating ethical dilemmas, including pornography, user isolation, social effects, and user's physical and digital protection.
  • Social Impact: There is a high level of concern over the negative influences of interactive VR environments towards social implications.
  • Some issues include people turning their backs on the real world and wandering around the synthetic worlds that fulfill their whims and the users who are engage in violence VR video games may become desensitized to their violent virtual actions and mimic that behavior in real world.