http://vr.isdale.com

jerry@isdale.com

1. What Is VR
1. Many Names of VR

Virtual Reality, Virtual Environment, Artificial Reality, Computer Generated Environment, Computer Simulated Environment, Synthetic Environment, Spatial Immersion, Cyberspace, Virtual Worlds, Virtual Presence, Metaverse

2. Definition of VR

(courtesy M. Capp, Naval Postgraduate School)

A Computer-Generated, 3D Spatial Environment in Which Users Can Participate in Real-time.

Virtual Environments Can Be:

• Fully Immersive, Encompassing Worlds

• Augmentations (Overlay) to the Real World

• "Through the Window" Worlds (Non-immersive)

3. Taxonomy Spectrum

VR is an area in the multidimensional taxonomy spectrum of computer systems.

1. Some Related Technologies

1. VR vs. 3D Computer Graphics

1. Immersion

1. Presence

1. Range of VR Systems

1. Early VR History

1. Why Use VR

1. VE Applications

1. Why Not Use VR

1. How- VE Technology

These topics are expanded in later sections

1. Programmatic

Project Management

Technical: Systems Engineering

Artistic: World Design

Business: Marketing, Funding

2. Displays

1. Input Devices

1. Processor Hardware

Graphics hardware/software

Main processor

Multiple processors

2. Interaction Techniques
3. World Database
4. Application Software
5. Network Connectivity (Optional)

1. Universal Interaction Tasks
1. Passive Experience

Not really "interaction", simply experiencing displays

2. Navigation

Locomotion: motor component

Way-finding: cognitive component

3. Selection

Choosing one or more objects from a set

4. Manipulation

Specification of object position & orientation

Specification of scale, shape, other attributes

5. System control

All other interactions, usually accomplished via commands

May be composed of other tasks

2. Display

Display: device which presents perceptual information

Often ‘display’ used to mean ‘visual display’

Goal: display devices which accurately represent perceptions in simulated world

1. Display What

Environment Display

Control Display (cockpit, buttons, menus, etc)

Systems Monitor (behind the scenes details)

2. Visual

Stimulus: light of wavelengths ~350-750 nm

Visual dominance: 50% of brain involved in processing!

1. Human Visual System

Physiology of Eyes

Fovial/Peripheral Vission

Rods: periphery, motion, B&W, sensitivity

Cones: fovea, static, color, acuity

Perception & Brain

1. 3D depth cues

1. Issues for Visual Displays

1. Desktop Displays

Standard Monitor

Multiple-Monitors

1. Stereo Techniques

LC Shutters (glasses or screen)

3-D Monitors

Head Tracking for motion cues and eye position

2. Head Mounted Displays (HMD)

Symbol of VR to most people

Display and Optics mounted on Head

May or may not fully occlude real world

1. HMD Description

Support System

Display (pixels vs triads)

Optics

Binocular/Monocular/Biocular

Tracker strongly recommended

2. HMD Issues

Cumbersome to wear

Single user

Small Field Of View

Adjustable Field of View

Exit Pupil Size

Inter-pupil distance

Resolution

Costs

3. Projection Displays

One or more projectors and screens

Front or Rear projected

Larger size than monitors

Don’t occlude reality

Viewed by many people at a time

1. Example Projection Displays

Table ( 1 or more surfaces)

Wall (curved, flat)

Room (3 to 6 sides)

Dome (desk to Planetarium))

2. Projection Display Issues

Projector, screen and space are main $$

Seam blending, and Sync issues

Brightness (front/rear projection)

Visibility Angles

4. Other Visual Displays

Push BOOM - Fakespace Labs

WindowVR - Virtual Research

Tablets - Wacom

Cybersphere - VR Systems

5. Which Visual Display to Use?

1. Audio

2^nd most studied sense

Adds tremendously to experience, if well executed

stimulus: disturbance of molecules in a medium (air)

perceptions: pitch, loudness, location

1. Sonification of Interface

Alerts and Interaction Feedback

Ambient Sound

Modern systems are too quiet, lack operating noise

David Theil MS Research

2. Sound Effects

Classic Multimedia, game tool

intensity fall-off (1/d²)

headphones also block out real-world noises

ambient sound (e.g. stream)

3. Environmental Audio

Sound occlusions

Reverberation from surfaces

Room acoustics

Environmental Audio

Direct X API

4. Spatialized Audio

2 ears allow localization

Works well in plane of ears

Interaural intensity differences

Interaural time differences

Pineal shape effects

Head Related Transfer Function (HRTF)

5. Speech Generation

Avoids text display which breaks illusion

Extra processing required

Vocal Tract simulation

Prerecorded words, phonemes and diphones

Intelligibility may require practice

Lacks tonality, rhythm, emphasis, general expressiveness

6. Audio Display Devices

PC sound boards (mid to high range)

External synthesizer, mixer

Stereo Headset (with or w/o HRTF)

Stereo speakers (HRTF with cancelation)

Multiple Speakers

Bass Shaker Speaker

Amplifiers, wiring

2. Haptics

Adds greatly to VE when you interact with objects

Tactile, Kinesthetic

1. Tactile

Temperature, skin curvature and stretch, velocity, vibration, slip, pressure and local force

Vibration fairly easy and cheap (pager parts)

Texture is harder to reproduce (pin arrays)

Gross vs local temperature

2. Force

Force feedback, Force reflection

Exoskeleton (CyberGrasp)

Armature (Sensable Phantom)

PC Joysticks & Steering wheels

Mice (vibration or forces)

3. Motion Platform

Vestibular and Proprioception

Impulse Force and Onset Cues

Washout

Degrees and Range of Motion (1, 3, 6)

Stewart Platform (aka Hexapod)

Walking Simulators

Virtual Motion Headset (eletrostimulation)

4. Issues for Haptics

Detailed Geometric Modeling

Complex Force Calculations

Input Coupling

Often requires separate processor

3. Other Displays

1. Inputs

Hardware that allows the user to communicate with the system

Discreet, event based device

Continuos, sampled devices

1. Trackers

1. Types of Trackers

1. Tracker Issues

1. Gloves

The other "classic" VR device

Finger position sensing gloves

Discrete gloves (tips touching: Fakespace Pinch)

Armature tracked

Bend sensors (optical, resistive, other)

Tracker required for overall position/orientation

2. Wands

Free space joystick

Buttons plus tracker

Variety of shapes and configurations

3. Speech Recognition

Frees hands for other devices

Discreet Command vs Continuos Speech

Dialog Management

Ambient Noise

False positives

Training

4. Gesture Recognition

Single posture

Multiple posture

Posture plus motion

Face, hand, arm gestures

5. Locomotion Devices

Tread mill

Stair Stepper

Bicycle, Unicycle

Tilt platform (skate/snow board)

6. Other Devices

1. Basic VR System Loop

Input Processing

Simulation Update

Rendering (to displays and network)

Processing Latency between input sensing and display

Multi-threaded processing

2. World Model - VE Database
1. Extended Scene Graph

Scene Graph: hierarchy of objects and spaces (geometry and lighting) with position & orientation linkages

Extended to include input devices (trackers, etc), sounds, behaviors, links to other worlds, etc.

2. 3D Modeling
1. Geometry Types

Polygon Models

3d Curve Surface

Solid Modeling (voxels, geometry)

Procedural (Fractals, etc.)

Articulated Objects

Landscape (height field)

2. 3D Object Scanning

Quickly model objects

Volume (small, medium, large)

Laser Digitizers

White Light Digitizers

Tracker based wand

Photogrammetry

3. Virtual Human Avatar

Represents user and others in virtual world

May or may not be humanoid

Some standardization efforts (MPEG-4, H-Anim)

National Library of Medicine Virtual Human Project

4. Texturing

Substitute image for geometric detail

Highly effective

Many techniques for texturing (MIP, Reflectance, etc)

5. Levels of Detail

Different model descriptions for same object

Use lower complexity models when far away or cluttered scene

Switching objects can cause artifacts

Particular to Simulator and VE systems

3. Lighting

Diffuse, specular, ambient light

Light Sources (spot, flood, ambient, colored, object specific)

Light Maps (radiosity and global illumination)

4. World Assembly

World vs Object Space

Multiple Worlds

Environmental Effects (fog, etc.)

3. World behaviors
1. Object Motion

Animation

Constraints

Motion Capture

Simulation

2. Collision Detection and Response

Very important for realism

Many fast detection algorithms emerging

Reactions may require force simulation

3. Physical Simulation

F=MA

Modeling Newtonian Forces

Kinematics and other forces

Fake physics for expediency

4. Scripting

Data flow programming

Java and other scripting in VRML

5. Area of Interest Culling

Reduce scene in memory by area of interest

6. World Linking

Jumping to different world databases

Web based links in VRML

7. Artificial Life

Simulated life with goals, plans, etc.

"Natural" selection and evolution

Neural Net driven life forms

4. Display Rendering
1. Real time 3D Graphics
1. Graphics Pipeline

Area Culling

View Frustrum

Clipping

Hidden Surface

Shading/Textures

Rasterizing

AntiAliasing

Optimization

2. Software Systems

OpenGL

DirectX

VRML

3. PC vs Workstation

1. Non Visual Rendering

1. Types of Net VE

Streamed Browser Worlds (single user)

Community Chat Worlds

Collaborative Design Worlds

Distributed Training Environments

Internet Gaming

2. Challenges

(courtesy D. Gracanin)

Network Bandwidth

Heterogeneity

Distributed Interaction

Real-Time System Design and Resource Management

Failure Management

Scalability (geography of scene, net distance, population,etc)

Deployment and Configuration

3. Centralized Model

One computer (database) collects all data and sends updates to the users.

Simple structure.

Not scalable, the database is the bottleneck.

4. Distributed Model

1. Systems Engineering

Creating a VE system requires integration of hardware, software and artistic creations

Many interrelated issues between these domains

Systems Engineering approach needed to manage these issues.

2. World Authoring

Much of the authoring tools and techniques are shared with other 3D graphics domains

VR demands more speed from models, so there are trade offs and special techinques for the world database

3. Design Concepts
1. Usability & Human Factors

Usability as measure of quality

To show human performance can be improved through use of VE

VE Usability Research (U.Va, UK, etc.)

3D UI Web (http://www.mic.atr.co.jp/~poup/3dui.html)

Excellent detailed tutorials

1. Goals of Interaction Design

1. Philosophies Of 3D Interaction Design

1. VE Design Theory

Virtual Environment Design is a relatively new field, open to experimentation.

Draws from many areas of design including Architectural Theory

Benedikt's 1992 Cyberspace: First Steps

Chapter in forthcoming "Handbook of Virtual Environments"

1. Styles of World Building

Mike Heim, Art Center College of Design, Pasadena CA

Additive: draw from pre-built libraries

Authoring: create new objects and spaces

Realistic Construction: reflects real world environments

Fantastical Construction: capitalize on unique nature of VR

2. Church/Murray Aesthetics

1. Perceptual Opportunities

1. VR Software Features

Importing models

Libraries

Level of detail (LOD)

Object transformation

Constraints

Articulated features

Animation

Collision detection

"Parallel worlds"

Light sources

Event handling

Audio

Control language

Sensors/Tracking Device Support

Stereo viewing

2. Complete vs Toolkits
1. Few Off the Shelf Systems,

1. Lots of Toolkits

Flexibility of Device Interfaces

Complexity of Development

2. Toolkits for Various Levels

1. System Styles

1. ExampleSystems

1. Entertainment

Location Based Systems

Arcade Games

PC and Gamebox

2. Education/Training

Vehicle training

Maintenance training

Medical training

Patient Education

Situational Training

Mission Rehearsal

Virtual Heritage - History

Hazardous Operations

Science & Math Education

3. Research

Geoscience

Psychology

Perception

VR research

4. Design/Development

Prototyping

Product Design reviews

Architectural review/presentation

5. Medical

Training

Psychological Assessment

Perceptual Assessment

Rehabilitation (psych and physical)

6. Marketing

Loss leader attractions

Product awareness

Interactive Catalog

7. Visualization

1. Books

Several to appear in 2001 including "Handbook of Virtual Environments"

John Vince, "Essential Virtual Reality"

Singhal and Zyda. "Networked Virtual Environments"

Lots of older titles and web centric ones

2. Magazines

Game Developer

VRNews

Real Time Graphic News

Computer Graphics World

Advanced Imaging

Presence Journal, presence lite

VR Psychology

3. Societies

ACM SIGGRAPH

ACM SIGCHI

ACM SIGMM

IEEE Computer: CG&A magazine

SPIE

HFS

4. Conferences

IEEE VR

ACM Siggraph

Lots of workshops and sections of other conferences

5. Web Sites