Head-mounted Displays (HMD)

What are Head-mounted Displays?

Head-mounted displays or HMDs are probably the most instantly recognizable objects associated with virtual reality. They are sometimes reffered to as Virtual Reality headsets or VR glasses. As you might have guessed from the name, these are display devices that are attached to your head and present visuals directly to your eyes. At a minimum, if a device conforms to those two criteria you may consider it an HMD in the broadest sense.

HMDs are not the sole purview of virtual reality, they have been used in military, medical and engineering contexts to name but a few. Some HMDs allow the user to see through them, allowing digital information to be projected onto the real world. Something which is commonly referred to as augmented reality.

When we look at the diversity of HMDs that exist today within the context of virtual reality, it becomes apparent that there’s much more to these devices than strapping two screens to your eyes. In order to allow for an immersive experience either as a personal media device or as a full-on virtual reality interface, there are a number of technologies that can be incorporated in an HMD. Let’s have a look at the most important ones you should be aware of.

Display Technology

Clearly the display is one of the most important components in an HMD. After all it’s the part of the device you’ll be most conscious of during use. Today HMDs use various technologies to get pictures to eyeballs, but the most common display technology uses liquid crystals. More commonly known as an LCD panel, the same type of panel used in smartphones, televisions and computer monitors. Another similar looking display technology known as OLED (Organic Light emitting Diode) is also finding its way into these devices and there are HMDs with OLED displays out there already.

Pixels and displays

Thanks to smartphones and tablet computers there has been somewhat of an arms race to produce small displays only a few inches across with very high pixel densities. Pixel (short for picture elements) are the little dots that make up a picture. The more of them you have in every square inch of display the crisper the image. According to Steve Jobs, the late CEO and founder of Apple corporation, once you have more than 300 pixels per inch (ppi) the human eye can no longer discern individual pixels at 10 to 12 inches. High end phone displays are now heading for double that pixel density, which means for normal smartphone use that extra density is wasted. However, in an HMD where your eyes are only a few inches from the display that extra pixel density can mean the difference between crisp images and a fuzzy mess.

Retinal projection

Another display technology that hasn’t yet seen widespread use, but does exist in some headsets such as The Avegant Glyph, is retinal projection. They use tiny digital projectors that use microscopic mirrors to project onto your retina. Effectively using the back of your own eyeball as the screen. Proponents of retinal projection claim many advantages in terms of quality and eye strain compared to LCD and OLED HMDs, but due to the current state of the technology retinal projection cannot yet provide the immersive field of view that other HMD technologies can.

Two final aspects of HMD displays that are quite important are refresh rate and latency.

Sony HMZ-T3W Head Mounted 3D Viewer

Refresh rate

Refresh rate refers to how quickly a display can change its contents within a span of time. Typically LCD computer monitors can do this 60 times per second or at 60Hz. This also corresponds to a maximum frame rate of 60 frames per second. One frame being one complete and discrete picture on the screen. Cinematic film typically runs at a framerate of 24fps. Lately some newer films like The Hobbit have transitioned to 48fps. To audiences this makes the film appear very smooth and “hyper real”, something that has had a mixed reception. For web video such as that found on YouTube 60fps is starting to gain support, especially for action film taken with cameras such as the GoPro. To put it simply, the more frames you display in a second, the smoother and crisper motion appears. Since virtual reality is meant to enable a feeling or presence and immersion it’s fair to ask what the right refresh rate to achieve that would be. It turns out that 60fps is a working minimum, but 90fps appears to be the sweet spot. Some HMDs even support 120Hz refresh rates.

Latency

Latency is the time gap between an input and an output. For example, if you turn your head in a virtual reality world, but the picture takes a second or two to catch up to your new head position, you are experiencing severe latency. In order to fool your brain’s visual system, virtual reality requires very low latencies. Usually 20ms or less for an absolutely top-notch experience. Unfortunately latency is not a simple issue to resolve and it isn’t solely the result of your display choice. The total latency between input and output is the result of the entire chain between those two points. From the positional sensors to the computer hardware rendering the image to the display itself, each component adds a small delay to the total time. Therefore a low-latency display is a must, but it is not always enough by itself.

Optics

If you were to take a phone LCD display and hold it to your face, chances are it wouldn’t do much for you. In order to create the immersive feeling of being in a virtual world it is necessary to take the flat image on the screen and magnify it to fill our visual field. Careful experimentation by a team at the University of Southern Carolina indicated that any HMD that wanted to achieve the edge-less, immersive visuals needed for convincing virtual reality would need a field of view (FOV) of between 90 and 100 degrees. The lenses in an HMD play a key role in taking the flat image on the screen and turning it into something that fills a substantial area of our visual field. Our field of vision isn’t rectangular like a screen, nor is it flat, so optical trickery is a necessity to make the illusion work. There are many different optical designs for HMDs and also different approaches to what lenses should be used and why, but one universal is that the quality of the lens is important. An HMD that uses cheap lenses may have poor picture quality, clarity and unwanted distortion. Often the most drastic after-market upgrade that can be done on an HMD is the installation of superior lenses.

Head Tracking

It’s all good and well that you can see the picture clearly, but without knowing the position of your head the computer doesn’t know where you are looking. Modern HMDs use various technologies in order to accurately track head position. Thanks to advances in smartphone technology we can now put a multi-axis accelerometer on a chip and infrared tracking cameras can accurately watch markers on the HMD, relaying positional data to the computer. Mobile HMDs that are not for use in a fixed location can’t make use of external camera tracking, for obvious reasons, but some new technologies such as the Microsoft Hololens and Google Project Tango can use multiple sensors in addition to accelerometers for positional calculation.

It’s important to note that some HMDs, especially those that use your smartphone, can only track what direction you are looking. Dedicated HMDs often track another axis, also letting you “lean” in for a closer look. This is an important element of immersion, since that one of the ways we look at real objects in the real world.

Eye Tracking

At the time of writing only one HMD, the FOVE, promises to integrate eye tracking technology. 3rd parties are however offering upgrade packages for other HMD products.

Eye tracking allows the HMD to calculate where your eyes are looking and then do something with that information. For example, it could change the depth of field of the visuals on screen to simulate natural vision more closely, virtual characters can now react to your gaze or you can now use your eyes to quickly select menu items in the virtual world.

Eye tracking could be a very important input for general purposes, allowing us to interact with user interfaces in more natural ways.

It is still early days for eye tracking technology in virtual reality, only time will tell what use cases developers will come up with.

Audio Hardware

There isn’t much to say about audio in HMDs, some HMDs include headphones and others do not. More often than not you will have the option of using your own headphones, with any provided pair being removable, There are a range of audio options available, including positional, multi-speaker headsets.

Computer Hardware

An HMD is both an input and an output device, tracking your head movements and relaying graphics to your eyes. In between those two processes lies computing hardware. There are really only three categories of HMD here. The first is completely self-contained and possesses all the computer hardware necessary for VR within the HMD itself or otherwise attached to the body. These are mobile, battery powered systems. Usually this hardware is repurposed from smartphones or might literally use a smartphone to perform the needed tasks. The second type of HMD does not have any onboard computing power, but interfaces with an external computer. Usually the HMD accepts a High-Definition Multimedia Interface *(HDMI) input and uses a Universal Serial Bus (USB) connector to send head tracking data. The third class of device is one that acts as both, having its own onboard hardware, but also allowing input from external devices.

Although smartphone hardware has become powerful enough to provide reasonable virtual reality experiences, they still lag far behind what is possible  with powerful computer hardware or the major mainstream video game consoles. In terms of pure visual fidelity and frame rate therefore, dedicated external computers are still the best choice. Using such a computer for virtual reality in future doesn’t need to leave us tethered to our desks though. Wireless display links exist, but getting them to work for virtual reality within the tight latency requirements is easier said than done.

Other Hardware

Now we are left with more mundane things such as the housing and other creature comforts. HMDs are made from all sorts of materials: cardboard, plastic, metal and anything else that will hold the parts together. It’s important to consider what adjustments are available on a particular HMD. The adjustment range of the headstrap is important in this regard. If you wear glasses make sure the HMD will accommodate them or allow for lens adjustments that makes them unnecessary. Finally, the comfort padding and ergonomics of the HMD are often overlooked, but very important. After all, the HMD spends a lot of time strapped to the user’s face.

Companion Input Devices

As mentioned above, the HMD can capture information about your head position, but unless you are happy to stand in one spot without moving or interacting with anything, more forms of input are needed. We deal with these input devices in detail in the appropriate section of the site, but for the sake of completeness in this overview it is worth mentioning a few. At present the most mainstream way of navigating virtual worlds is with existing videogame peripherals. These include gamepads, flight sticks, racing wheels and of course the keyboard and mouse. Several more immersive devices meant specifically for VR are available or in development, such as omnidirectional treadmills and specialised devices such as the SteamVR controllers.

At the very high end you might find full-body suspension and motion tracking systems, active mechanical force feedback or elaborate hydraulic vehicle simulation rigs. These all work in collaboration with the HMD to allow for interactivity and even greater immersion.

So How Does It All Work?

Setting aside less common technologies such as retinal projection, most HMDs that use LCD or OLED displays work by presenting each eye with a similar, but slightly offset image. This provides the illusion of stereoscopy. What most people think of as 3D imagery. As you might guess this needs a separate display for each eye, but in order to save on cost and complexity most HMDs use a single display panel that shows both images, but uses a plastic divider to prevent each eye from seeing the other eye’s image.

The actual images do not fill the display from edge to edge and are not perfectly square. If you were to look at the screen directly you’d see two images with fuzzy grey edges, this is a simulation of our visual field with the sharp image at the centre with curvature and gradual loss of acuity towards the edges of the image. Viewed through the lenses at the right distance, the picture neatly fits into our visual field and appears natural, as if we are looking at the real scene, not a picture of it.

So, when it all comes together you will feel like you are present in a virtual world. Wherever you look, you will see a virtual reality, replacing the real world around you. This is how the HMD achieves the illusion of virtual reality.

Conclusion

This was a broad overview of HMDs, be sure to check our in depth articles on individual HMD products that are on the market or are in development. Armed with the knowledge above you will have no trouble at all understanding the range and variety of devices on offer.