Chasing Pixels - Finding Gems - Home
Pixel Perfect Is Critical in Augmented Reality
JAN 11, 2017 17:54 PM
A+ A A-
Pixel Perfect Is Critical in Augmented Reality
By  Dr. Jon Peddie
By now anyone who hasn’t heard of augmented reality (AR) and/or virtual reality (VR), have either been living without Wi-Fi, or on a deep space mission. Most people have probably also heard of mixed reality (MR). However, hearing about it, and possibly even trying it, does not equate to understanding it. Most of us drive a car, but how many understand how it works? And because all three technologies involve a headset, it is too often assumed they are somehow related, similar, or a subset of one another—they are not. 
I want to be augmented, not virtual
One of my currently favorite games is Deus Ex: Mankind Divided. It’s a (AAA FPS) game about people in the future who have been significantly physically enhanced with cybernetics, nanites and nano-technological implants, creating an advanced super-human superior to an ordinary one. We’re a long way from that, but we can be enhanced today in terms of information access. I’m talking about the use of augmented reality devices, which can be a mobile device (tablet, smartphone, or notebook), headsets and add-ons for glasses, helmets, head-up displays, and maybe even contact lenses.
Whether it's the heads-up display in the car or the jet-fighter, the highlighted scrimmage line in American football and digital signboards in the baseball outfield, the notifications in Google Glass, the Pokemon Go monsters dancing on your screen, or the super-imposed language translations on the phone, AR provides added information that can be very beneficial. 
But it doesn’t come easily. There is an enormous amount of technology needed to do it right, and to do it robustly. 
Where am I (and where was I)?
Consider the scenario where you are wearing an attractive, inconspicuous pair of glasses, maybe sun glasses, that are augmented reality glasses. These are the best augmented reality glasses available and record everything you see and hear, while superimposing on what you see and hear additional information that you have either specifically asked for or that your profile and current circumstances can use. You are wearing a personal black box which becomes the honest witness in any situation you are in or see. 
However, to be that honest witness, your glasses have to know precisely where you are, and where you have been. That is a lot harder than it sounds, especially when you add of the demand that they be lightweight, mobile (battery powered), and unremarkable in appearance.
To do that you need an inertial measurement unit (IMU), a gyroscope, and a barometric sensor. They all have to update at least 100 times a second, compensate for drift, temperature, and vibration, and be very accurate with high resolution. And they have to be tiny, and not use much power, 
Now we know where you are, and have been in three geometric-axis and one temporal axis, that information gets stored and transmitted.
Where am I looking (and at what)?
But there’s more to life than just four dimensions, because we need to let our augmented reality device know precisely what we’re looking at. We are seldom exactly perpendicular to the surface of the earth (in x or y), we tend to lean which is called roll and pitch, and sometimes twist about (which is called yaw), which is our pose. More sensors are needed, and because we need to know where we are when we’re rolling and pitching, we could really use a map to help us get oriented. But what if a map isn’t available, such as when in a building or the underground? Well if we had one good location fix, we can with help of some clever software called simultaneous localization and mapping (SLAM) figure it all out. Then by identifying way posts (doorways, walls, windows, etc.,.) we can build a map. 
All that positional information is used with what the front-facing camera is seeing. And in the case of helmets and sports augmented reality glasses, there is also a rear-view camera that needs positional information too.
Now we’ve got the platform. And with it we can superimpose synthetic images and have them look correctly placed. But and but, those images have to have their own set of eight dimensions and in addition to time being the seventh, we also need to know their perspective (vanishing point). By now you’ve got it that there is a lot matrix math and transformations going on that’s linked to real world coordinates and virtual objects’ dimensions. And it’s being updated at a refresh rate of at least 30 fps and potentially has HD resolution, in full color. This is a very demanding computer graphics requirement. 
But wait there’s more. The virtual object has to be in the proper field of focus, what’s known as our vergence. You don’t want a Pokemon monster bigger than the building he’s standing next to, nor do you want him out of focus, any more than you want your navigation instructions, or translations out of focus.
What am I seeing (that’s not really there)?
So now we’ve located ourselves, and know how fast we’re traveling, and how much we’re leaning, or swaying, and trying to impinge virtual objects (which could be marquee or simple couple of lines of text) on to (or into) what we are seeing. If it doesn’t look right our amazing eye-brain system will detect that in a couple of milliseconds. This is very similar to what happens in the case of the uncanny valley when animated characters aren’t quite real enough. We sense its falseness and then struggle trying to make it right or coping with it. That cause stress, fatigue, and if persistent and annoying enough, abandonment.
To get all this right, 30 times a second or more, it has to be pixel perfect. Pixels and position perfect. It has to be in focus, it has to be perspectively and color correct, it can’t sparkle, flicker or jitter, and it has to properly opaque. Pokemon can occlude things behind him, but navigation instructions can’t. How transparent overlaid information is will be function of ambient lighting conditions, yet another parameter to be dealt with.
If you’ve ever experienced virtual reality, by now you should be either shaking or nodding your head and thinking, boy, is augmented reality ever more complicated and challenging than virtual reality. And if you’ve experienced or read about virtual reality you’ve no doubt heard of or experienced the issue of latency. If you think latency is an issue in virtual reality, wait till you experience it in augmented reality (let’s hope you never do.). If the virtual objects or images swim around when you move your head or eyes, it will take less than three minutes for you to either rip off the headset and/or throw up. 
And that raises one more issue (and I promise I’ll stop with this one) – birefringence or aberration. Because the optics in an augmented reality system are so much more demanding than in virtual reality, they have to be perfect. Here again, our wonderful eyes, with their amazing high-speed cones and crazy high-dynamic range rods will sense any optics distortions in milliseconds or less. 
 Did I say pixel perfect? If I had given you specification requirement with the above parameters, you probably would have told me to come back in 20 twenty years when the technology might be ready. And yet there are companies working on this today and getting ever close to such amazing systems. They all have a compromise somewhere and because of that we don’t really have a consumer version yet, and all of the industrial-scientific versions are targeted at specific vertical markets. But it’s coming, and coming fast, and in 2017 you see the first consumer glasses, that look like normal glasses (and even have corrective lenses). 
Me, I can’t wait to be augmented.
[%= name %]
[%= createDate %]
[%= comment %]
Share this:
Please login to enter a comment: