Dr Simon Taylor, Founder & Research Director, Zappar
Last week we attended the Wearable Technology Show at the Excel Arena in London. The event included an Augmented Reality track, and I gave a talk based on my previous blog giving my personal opinions on the requirements for AR wearables if they are to deliver engaging consumer-facing experiences and reach mass consumer adoption.
As the overall event was focused on wearables, I was keen to explore how Zappar’s tools and platform might enable AR experiences on these potential hardware devices of the future. I had the thought a couple of weeks before the show that Google Cardboard might provide an interesting platform to begin to explore how AR in general and Zappar in particular might work in this more immersive setting.
For those unfamiliar with Google Cardboard, it was given out to attendees at the Google IO 2014 conference last summer. It’s a flat-pack cardboard VR viewer along with a couple of cheap lenses that uses a smartphone to provide both the display and the motion sensors to enable 360 degree VR experiences. Google released the design with an open license, and it's now possible to buy entire kits to make your own from a range of sellers on the Internet for roughly £15/$20.
My now slightly battered development Google Cardboard device. There are a few modifications of note: the view on iPhone 6 cameras is no longer blocked, I can now use it without needing to remove my glasses, and a disposable tissue has been added to reduce the issue of unsightly and probably unhygienic sweat marks on the cardboard!
After pulling a few late nights with some other members of the team we were able to get the first milestone completed so we could unveil a proof-of-concept demo at the show. Modifying Zappar’s rendering code to produce stereo images proved reasonably straightforward. A trickier challenge was using the correct viewing angles for the eye views and correctly applying the distortion correction to the rendered images – both functions that are provided in the Google Cardboard SDK. Google's Cardboard SDK is only available for Java, and integrating that with the C++ Zappar rendering code would have been a reasonably involved task. Thankfully there is an open-source port of the Cardboard SDK to Objective-C, which was far easier to integrate with our core rendering code and allowed us to build the demo on iOS (which also offers a better development and debugging experience for C++ and Objective-C code).
We put together a very simple demo zap using an animation of a bunny coming out of a hole in the ground that we had used on a previous Easter campaign with Asda. The first thing to notice in the screenshots below is the eye field of view is significantly larger than the field of view provided by the camera. We decided to render the virtual content over the full field of view available to the eye to provide the most immersive experience possible.
As long as the real physical size of the tracked image is known then it is possible to render the virtual content with the correct stereo cues so that it appears the correct distance away from the user. However rendering the camera background is not as simple as with only a single camera image it is not easy to calculate the depths of the different objects in the view. The approach we took for this first proof-of-concept was to simply render the camera image as if it was a view of a flat surface 1.5m away from the user. Although that worked reasonably well for “scanning mode” and with the bunny content we were using for the demo, with smaller target images viewed from closer distances the content appears to float above the target.
The demo felt better in practice than I was expecting, and served its purpose as an interesting talking point at the show. It's a compelling enough experience to justify continuing development to see how close we can get to a mock-up of the potential uses of these future AR eyewear devices using nothing but current smartphones, a bit of cardboard, and a couple of lenses.
There are a few areas that we're planning to work on to improve the current proof-of-concept. Probably the biggest issue is the complete lack of any input - Zappar currently relies on touchscreen input to interact with zaps, and obviously this is not possible when your phone is placed in the Google Cardboard. Cardboard comes with a magnet switch that is detected by monitoring the phone’s magnetometer and reported by the SDK. Unfortunately it’s pretty flaky on many devices, and is the equivalent of a single button, meaning most UIs in other apps available on Cardboard involve turning your head to look at an option and then flicking the magnet switch. I have some ideas about how to provide richer input support for Zappar Cardboard that I'll blog about later if it ends up working!
Zappar’s current experiences tend to take place on and around a target image. One thing I noticed with people trying the Zappar Cardboard demo at the show is that they tried to look around a much larger environment. That is potentially due to their previous exposure to VR experiences based on head orientation tracking, or it may just be a natural response to the more immersive setting. A simple approach here would be to use the gyroscope when the target image is not visible, but I'd like to see if it's possible to do better than that and provide full 3D head movement tracking (rather than just orientation) in larger environments.
Finally there's the issue of rendering the camera image with appropriate stereo depth cues so the AR content feels truly part of the real world environment. The next step here when looking at simple flat targets is just to render the camera image as though everything in the image is on the same surface as the target image, which should solve the “floating” content issue I described earlier. Things get more complicated in larger environments, but that's a challenge for slightly further down the road.
Are we going to release this? Clearly we don't consider the Cardboard hardware a realistic prospect for mass consumer usage, and we’re not currently planning to enter the hardware market ourselves. However once we solve a few more of the issues described above I'd love to share some demos with the enthusiast community who have already purchased Google Cardboard devices.
We'll be continuing to work on improving the Zappar Cardboard experience along with improving the flexibility of our tools and platform to handle the differences between viewing methods, so keep following the Zappar Blog for progress updates!