How Does Augmented Reality Affect the Automotive Industry Today?

Years ago, Morton Heilig, a cinematographer and father of VR, might not have suspected that people with small handheld devices would rush the streets in a search of their virtual friend. Сraze for Pokémon Go, beards and rabbit’s ears in MSQRD app and Snapchat lens – augmented reality has been a toy in our hands for many years. The affinity with VR largely exploited in gaming industry has slightly cast a shadow over augmented reality attributed mainly to an entertainment sector.

Augmented Reality Importance for Automotive Industry

However, AR goes beyond gaming and AR masks creation. Besides already existing educational apps (e.g., BBC’s Civilisations AR) or instructional augmented reality apps (e.g., Street View mode in Google Maps overlaying AR walking directions atop the real world; Gatwick’s ARpassenger app for easier navigation in the airport terminal buildings), AR finds a variety of ways to be innovatively applied in the automotive industry:

• Inside a vehicle (e.g., Holographic Augmented Reality Navigation System built into a car – read more in the Use Cases section below);
• Outside a vehicle as a part of its maintenance (e.g., Volkswagen’s MARTA system for service technicians);
• And for the vehicle creation. Augmented reality in manufacturing becomes a far-reaching integration that automakers aspire implementing on their factories. For example, Tesla has patented an AR-based system to automate their models setup, configuration, calibration, and quality checkup.

Generally, today all industries see larger potential for adopting augmented reality. For example, virtual reality and augmented reality market is foreshadowed to hit $209.2 billion by 2022 whilst 86% of business representatives reckon on livening up their augmented reality strategies within 12 months.

Going by the number of investments in AR, automotive industry will apparently leapfrog other industries by 2020. See the table below:

Source

So, why do automakers and automotive software/hardware companies anticipate implementing augmented reality as part of their business strategies?

Augmented reality accompanied with computer vision and IoT sustains the core of advanced navigation solutions and ADAS systems. The latter results in enhanced safety for a driver and contributes to the development of an automotive connectivity ecosystem. Besides, AR will be soon indispensable from autonomous augmented driving features offering a new level of interaction.

Recently at CES 2019, Nissan has presented its groundbreaking Invisible-to-Visible technology (I2V) concept that creates “a 360-degree virtual space around the car” to provide real-time road data and make manual navigation assistance smarter and safer. Actually, this is a new-age AR HUD that extends beyond the limits of a windshield glass. But what about distractions? A half-autonomous driver support system is expected to assist driver in becoming more focused together with Omni-sensing technology, onboard AI system that can also track whether a driver is focused on the road. Moreover, merging data from sensors inside and outside the vehicle and cloud, the AI-enabled system is capable of monitoring the vehicle’s nearby objects and everything that lays far ahead. In the future of fully autonomous vehicles, the I2V technology relying on Metaverse (a virtual environment used by avatars of real people for communication and their personal representation) will be used for the social interaction with virtual avatars of friends or family members appearing inside the vehicle. Watch this video for more details:

Another reason why AR should be considered by businesses is that it will continue stirring up drivers’ interest who are eager to try a new toy in their hands. Most likely, these are the drivers who have unintentionally created the virus effect around augmented reality mobile applications. So, imagine how many potential buyers automakers can receive from the list of mobile app users.

Finally, implementation of AR systems for vehicles has become easier these days. Relative affordability of hardware and development frameworks creates broader opportunities for augmented reality adoption. Besides, technological advancements let augmented reality applications move from labs to the world of commerce bringing it closer to people. Among them are:

• Appearance of 3G, 4G, and 5G networks enabled higher speed data transfer and lower latency;
• Smart cameras and faster speed of rendering. Here’s a good example of a powerful smart camera used in modern vehicles and driven by Computer Vision and Data Fusion.
• Appearance of more powerful processors, sensors and IoT devices;
• Higher computation power;
• Open source AR SDKs, tools, frameworks at AR software developers’ disposal.

What is Augmented Reality Compared to Virtual and Mixed Reality?

Basically, there are three types of extended reality (XR) depending on the degree of the user’s immersion. They can be used interchangeably or concurrently to cover the whole range of business purposes.

Augmented reality is the process of overlaying digital objects on an existing natural environment to provide the partial immersion into the virtual environment.

Unlike AR, virtual reality substitutes the real world with an artificial surroundings creating total visionary and sensory immersion.

Mixed reality is about naturally adding the virtual objects (holograms) to our physical environment in such a way that you can interact with them by using holographic devices.

So, how does augmented reality work? To understand this, have a look at this visually rich comparison of all XR types:

Source

Augmented Reality Hardware and its Use Cases in Automotive

When you interact with an augmented reality app, your real world and experience as well can be enhanced with the added computer-generated text, graphics, sounds, and even haptic feedback. For many users, AR is about headsets, smartphones, and applications. But do you know what actually stands behind AR technology? Here are some key components it reposes on:

• Sensors and cameras that allow collecting data by scanning our physical space and objects. At this phase, depth tracking and simultaneous localization and mapping (SLAM) technologies are utilized to determine distance to the objects.
• Augmented reality devices require enough processing power to function as mini computers to accurately process gathered data from sensors and cameras. Usually, standard AR devices include such elements as CPU, GPU, flash memory, RAM, Bluetooth/WiFi microchip, and GPS microchip. However, the advanced AR devices like Microsoft HoloLens rely on their custom holographic processing unit (HPU) and inertial measurement unit (IMU) consisting of accelerometer, magnetometer, and gyroscope.
• Mounted projector, which sends the collected data onto the surface of AR headsets, smart glasses or lenses.
• Reflection mirrors that by reflecting light to a camera and the human’s eye properly align an image.

Depending on the hardware used by automakers, there’s a variety of possible use cases in automotive:

Augmented reality glasses is an extension of a smartphone displaying all digital information in front of the user’s eyes, e.g., Google Glass, ODG’s R7/R8/R9, Vuzix Blade, Solos for cyclists, Magic Leap augmented reality goggles, etc. They offer activity tracking notifications (e.g., heart rate, speed detection), turn-by-turn navigation, voice assistance, and most importantly, necessary navigation alerts when driving. AR smart glasses are still in their infant stage because of the limited battery life, bulky forms, intrusiveness and limited number of use cases apart from its entertainment and information presentation purposes. However, in 2018 Intel has made a leap forward greatly improving the AR glasses technology introduced in Vaunt product based on low-power laser VCSEL: Unfortunately, the project was soon closed which again proves that AR glass hardware is still not ready for the mass adoption.Much-feared future described in one Black Mirror episode is more of reality today than ever. Sony and Samsung have already received patents for the development of AR contact lenses fitted with a teeny-tiny camera, display, antenna and sensors that will spot the movement of voluntary or involuntary blinks to record and store data and allow a user to interact with the displayed virtual objects on the lenses.

Virtual Reality and its Use Cases in Automotive

Although today’s VR hardware surpasses first VR headset appliances fixed to the ceiling not to crash a man’s head, it is still too bulky and expensive for the mass use. Moreover, as VR has a long history of its use for gaming and entertainment needs (e.g., watching 3D movies, 3D virtual tours etc.), businesses see less opportunities for investing in VR – $90 billion of investments in AR compared to $30 billion in VR by 2020 according to a Digi-Capital study. Moreover, the choice for using VR in automotive is restricted to a few use cases, apart from in-car entertainment (e.g., at CES 2019, Audi demonstrated a demo of its Experience Ride technology for the rear passengers). The most popular today is virtual reality apps for test drives that actually transform a traditional approach to the first ride. At the pre-sale stage, VR can be a sort of a marketing bait to entice potential buyers to try out a new vehicle model or features in the dealer’s centers. This way VR test drives allow generating more audience (e.g., 1,500 people tried Ford’s VR test drives during one auto show), cut costs on vehicle transportation and collect firsthand feedback.

Mixed Reality and its Use Cases in Automotive

This term says for itself combining the best parts of virtual and augmented reality (e.g., Microsoft HoloLens, Meta 2, Magic Leap). Some sources also call it an advanced form of augmented reality when the connection of a user with the real world isn’t blurred but enhanced with virtual objects realistically filling the space around us. Imagine one of the scenes from the famous Jumanji movie, like rhinos naturally crashing through the living room or monkeys riding the police car. By using mixed reality technology, ginning up similar game settings becomes quite attainable today.

For the vehicle design and engineering teams, the potential of AR and MR allows achieving the desired accuracy of the car model virtual representation. Thanks to it, conventional clay modeling can be replaced with 3D visualization which designers can interact with in real-time. Most importantly, it vanishes the need to create a physical prototype allowing to create dozens of iterations of a vehicle model or its separate elements, their shape, position, etc. This way AR helps speeding up the development process and delivering the vehicle prototype faster.

A good example is Ford’s new-age design studio that puts into practice Microsoft’s HoloLens Mixed Reality to reshape own approach to the layout process of a new car model:

Mixed Reality and Haptic Feedback

What if you could touch holograms and sense what they are, feel their texture and motion? Haptic feedback is a next level of mixed reality which still needs to be developed, tested and refined until it becomes ready for mass distribution and use. The only form of haptic feedback we can easily access today is our smartphones’ ringtone vibration. Others require special haptic glove which is still in its development stage.

There are two essential kinds of haptic feedback that can help us sense the virtual world:

• Kinesthetic haptics or force feedback is about influencing the human’s body with force, vibration, or motion. Gamers are well accustomed to force feedback which is already available in their gaming wheel, seat, mouse and other parts of a playset. But even with all its advantages, a study of a human-robot interaction reports that force-based haptics is less effective in recreation of realistic VR experience.
• On contrary, cutaneous haptics that influence our skin with a complex of slight vibrations and touches is able to fool the human mind better making it believe that it actually touches something real.

Let’s look at the way such types of feedback are realized in a haptic glove relying on 120 sensors to make a person feel the legs of spider or rain:

Although the bulky glove with wires can hardly fit your home settings, we are still to see great advancements in this area. Besides, the ability of haptic feedback to result in genuine senses may ramp up impact of VR on larger audiences and justify high costs of devices. For automotive industry, haptic feedback may completely change an approach to sales process, replacing a standard dealership center with a comprehensive VR showroom for making a test drive of any vehicle at any time.

The bottom line is that AR, VR and MR have a full potential to amplify a number of other use cases and integrate into the ecosystem of a vehicle, its production, sales process and generally, the entire automotive value chain. The more hardware improves, the deeper AR will root into all aspects of automotive industry. Find out in our second article “Augmented Reality Benefits for the Entire Automotive Value Chain” what opportunities AR brings to OEMs and Tier-1 suppliers and what challenges should be tackled to start large-scale AR adoption.

Contact us today to discuss partnership options and opportunities for your automotive AR project.