IN-CAR AR

IN-CAR AR

OVERVIEW

OVERVIEW

OVERVIEW

This is my summer (2017) internship work at Harman International, Mountain View, CA.

In 2013, the future experience team at Harman had patented the concept of in-car virtual instrument cluster. I have extended this concept to design and prototype an in-car shared hologram experience.

Due to NDA compliance, I will be discussing the content which is either publicly disclosed or permitted by the company. The content presented here represents my ideas and does not reflect the company’s views.

DURATION
3 months    Jun, 2017 — Aug, 2017.

This is my summer (2017) internship work at Harman International, Mountain View, CA.

In 2013, the future experience team at Harman had patented the concept of in-car virtual instrument cluster. I have extended this concept to design and prototype an in-car shared hologram experience.

Due to NDA compliance, I will be discussing the content which is either publicly disclosed or permitted by the company. The content presented here represents my ideas and does not reflect the company’s views.

DURATION
3 months    Jun, 2017 — Aug, 2017.

RESPONSE TO THE DEMO

RESPONSE TO THE DEMO

“The progress you have made on the project surpassed my expectations by far — fantastic job!”
 

“The progress you have made on the project surpassed my expectations by far — fantastic job!”
 

Dr. Stefan Marti, VP, Future Experience, Harman International, Mountain View, CA.

Dr. Stefan Marti, VP, Future Experience, Harman International, Mountain View, CA.

AR EXPERIENCE

AR EXPERIENCE

VIRTUAL INSTRUMENT CLUSTER

VIRTUAL INSTRUMENT CLUSTER

In this internship project, I have extended the patented virtual instrument cluster concept with redesigned infotainment system constituents and newly discovered use cases to rapid prototype an augmented reality experience. 
In this internship project, I have extended the patented virtual instrument cluster concept with redesigned infotainment system constituents and newly discovered use cases to rapid prototype an augmented reality experience. 
incar-patent

Harman’s In-Car Virtual Instrument Cluster concept

Harman’s In-Car Virtual Instrument Cluster concept

I have referred to the NHTSA AAM distraction guidelines, the patent itself, car ergonomics, and the concept video to establish my domain knowledge.

The final demo experience was presented in the car during which driver and the passenger wearing two Microsoft HoloLens devices can see the same virtual instrument cluster projected on the steering wheel. Also, the interactions (gesture, voice) were synchronized to present shared experience.

I have referred to the NHTSA AAM distraction guidelines, the patent itself, car ergonomics, and the concept video to establish my domain knowledge.

The final demo experience was presented in the car during which driver and the passenger wearing two Microsoft HoloLens devices can see the same virtual instrument cluster projected on the steering wheel. Also, the interactions (gesture, voice) were synchronized to present shared experience.

MY ROLE

MY ROLE

I led the project with product research, interviews starting in June 2017 and core interaction design, rapid prototyping in July 2017.

I worked alongside a creative lead, UX engineers, and the VP of future experience team.

Every week, I demoed my work to the team, collecting feedback on the prototype and use cases.

After the last iteration, I presented my work during the last week of August 2017.

I led the project with product research, interviews starting in June 2017 and core interaction design, rapid prototyping in July 2017.

I worked alongside a creative lead, UX engineers, and the VP of future experience team.

Every week, I demoed my work to the team, collecting feedback on the prototype and use cases.

After the last iteration, I presented my work during the last week of August 2017.

FINAL INTERACTIVE AR PROTOTYPE

FINAL INTERACTIVE AR PROTOTYPE

FINAL INTERACTIVE AR PROTOTYPE

During the last week of my internship following 8 prototype iterations, 3 variants of the prototype, I demoed the final shared hologram experience using two HoloLens devices in the BMW.

During the last week of my internship following 8 prototype iterations, 3 variants of the prototype, I demoed the final shared hologram experience using two HoloLens devices in the BMW car.

ezgif.com-optimize (26)

Actual working prototype demo captured using HoloLens

Actual working prototype demo captured using HoloLens

CHALLENGES

CHALLENGES

Crafting this whole demo experience presented challenges in terms of designing the hologram and creating the interactive prototype.

Crafting this whole demo experience presented challenges in terms of designing the hologram and creating the interactive prototype.

VIEW OBSTRUCTION

VIEW OBSTRUCTION

VIEW OBSTRUCTION

Designing motion and animations without occluding drivers’ view.

Designing motion and animations without occluding drivers’ view.

SEAMLESS INTERACTION

SEAMLESS INTERACTION

How might we enable user inputs in the real world to be absorbed in the virtual world to affect the behavior of virtual content?

How might we enable user inputs in the real world to be absorbed in the virtual world to affect the behavior of virtual content?

LIMITING TOUCH-BASED INTERACTIONS

LIMITING TOUCH-BASED INTERACTIONS

Emphasizing the need to transcend from touch interactions towards the voice, mid-air gesture, and gaze interactions.

Emphasizing the need to transcend from touch interactions towards the voice, mid-air gesture, and gaze interactions.

SHAPE-SHIFTING INTERFACE EMULATION

SHAPE-SHIFTING INTERFACE EMULATION

Perfecting the shape-shifting armrest interface synchronization with the dashboard active content.

Perfecting the shape-shifting armrest interface synchronization with the dashboard active content.

THE CONCEPT

THE CONCEPT

My first source of the concept was the video prototype provided by the team. The video indicated some of the use cases and high-level interactions while driving.

My first source of the concept was the video prototype provided by the team. The video indicated some of the use cases and high-level interactions while driving.

ezgif.com-optimize (27)
ezgif.com-optimize (15)
ezgif.com-optimize (14)

Layered interactive hologram positioned over the steering wheel

Layered interactive hologram positioned over the steering wheel

A SNAPSHOT OF THE JOURNEY

A SNAPSHOT OF THE JOURNEY

A SNAPSHOT OF THE JOURNEY

I interviewed my 4 teammates and then, 3 UX designers from Harman automotive team to synthesize potential use cases.

Considering the design challenges and prototyping requirements in mind, I have explored various Augmented Reality technologies and head-mounted-displays available.

I interviewed my 4 teammates and then, 3 UX designers from Harman automotive team to synthesize potential use cases.

Considering the design challenges and prototyping requirements in mind, I have explored various Augmented Reality technologies and head-mounted-displays available.

incar design process

Converging at various stages of prototyping | Three parallel prototypes | Final hi-fi prototype

Converging at various stages of prototyping | Three parallel prototypes | Final hi-fi prototype

DISCOVERY AND INSIGHTS

DISCOVERY AND INSIGHTS

From the interviews and literature survey, I articulated insights that predominantly mapped to navigation, communication while driving and interaction with the infotainment system.

From the interviews and literature survey, I articulated insights that predominantly mapped to navigation, communication while driving and interaction with the infotainment system.

EYE-GAZE DEPENDENT HOLOGRAM TRANSITION-IN AND TRANSITION-OUT

EYE-GAZE DEPENDENT HOLOGRAM TRANSITION-IN AND TRANSITION-OUT

Ability to position interface elements within the human FOV when desired and transition-out based on the point of eye-gaze.

Ability to position interface elements within the human FOV when desired and transition-out based on the point of eye-gaze.

incar-one

ASSOCIATING SPATIAL SOUND WITH THE HOLOGRAM COMPONENTS

ASSOCIATING SPATIAL SOUND WITH THE HOLOGRAM COMPONENTS

Depending on the spatial positioning of hologram components with respect to the driver, the spatial sound associated with them supplements accurate interactions.

Depending on the spatial positioning of hologram components with respect to the driver, the spatial sound associated with them supplements accurate interactions.

incar-two

AUGMENTING A 360° EXTERNAL VIEW AS A SEE-THROUGH

AUGMENTING A 360° EXTERNAL VIEW AS A SEE-THROUGH

Ability to look beyond the car internals by augmenting the outside view virtually for the driver assistance. A potential use case can be while parking the vehicle.

Ability to look beyond the car internals by augmenting the outside view virtually for the driver assistance. A potential use case can be while parking the vehicle.

incar-three

IN-CAR INTERACTION WITH FELLOW PASSENGERS

IN-CAR INTERACTION WITH FELLOW PASSENGERS

Preview of the fellow passengers while having a conversation (requires careful unobtrusive interaction).

Preview of the fellow passengers while having a conversation (requires careful unobtrusive interaction).

incar-four

BRAINSTORMING

BRAINSTORMING

Next milestone was to brainstorm with the team and explore ideas. In this phase, I have extended ideation based on the patent documentation and aligned with the insights found. Later, I developed basic information architecture for the hologram layers based on the literature survey recommendations.

I have converged onto three different experiences which are composed of predominant use cases that were identified.

Next milestone was to brainstorm with the team and explore ideas. In this phase, I have extended ideation based on the patent documentation and aligned with the insights found. Later, I developed basic information architecture for the hologram layers based on the literature survey recommendations.

I have converged onto three different experiences which are composed of predominant use cases that were identified.

THREE EXPERIENCES

THREE EXPERIENCES

  1. Video see-through, infotainment, in-car interaction
  2. Navigation, shape-shifting, infotainment
  3. Music, video-call, navigation
  1. Video see-through, infotainment, in-car interaction
  2. Navigation, shape-shifting, infotainment
  3. Music, video-call, navigation

AR PRODUCT RESEARCH

AR PRODUCT RESEARCH

With the design requirements and technical demands in place, I reached out to various points of sources to identify the most suitable head-mounted-display.

  1. AR/VR events
  2. HMD product manuals
  3. Contacted personal from product companies.
  4. Took expert reviews and suggestions.
  5. Online resources on product reviews

With the design requirements and technical demands in place, I reached out to various points of sources to identify the most suitable head-mounted-display.

  1. AR/VR events
  2. HMD product manuals
  3. Contacted personal from product companies.
  4. Took expert reviews and suggestions.
  5. Online resources on product reviews
INCAR-AR
INCAR-ARPRODUCTS

THE ONE

THE ONE

After several discussions with the team and considering the overall functional and non-functional requirements, I chose Microsoft HoloLens for the experience prototyping.

In particular, it contained sufficient online documentation, could be extended to design a shared hologram experience, demonstrated significantly better usability standards.

After several discussions with the team and considering the overall functional and non-functional requirements, I chose Microsoft HoloLens for the experience prototyping.

In particular, it contained sufficient online documentation, could be extended to design a shared hologram experience, demonstrated significantly better usability standards.

ms

PROTOTYPING TOOL DISCOVERY

PROTOTYPING TOOL DISCOVERY

It was challenging to find a tool that can do the just-enough-job during the initial phase of prototyping.

Based on the suggestions from my creative director, I explored Google Blocks VR and due to familiarity with Google Brush, I was able to quickly learn and prototype 3D assets, add simple transitions and interactions using Unity 3D.

It was challenging to find a tool that can do the just-enough-job during the initial phase of prototyping.

Based on the suggestions from my creative director, I explored Google Blocks VR and due to familiarity with Google Brush, I was able to quickly learn and prototype 3D assets, add simple transitions and interactions using Unity 3D.

INCAR-TOOL

THROWAWAY PROTOTYPING

THROWAWAY PROTOTYPING

Considering the concept prototype which was provided by the FX team at Harman, I moved to throwaway prototyping using Google Blocks.

My initial prototypes have been evaluated at the end of the week by the team. This presented me with the opportunities to gather feedback on the design functionalities and on the performance measures.

Considering the concept prototype which was provided by the FX team at Harman, I moved to throwaway prototyping using Google Blocks.

My initial prototypes have been evaluated at the end of the week by the team. This presented me with the opportunities to gather feedback on the design functionalities and on the performance measures.

INCAR-THROWAWAY

CONTENT ARCHITECTURE

CONTENT ARCHITECTURE

INSTRUMENT CLUSTER HOLOGRAM

INSTRUMENT CLUSTER HOLOGRAM

Based on the design principles, I created mockups of the virtual instrument cluster layers.

Minding the driver’s viewport, guidelines for positioning infotainment interfaces inside a car, and human FOV specifications, the active layer of the hologram is segregated into 3 layers.

Based on the design principles, I created mockups of the virtual instrument cluster layers.

Minding the driver’s viewport, guidelines for positioning infotainment interfaces inside a car, and human FOV specifications, the active layer of the hologram is segregated into 3 layers.

incar-steering

Hologram information architecture showing the 3 segments structured based on proximity to the driver’s viewport.

Hologram information architecture showing the 3 segments structured based on proximity to the driver’s viewport.

PRINCIPLE OF PROXIMITY

PRINCIPLE OF PROXIMITY

"Elements that are close together are perceived to be more related than elements that are farther apart."

I have adapted this principle to structure information, controls, and the status label hierarchy within each layer of the hologram.

"Elements that are close together are perceived to be more related than elements that are farther apart."

I have adapted this principle to structure information, controls, and the status label hierarchy within each layer of the hologram.

PRINCIPLE OF PROGRESSIVE DISCLOSURE

PRINCIPLE OF PROGRESSIVE DISCLOSURE

"Only necessary or requested information is presented to the driver at any given time."

Adhering to this principle, I designed secondary and tertiary layer content’s dynamics based on the modifications in the primary layer elements.

"Only necessary or requested information is presented to the driver at any given time."

Adhering to this principle, I designed secondary and tertiary layer content’s dynamics based on the modifications in the primary layer elements.

SHAPE-SHIFTING ARMREST HOLOGRAM

SHAPE-SHIFTING ARMREST HOLOGRAM

Another exciting concept from Harman showcased the use of shape-shifting armrest. I have extended my internship role to design and emulate the shape-shifting armrest concept using AR.

Another exciting concept from Harman showcased the use of shape-shifting armrest. I have extended my internship role to design and emulate the shape-shifting armrest concept using AR.

ARMREST03
ARMREST02
ARMREST01

Harman Shape-shifting controller concept | Interactions in sync with the infotainment system

Harman Shape-shifting controller concept | Interactions in sync with the infotainment system

Harman’s Shape-shifting controllers concept video

Harman’s Shape-shifting controllers concept video

I have synchronized the instrument cluster layers and the shape-shifting hologram interface to achieve the closest interaction experience.

Basically, the shape-shifting surface adapted its active controls with the active layer in the instrument cluster hologram.

I have used a tessellation based shader for achieving the target shape-shifting armrest interface.

I have synchronized the instrument cluster layers and the shape-shifting hologram interface to achieve the closest interaction experience.

Basically, the shape-shifting surface adapted its active controls with the active layer in the instrument cluster hologram.

I have used a tessellation based shader for achieving the target shape-shifting armrest interface.

SHAPE-SHIFT01

Polar substrate and frontal textures used for designing shape-shifting armrest

Polar substrate and frontal textures used for designing shape-shifting armrest

1_F-4-aYZYLJtP1pX06qCPQw
1_9QM3mIFaoLsQlwz0y3nbog

SHARED EXPERIENCE

SHARED EXPERIENCE

As the core outcome of this project was expected to be a presentable experience, I integrated the HoloLens toolkit server with the Unity prototype.

This enabled me to extend the experience to multiple HoloLens devices. Next, I created two versions of the prototype — one for Primary HoloLens (driver) and another for Secondary HoloLens (Passenger).

My initial prototypes have been evaluated at the end of the week by the team. This presented me with the opportunities to gather feedback on the design functionalities and on the performance measures.

The hologram transitions and the interactions were synced to multiple HoloLens through the server. Once the user on Primary HoloLens positions the hologram, it gets synced on Secondary HoloLens devices automatically.

As the core outcome of this project was expected to be a presentable experience, I integrated the HoloLens toolkit server with the Unity prototype.

This enabled me to extend the experience to multiple HoloLens devices. Next, I created two versions of the prototype — one for Primary HoloLens (driver) and another for Secondary HoloLens (Passenger).

My initial prototypes have been evaluated at the end of the week by the team. This presented me with the opportunities to gather feedback on the design functionalities and on the performance measures.

The hologram transitions and the interactions were synced to multiple HoloLens through the server. Once the user on Primary HoloLens positions the hologram, it gets synced on Secondary HoloLens devices automatically.

SHARED

Primary and secondary HoloLens devices synchronized during a shared experience

Primary and secondary HoloLens devices synchronized during a shared experience

INSTRUMENT CLUSTER HOLOGRAM VR MOCKUP

INSTRUMENT CLUSTER HOLOGRAM VR MOCKUP

INSTRUMENT CLUSTER HOLOGRAM VR MOCKUP

Three use cases which were chosen for the prototype were structured into layers of the hologram.

With a focus on achieving a presentable experience, the layers were automated with self-transitions in a cycle. The foremost layer is the active and engaged one.

Three use cases which were chosen for the prototype were structured into layers of the hologram.

With a focus on achieving a presentable experience, the layers were automated with self-transitions in a cycle. The foremost layer is the active and engaged one.

PARALLEL PROTOTYPING

PARALLEL PROTOTYPING

After sufficient research on HoloLens with understanding on spatial mapping, voice, gesture interactions, I approached the next phase of parallel prototyping.

This is when I created three multiple low-fidelity prototypes that varied in interactions, how they transited between different use cases and the visual aspects of it.

After sufficient research on HoloLens with understanding on spatial mapping, voice, gesture interactions, I approached the next phase of parallel prototyping.

This is when I created three multiple low-fidelity prototypes that varied in interactions, how they transited between different use cases and the visual aspects of it.

1_K-nd2GCbcjD–h-fcCFblw

Architecture of various units of the final experience.

Architecture of various units of the final experience.

LIMITATIONS

LIMITATIONS

I was able to identify some of the limitations with the help of my team members and other fellow UX interns.

I was able to identify some of the limitations with the help of my team members and other fellow UX interns.

LIMITATIONS

PERFORMANCE
The inclusion of additional HoloLens causes performance issues.

USABILITY
Prolonged use of HoloLens can cause fatigue.

MAPPING
Hologram spatial positioning requires continuous in-car mapping.

PERFORMANCE
The inclusion of additional HoloLens causes performance issues.

USABILITY
Prolonged use of HoloLens can cause fatigue.

MAPPING
Hologram spatial positioning requires continuous in-car mapping.

TOOLS AND TECHNOLOGIES USED

TOOLS AND TECHNOLOGIES USED

TOOLS

REFLECTIONS

REFLECTIONS

The initial phase of prototyping was challenging as there were few tools which I could use for creating 3D throwaway prototypes. And, having to use Google Blocks significantly aided my just-enough-prototyping. Also, using Unity for designing transitions and interactions turned out simple and robust.

Since the core requirements of the experience were already made clear, the interviews with the fellow UX designers pointed to other creative scenarios to consider.

ART OF SIMPLIFYING
One another approach that this prototyping experience taught me was to:

Redefine the problem known with additional insights, disintegrate the envisioned functional prototype or the experience, converge all the core dissimilar units from the disintegrated parts. This gets us to the input for the initial prototyping phase.

The initial phase of prototyping was challenging as there were few tools which I could use for creating 3D throwaway prototypes. And, having to use Google Blocks significantly aided my just-enough-prototyping. Also, using Unity for designing transitions and interactions turned out simple and robust.

Since the core requirements of the experience were already made clear, the interviews with the fellow UX designers pointed to other creative scenarios to consider.

ART OF SIMPLIFYING
One another approach that this prototyping experience taught me was to:

Redefine the problem known with additional insights, disintegrate the envisioned functional prototype or the experience, converge all the core dissimilar units from the disintegrated parts. This gets us to the input for the initial prototyping phase.

email-outline (1)
linkedin-box1
github-box1
twitter-box1
dribbble
medium

Copyrights 2018 @ Pruthviraj. All rights reserved.