Augmented Reality: Into the Factory and Beyond

Advanced Visualization Technology (AVT) has come a long way in the past 20 years, fueled by improvements in computer processing, Graphics Processing Units (GPU), micro optics, and the increasing availability of applications to support a growing set of use cases. Raytheon has made significant investment in the research and development of AVT capabilities to improve productivity and quality both in the factory and in the field. In this article we discuss various applications of AVT with a focus on Augmented Reality and present several examples of how Raytheon has taken advantage of these.

AVT addresses multiple “realities,” with Virtual Reality (VR) and Augmented Reality (AR) being the most widely referenced. Each reality is tied closely to its visualization technology, and each has its own strengths and weaknesses depending on the given application. Figure 1 represents how these realities can be considered as being on a continuous spectrum ranging between the real world and the virtual world. Specifically, Mixed Reality (MR) provides different amounts of spatially registered digital content overlaid onto views of the real world, and it handles real world occlusion of the content. Informed Reality (IR) provides non-spatially registered digital content overlaid onto views of the real world; AR provides spatially registered digital content overlaid onto views of the real world and VR places the user in an entirely virtual environment and occludes the real world. Note that IR and AR are very similar, differing mainly in whether information is locked down to a specific point of reference.

Figure 1: Virtuality Spectrum

If you are a gamer or have seen visually stunning advertisements and media on TV or at the movies, odds are, you are familiar with today’s nearly photo-realistic VR environments. The Entertainment Industry continues to mature the underlying AVT for creating and consuming MR based content, and companies like Raytheon stand to benefit from these advances. While VR leverages a synthetic totally immersive computer generated 3D environment, AR based applications leverage computer generated information overlaid onto the user’s real world environment. Through closed visors or goggles, VR blocks out the user’s current surroundings and transports him or her to another place. AR, however, takes current reality and adds information to it. It “augments” a user’s current state of presence, often displayed on a smart phone, clear glasses or visor. While VR may be more in the gaming mainstream, AR is gaining momentum in supporting a growing set of business use cases. AR’s ability to effectively provide hands-free information while maintaining the user’s situational awareness is a critical attribute. AR helps facilitate the completion of a real world task and enables Raytheon to do business in an innovative and more cost effective manner.

AR wearables (ARW) support a wide variety of use cases across a breadth of industries, from commercial to Aerospace and Defense. They are beginning to replace computers and monitors in warehouses and on the factory floor. Common examples of this include factory assembly guidance, inspection, test support, warehouse logistics, troubleshooting, training and field support. Mobile hands-free information and remote collaboration capabilities are enabling more efficient diagnosis and resolution of problems both in the factory and out in the field. Across industry, companies are realizing the benefits of implementing AR technology in everything from product assembly, diagnostics, test and inspection to remote technical support and customer product preview. ARW promote hands-free operation contributing to faster manufacturing cycle time, improved quality, and significant reductions in post deployment problem resolution and repair times.   

At the recent Enterprise Wearable Technology Summit (EWTS 2018), the Gartner® Technology Hype Cycle was referenced multiple times in regard to the fast moving ARW industry. The Hype Cycle is a representation of a new technology’s maturation based on the evaluation of its position in five key phases of a technology’s lifecycle: Innovation Trigger; Peak of Inflated Expectations; Trough of Disillusionment; Slope of Enlightenment; and Plateau of Productivity. Based on demonstrated successes, Information Technology (IT) organizations at multiple companies are taking a leadership role in championing the push of ARW up the Slope of Enlightenment onto the Plateau of Productivity.1 As with many innovative and disruptive technologies, however, the key to success in a company is to ensure delivery of business value.

Mixed Reality Applications at Raytheon

Raytheon began evaluating AVT several years ago as a means to reduce total costs to develop, manufacture and support technologically advanced products. Since that time, many benefits of AVT have been appreciated through multiple integration, pilot and deployment efforts across the enterprise. Figure 2 describes some of the observed benefits and associated functions where these benefits are realized.

Figure 2: Augmented Reality (AR) benefits and associated functions

With the maturation of cost-effective ARW technology, users now have hands-free access to information, checklists and instructions, along with on-demand technical support. Wearables provide real-time information while allowing the user to maintain focus on the physical environment. With the advent of smartglasses (ARW with onboard processing and communication), the next generation ARW spawns a more comfortable form factor with much greater capability. While ergonomic comfort and battery life remain high on the list of ARW challenges, AR based technologies have matured to where they are now being effectively used across Raytheon.

Raytheon personnel in the factory experience significant advantages from mobile, real-time, hands-free information. Opportunities in this area include material movement logistics, assembly, inspection and test. Projected Work Instructions (PWI) augment an assembler’s reality by leveraging a vision system to detect hardware and project context sensitive step by step guidance onto a work surface. While PWI solutions come in many configurations for both small and large scale assembly, the implementation concept is the same. These systems can highlight specific locations to draw the assembler’s attention to the proper assembly points and sequence. PWI based assembly has proven to be faster, with improved quality and significant savings over traditional assembly methods.

Furthering the information visualization advantage is the ability to go mobile and hands-free. Today’s ARW smartglasses provide the comfort and capabilities to give factory floor personnel the information they need, when they need it, while maintaining attention to their assembly or inspection tasks. Figure 3 shows conceptually how ARW technology is being used today.

Figure 3: AR-based Wearables enable hands-free information display

Many next generation smartglasses hold the promise of eliminating tethers to computing devices, increasing both mobility and safety. Similar AR Wearables are being used today at Raytheon in conjunction with Universal Serial Bus (USB) camera microscopes to enable remote supplier inspection. With just one person at the supplier utilizing this remote collaboration technology, the remainder of the inspection team can provide support as needed from the home location. For example, a Subject Matter Expert (SME) can capture video evidence of the inspection remotely, eliminating the cost of traveling to the supplier’s location. Moving forward, Raytheon continues to work with customers who are interested in, and can take advantage of, remote inspection technologies.

Given the complexity of today’s products and systems, even trained maintenance personnel may need help with troubleshooting and repair. AR based remote collaboration is a key enabler in this regard, connecting the remote worker with the appropriate SME, on-demand, utilizing the expertise required for rapid problem resolution. In this way, the SMEs are able to more effectively share a broad knowledge base when and where it is needed.

In the advancement of a remote maintenance capability, Raytheon has partnered with strategic AR hardware and software suppliers and leveraged Commercial Off The Shelf (COTS) technologies to create a secure AR based remote collaboration environment. The Remote Maintenance for Reduced Manning (RM2) capability is a platform agnostic, mobile, secure augmented reality based system that enables multi-user communication for remote collaboration. Figure 4 highlights examples of the RM2 system used internally between Raytheon factories and program engineering personnel, as well as externally with suppliers and customers. In application, a SME sees the remote system via a secure encrypted audio and video connection from the remote user’s tablet or AR glasses. Their hands, posture, tools and parts are captured by a downward looking camera on the pointing pad, and the resulting image is projected back in a real-time AR overlay in the remote user’s field of view. In this way, a SME can point or use telestration (superimposing a markup over the picture) to direct the remote user with ‘ghost hands.’ And the remote user is then able to place their hands in alignment with the SME to ensure correct hand posture and position.

Figure 4: A Remote RM2 user with tablet (top), a Remote RM2 user with smartglasses (middle) and an SME’s ‘ghost hands’ providing instruction in the remote user’s field of view (bottom)

The technology and CONOPS (concept of operations) of the RM2 augmented reality hardware and software environment are well established and provide a secure responsive collaborative infrastructure worldwide. It enables a broad spectrum of collaboration across the enterprise for many use cases, from repair and upgrade to training and inspection. The primary objective is to provide effective technical support communication between a remote user (depot or field) and subject matter expert. RM2 is helping programs significantly lower manning costs and improve responsiveness through reduced reliance on travel. It leverages SME time and resources to improve utilization of key intellectual assets, while helping new programs with remote maintenance needs. RM2 is also being deployed internally to Raytheon factories, providing internal and external customers with more timely, lower cost collaborative technical support across sites. RM2 helps to significantly cut travel costs while promising a positive impact on product availability.

Challenges to AR Adoption

While AR Wearable suppliers are still improving field of view, battery life and ergonomic fit, current ARW technology is viable for office and factory use. And while most devices are not rated for harsh physical environments, deployments into some austere and remote field locations are possible. Most companies, however, still face barriers to broad implementation. Today’s AR smartglasses are wearable computers. They include a myriad of onboard sensors and networking technologies such as Wi-Fi, Bluetooth and GPS. Several versions are equipped with a variety of microphones, cameras for digital photos and video, thermal sensors and 9DOF (nine degrees of freedom) Inertial Measurement Units (IMU).2 Users of these devices must understand and comply with Information Assurance policies that address operating systems; applications; security; software updates; wireless connectivity to corporate network infrastructure; and procedural approvals for safe use of video recording equipment. If AR Wearables are to be used outside of the continental United States, Export/Import regulations must be observed and local country telecom regulations must be addressed. Questions also frequently arise regarding data ownership and licensing rights. For organizations that seek the benefits of adopting emerging technologies like AR, a key consideration is oftentimes striking the right balance between governance and innovation.

The Future of Mixed Reality at Raytheon

While Raytheon and its customers are now benefiting from the use of AR technology in the factory and in the field, we have yet to reach the Plateau of Productivity.3 Technologists in Raytheon’s companywide network of Immersive Design Centers continue to evaluate the latest generation of AR Wearables in an effort to provide lower cost AR solutions with improved comfort and performance. Feature recognition capabilities and pertinent tactile haptics are also being researched today for incorporation into next generation AR solutions. Additionally, RM2 capabilities continue to expand as the latest COTS technologies and Raytheon patented and patent pending features become available.

Keith Janasak, John Cogliandro, Brent Dingle, Adam Feccia & Kristen Stone

 

1 https://www.gartner.com/en/research/methodologies/gartner-hype-cycle

2 9DOF IMU refers to a combined 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer.

3 https://www.gartner.com/en/research/methodologies/gartner-hype-cycle