Simulator display systems: More than projection alone

Dr Joetey Attariwala

Simulator display systems

In order to adequately prepare modern warfighters for the battlefield, demands on simulator display systems are growing, calling for increased brightness, contrast and visual acuity. While achieving near-reality used to be the goal, now it is a requirement.

CAE’s Prodigy IG leverages Epic Games’ Unreal Engine, a state-of-the-art gaming engine that delivers high-fidelity graphics. (Photo: CAE)

Realistic displays have been a requirement for military simulators for over 40 years. From Tron movie grid-like experiences in the 1980s to the newest immersive HMDs, this demand has grown to include larger FOVs, higher resolution and more portable systems. These factors are especially important for flight simulators, which play a crucial and growing role in training military aircrews worldwide.

Driving the quantitative and qualitative demand for FFS are increasing costs of live training, growing restrictions on training airspace, pilot shortages and the need for highly realistic training scenarios. Combined with technology evolutions related to the core of FFS – projectors and image generators (IGs) – this results in significant opportunities and risk management challenges.

‘No aspect of a simulator system operates in isolation; all hardware components and software must be able to integrate seamlessly in order to provide a training experience for the user that replicates reality as closely as possible,’ said Richard Rybacki, co-founder of MVRsimulation.

‘For this reason, a lot of work goes into making sure our Virtual Reality Scene Generator (VRSG) image generator software – which provides the 3D images and models that create the simulated “world” on the display – is able to render according to that device’s requirements, both in order to allow users to maximise the advantages each device offers and to minimise any negative effects caused by its limitations.’

Growing demands

The visual display can make or break the successful implementation of a multimillion-dollar FFS. Displays encompass a variety of form factors, capabilities and projection techniques. Training organisations and industry partners must therefore be well versed in the ever-evolving technologies in this field.

Military FFS displays are specified and designed to meet stringent demands related to visual acuity, brightness, contrast, image stability and others. Particularly challenging is the fact that military trainers often must replicate accurate day and night scenes, which requires the simulation or stimulation of operational night vision devices.

‘The original training methodologies, still in sorrowful use today, were based on World War II needs. Basic airmanship, leading to bombs on target and the return home, is no longer enough,’ said Dr James J Frey, former USN pilot and executive director of strategic growth at PAR Government. ‘The modern warfighter has a far more complex task at hand, and, in many cases, they are positioned to have to make globally consequential decisions on the fly,’ he noted.

‘Industry is now moving toward a common language in search of Joint Mission Rehearsal and Support. This means that a common and correlated highly detailed “now terrain environment” is needed for formation and joint training. Visuals have to get more precise than they used to be since visual acuity is increasingly important. Rehearsal with near-reality has always been the goal, but now it’s a requirement.’

Display systems for military applications have typically used domes or large surfaces requiring detailed calibration procedures. More recently, lighter LED monitor configurations have been introduced which are combined and aligned to reduce mullion (bezel) effects. Industry is further enabling military customers with the ability to train through HMDs which are used for XR.

MVRsimulation has integrated its Part Task Mission Trainer with the Varjo XR-3 MR headset to provide a fully immersive world for the trainee. (Photo: MVRsimulation)

It is for this reason that MVRsimulation designed VRSG to render on regular and curved gaming screens, dome displays and HMDs. ‘We are seeing improved training realism from the use of mixed-reality headsets such as the Varjo XR-3, and this is a trend we expect to continue,’ said Rybacki. ‘We have designed our own simulator systems – the Deployable Joint Fires Trainer and Part Task Mission Trainer – to integrate with the XR-3 in order to provide a fully immersive world for the trainee.’

An example of recent advances is the USAF Academy’s Multi-Domain Laboratory (MDL) which employs virtual machine cloud architecture with allocatable CPUs and GPUs per simulator. The MDL, developed by Salient, PLEXSYS and ZedaSoft, opened in September 2021 and showcases the latest in gaming-style flight simulators.

Types of displays

The primary displays in use incorporate direct, rear and collimated projection techniques. While collimation does involve rear projection, ‘collimated displays’ differ from ‘rear-projection (RP) domes’ as they satisfy different and specific requirements.

Front (or direct) projection displays entail pointing multiple projectors (or channels) directly at a curved or cylindrical display. Front projection solutions can range from three to 15 channels and support not only FFS but also driving or naval simulators that often utilise cylindrical screens.

FFS generally deploy curved screens that wrap around the ‘cockpit’ to create the desired FOV. Larger direct-projection displays can support FOVs of 300+° horizontal by 120+° vertical. Visual requirements can vary widely for front projection solutions.

Front projection displays are the most widely used for military applications as they can support single and dual eye-point configurations, depending on the cockpit type and training requirements. They are also cost-effective and flexible (with respect to number of channels) and have lower operating expenses (OPEX) than collimated displays and RP domes. They can be either static or motion based.

Collimated cross-cockpit displays (CCCDs), meanwhile, depict out-the-window (OTW) imagery and are generally designed so scenery appears at a distant focus (infinite view). Collimated is derived from ‘co-linear’, which results in two pilots side by side seeing essentially the same OTW imagery without angular errors or distortions. CCCDs are used to support wide-body fixed-wing and helicopter training, can be deployed in static and motion-based configurations and are widely regarded as the optimal way to support complex day and night training requirements.

A critical element of CCCDs is the wrap-around collimating mirror used to reflect the image; these can be made from reflective film or solid glass. Both have advantages. Reflective film is lighter and cheaper, while glass mirrors provide wider FOVs and more accurate imagery with less distortion. For this reason, glass mirrors are becoming the standard for military CCCDs. However, production of these components is difficult.

‘Glass mirrors for CCCDs represent a unique challenge for industry. Quality and performance demands are very stringent for these capital- and labour-intensive mirrors. However, the visual performance is undeniably better. The key is delivering a strong mirror that will not break during deployment or operation with a superior coating that will not degrade over time,’ said Mark Saturno, president of Treality Simulation Visual Systems.

Treality has delivered glass mirror CCCDs for both static and motion-based applications. Its latest CDG-2460 systems support an international C-130 training programme. (Photo: Treality)

Treality is one of a few companies worldwide that has delivered glass mirror CCCDs for static and motion-based applications. Its latest CDG-2460 system, supporting an international C-130 training programme, completed rigorous motion and visual performance acceptance testing before being deployed to the end-user site.

Phil Perey, head of technology for defence and security at CAE, added: ‘We have observed a trend – more strongly in the US – to requesting large glass or glass-composite collimating displays for helicopter and certain large-FOV, multi-crew applications. Glass is certainly not new as CAE designed and delivered a system in the 1990s, but new manufacturing methods have lowered the significant weight hurdle, making glass an alternative to traditional Mylar-based displays.

‘Glass offers enhanced optical qualities, including higher brightness and lower distortion, and design flexibility for odd-shaped displays. It still, however, remains a premium-priced offering for extreme FOV configurations as the brightness benefits are largely offset by newer-generation solid-state projectors.’

RP domes are the high-end simulator of choice for fast jet training. They generally envelop a single-seat cockpit with a contiguous or faceted display to provide the pilot with 360° visibility. RP domes also present challenges in reconciling current requirements for near-20/20 visual acuity at high speeds in both day and night scenes. Another unique requirement is the integration of operational HMDs common in today’s fighter aircraft.

While demand for RP domes is increasing worldwide, they require significant floor space and come with a higher price tag. However, RP domes have a proven track record of significantly contributing to mission rehearsal capability as well as transitioning pilots from trainer aircraft to next-generation jets.

The 2010 timeframe saw fixed portable fabric screen systems with lamped projectors come on the market. These produce a near-360° flight experience. ZedaSoft installed such a tent-style dome display in 2011 for the Air Force Research Laboratory’s Fighter Risk Reduction Program – Advanced Collision Avoidance Technology effort. The system could be moved and installed in one day.

Treality’s latest RP-X domes represent a significant leap forward. Utilising a patented ellipsoidal (or egg-shaped) dome, the RP-X offers low channel count options (ten or 12 projectors) and a small footprint while delivering 360° visual performance with a contiguous dome and solid-state projectors. According to the company, this results in the lowest capital expenditure (CAPEX) and OPEX solution available to meet visual requirements for fast jet training, which is currently being delivered to multiple end-users worldwide.

Treality’s latest RP-X dome offers a 360° flight experience with a contiguous dome and efficient solid-state projectors. (Photo: Treality)

3D perception, meanwhile, offers a range of turnkey, preconfigured Northstar display systems optimised for specific domains. These are available with a range of projector models and adaptable to accommodate differing programme requirements, facility constraints, cabins and other factors.

In 2021, the company was awarded a contract to supply its Draco fast jet mini-dome display systems by the US Air Force Test Center under Air Force Materiel Command for the Air Force Joint Simulation Environment for the service’s fourth- and fifth-generation aircraft. The Draco 240+ model selected for this programme has a 240° horizontal by 150° vertical FOV.

Critical elements

Crucial and complementary to the physical visual display itself are projectors and IGs. Companies such as Barco, Sony and JVC are long-time players in the liquid-crystal-on-silicon (LCoS) projector market with thousands of channels supporting devices still in use worldwide.

LCoS projectors are known for being powerful and inclusive of important features, such as onboard blending and warping that are pivotal to incorporating multiple channels onto a curved screen. LCoS devices, with their complex light engines and xenon lamps, are also more expensive to procure and maintain than alternative solutions that utilise LED/solid-state technology with no moving parts or repetitive lamp replacements.

LED projectors tend to have lower CAPEX and OPEX than LCoS but may not have the same performance features and benefits. Current providers of solid-state projectors for simulation include Barco and Norxe, among others.

‘Projector technology is advancing rapidly.’

‘Projector technology is advancing rapidly and changing the dynamics of hardware and software integration [HSI] within a simulation visual display, so it is important for integrators and end-users to understand the price, performance and risk trade-offs when formulating flight simulator requirements,’ said Peter De Meerleer, VP of sales and strategic marketing at Treality.

IGs provide the visual scenes of the simulation environment and are another important element of display HSI. There are many on the market, with some developed by FFS providers such as CAE, Collins Aerospace and FlightSafety International, and others by independent developers such as Aechelon Technologies, Diamond Visionics, MAK and Quantum3D.

IGs are themselves impacted by the processing power of the computers and graphics cards onto which they are loaded, which are also continually evolving. COTS graphics cards, such as those produced by Nvidia, are an important element of the IG variable in the display equation.

Finally, display management tools play a fundamental role in managing outputs from projectors and IGs to create and maintain imagery. Treality’s AutoAlign Suite is one example of an agnostic tool set that can support multiple projector types and IGs. Other FFS integrators have comparable tool sets to support their specific visual displays.

The critical link between projectors, IGs and display management tools manifests with their integration to meet visual performance requirements. If projectors do not have onboard blending and warping, this must be accomplished with display management tools. If the IG cannot produce the output required to meet resolution requirements, this can limit projector effectiveness.

‘The balancing of FFS visual display requirements with available projector hardware, IG software, COTS processing power and display management tools is a critical planning effort for which military training organisations and their industry partners must maintain constant and productive communications,’ added Saturno. ‘Working together on the front-end planning of FFS deployment and sustainment results in lower risks, reduced costs, greater efficiency in training pilots and better overall ROI [return on investment] for training dollars.’

Adding to the discussion of critical elements of visual displays, Perey said: ‘Pixel shift technology is perhaps the most overlooked advancement in display projector technologies. It is essentially the modern-day equivalent of interlaced rendering on old analogue TV sets.

‘Why is it important? It substantially improves “static” scene resolution for a given number of rendered pixels, reducing the demand on IG load, and facilitates 120Hz rendering, thereby improving “dynamic” scene resolution. This translates to the pilot seeing a higher-resolution display that is crisp in virtually all flight conditions.’

Beyond projection

While collimated displays and domes with arrays of projectors produce extremely realistic results, they are expensive to acquire and operate, difficult to maintain and require trainees to travel to a fixed location as they cannot be moved.

The prospect of using easily portable HMDs instead has long been envisioned but has not come to fruition due to deficiencies in the technology. Previously, HMDs have lacked the resolution, FOV and low latency needed to produce adequate image quality, causing cue mismatch that often leads to simulation or cyber sickness.

Recent developments in HMD technology now show a major shift is under way.

Recent developments in HMD technology now show a major shift is under way. VR fidelity and incorporation into training and simulation applications has increased due to the commercialisation of products such as OCULUS/Meta Quest.

Military training organisations are exploring opportunities to apply XR solutions given their cost-effectiveness, ease of use and physical flexibility. However, reviews and adoption of such devices are mixed given the wide range of applications desired, lack of standardisation and availability of mature training-ready devices. However, advancements are being made, and it is expected that XR hardware will continue to find its way into military environments.

Adoption of HMDs is being accelerated by the shift from mission training centres to training at the ‘point of need’. This concept was initially raised in the US Army’s Synthetic Training Environment Request for Solutions document in 2019. ‘Training at the Point of Need’ is defined as an operational location with immediate access (5G wireless) to training assets and courseware where the warfighter can be trained on a new system while being deployed in theatre. The requirement has now been adopted by all services and is accelerating the evolution of display systems.

With respect to military flight simulation, XR devices are becoming common in testing and evaluation environments to determine feasibility for wider adoption. Some organisations are currently using these devices for part-task training, particularly for new pilots. Many of today’s younger aviators are familiar with the technologies, so they are more comfortable with using them for flight training. For example, the USN is completely reworking its curriculum to leverage VR and MR devices to support basic pilot training.

Each display system has its drawbacks, which must be balanced with the positives to produce the best outcome for a specific training type. Take, for example, the issue of resolution. For a JTAC running a Type 1 control training scenario, available simulation systems are still inferior to the human eye. The latter can detect and recognise an A-10 aircraft at greater distances than can be currently replicated by technology.

‘A dome display system may try to increase resolution by increasing the number of projectors being used to depict the scenario, thus increasing power demand and system footprint; however, the “realism” of the scene will still not be as great as that offered by an HMD, which offers lower resolution but much higher image contrast. Better contrast can compensate for less resolution. So, there is no one-size-fits-all solution,’ said Rybacki.

The maturation and mass production of high-performance HMDs has enabled industry to offer a more compact option. ‘The combination of increased resolution and refresh rate means that these devices are real contributors to the spectrum of training devices,’ Perey noted.

You will need internet connection to view this video

‘One example is CAE’s MAVRC rear-crew training solution which enables whole crew training at positions such as hoist, gunner and confined area landings. Improvements in head tracking systems and miniature displays have made this simulation display technology viable for not only pilot/operator training but also for mission, maintenance, battlespace and even data science applications.’

XR devices are not expected to replace or deliver the more advanced training and mission rehearsal capabilities possible with FFS. ‘One of if not the pivotal element of an FFS is the fidelity and operation of the cockpit for the aircraft being simulated,’ said Saturno. ‘If pilots are not conducting live training, then they want the simulator to be as realistic as possible with respect to the tactile performance of the cockpit and acuity of the visual display. At this time, and for the foreseeable future, FFS are expected to be the preferred method of training for advanced pilot training and mission rehearsal,’ Saturno concluded.