Enhancing Displays by Blurring

Published in: SID Annual Meeting Proceedings, 1994


ABSTRACT
Some Enhanced Vision cockpit displays consist of synthetic imagery superimposed on a real image. The high spatial frequency components of the synthetic imagery can mislead an operator by masking features of the real image. We demonstrate that blurring the synthetic image prior to superposition reduces its masking effect in high- contrast regions of the real image, while maintaining its enhancing properties in regions of the real image where visibility is low.


INTRODUCTION
In Enhanced Vision cockpit displays, auxiliary visual information is derived from the output of various sources, such as an infrared camera, a radar or an on- board database. A typical design problem is to display this auxiliary information fused with the out- the-window image of the real scene. One way this can be accomplished is by a direct optical or digital superposition (fusion) of the sensor image and the real ("out-the-window") scene (Foyle et al., 1992 [1]). A problem with this kind of superposition is that, in low visibility conditions, the sensor image may mask critical features of the out-the-window image. This is especially undesirable when the information from the sensor is inaccurate because of low sensor signal-to- noise ratio, time delays or geometric distortions. One way to minimize this problem would be to reduce the contrast of the sensor image in selected high-contrast regions of the real scene. A robust local gain control mechanism to implement this scheme would require sophisticated algorithms to estimate the out-the-window visibility and correct the sensor contrast accordingly, and assumes the availability of the out-the-window scene in digital form. An alternative approach is to try to format or process the sensor image to minimize the masking effects. We investigated the effect of blurring the sensor image on the ability of a human operator to locate edges. The choice of this scheme was motivated by the well- known fact that edges with high spatial frequency content are very effective maskers of low spatial frequency image information (Harmon and Julesz, 1973 [2]).

Figure 1 - Top: out-the-window scene at d=500 ft. before touchdown in a simulated landing sequence under clear weather conditions. Bottom: same out-the-window scene under foggy conditions.


METHODS
We applied this idea to simulations of the scenario of an airplane on a final approach using a display of the out-the-window scene enhanced by sensor information. We generated two simulated landing sequences. These sequences simulate the final approach to landing of an aircraft, from about 1000 ft. before the touchdown point to about 300 ft., after touchdown, under clear-weather and dense fog (700 ft. visibility) conditions respectively. An example frame from each sequence (at 500 ft. before touchdown) is shown in Figure 1. To simulate a forward looking, weather penetrating sensor, we extracted the edges from the clear-weather scene (without runway markings). This can alternatively be regarded as simulating the output of an on-board database rendered by wire frame. We used the implementation of an edge extraction algorithm (Shaw, 1979 [3]) provided by the HIPS image processing package (Landy et al., 1984 [4]). The edges extracted from the frame at d=500 ft. are shown in the top panel of Figure 2. We processed these edges by low-pass filtering them with an ideal filter in the spatial frequency domain. We left unaffected all components whose spatial frequencies fell within a radius ² 0.5 of the 1-D Nyquist frequency in the Fourier Transform, and zeroed all components whose spatial frequencies fell above this value. After filtering, we rescaled the image so that the peak-to-peak amplitude of the low- pass image was the same as that of the original edge image. The result of this operation is shown in the bottom panel of Figure 2. We then proceeded to superimpose these low-pass filtered edges onto the out-the-window images (both clear weather and fog conditions) by simple addition. Next, we compared the effect of superimposing unprocessed and low-pass filtered misregistered edges onto the clear weather scene.


RESULTS
In the first demonstration, we produced an enhanced vision landing sequence in which the low-pass filtered sensor edges are added frame-by-frame onto the out-the-window, relatively low contrast, fog sequence. As the sample frame in Figure 3 shows, the low-pass filtered edges are evident and effective outlines of features otherwise invisible through the fog. Conversely, when these low-pass filtered edges are added onto the clear weather scene (Figure 4) they tend to provide a mere faint glow around the high spatial frequency feature edges. We take this as evidence of the promise of this approach to enhance low-contrast visual scenes without interfering with visual scenes in which the contrast is such that no enhancement is required.


Figure 2 - Top: Edges of runway and taxiways (extracted from Figure 1, top) as might be obtained from a forward looking on-board sensor. Bottom: Low-pass filtered edges.


Figure 3 - Low-pass filtered edges added to the foggy out-the-window scene are effective in enhancing the available low-contrast information in that scene.
Now consider another application of these techniques to situations where the misaligned edges may be misaligned because of misregistration. Our second demonstration compares the fusion of a misregistered sensor before and after low-pass filtering as described above onto the clear-weather out-the-window scene. When the sensor image is presented as a wire frame image with high-spatial frequency content, the sensor edges tend to divert the viewer attention away from the edges of the out-the-window image (Figure 5, top panel). Under these conditions, the sensor edges may be perceived as the 'true' edges and can lead the pilot to see the runway in a displaced position. As in the previous demonstration, the result of the filtering operation makes the edges less effective maskers of the out-the-window scene. When they are superimposed onto the clear-weather scene they are effectively masked by the high-contrast, sharp edges in that scene (Figure 5, bottom panel), and ideally they will not contribute to the pilot's perception of the runway location. We have observed this improvement in a very effective demonstration that we prepared in which the unprocessed and low-pass filtered lines are added to the out-the-window scene with increasing and then decreasing amounts of translation from their 'true' position, and the images thus generated are animated in a motion sequence. Once in motion, the unprocessed edges capture he observers' gaze very effectively and distract attention from the runway position, whereas this effects is greatly reduced when the low-pass filtered edges are animated in the same fashion.


DISCUSSION
We demonstrated that blurring a synthetic or sensor- based wire frame image can reduce undesirable masking effects. This approach can be used in situations in which various sources of information, differing in signal-to- noise ratio, need to be fused. In those situations, sensors with more uncertain outputs can be blurred prior to fusion. We are following up these demonstrations with experiments on the perceived position of combinations of different types of lines (Tiana et al. 1994 [5]). We hope to develop quantitative models for this task similar to the cue fusion model of Landy (Landy, 1993 [6]). Low-pass filtering the sensor information is not the only approach that can be conceived for the purpose of reducing masking effects. Other possible alternatives might include: periodic interruptions in the lines (dashed lines); a 'marching ants' display taking advantage of the perceived motion of the dashes in dashed lines when the dashes and the blank segments between them are alternately swapped; or a display in which pixel noise is added to the sensor lines with a standard deviation proportional to the amount of reduction in visual effectiveness required.


Figure 4 - Low-pass filtered edges added to the clear- weather visual scene have little effect upon the high-contrast information available from that scene.


Figure 5 - Top: high contrast edges from a misregistered sensor, added to the out-the- window scene distract viewer from the actual feature edges. Bottom: low-pass filtered edges do not mask the high-contrast information in that scene, making misregistration a less critical failure.
Finally, if a range map of the sensor scene were available, as is the case for example in the case of RADAR, similar processing o the data might be feasible based on out-the-window scene characteristics. In low visibility conditions, RADAR- extracted, distant features might be enhanced relative to close-up features that might be less affected by poor visibility. Each of these schemes could have to be evaluated in a task dependent fashion, since its effectiveness will probably well be highly task and situation dependent. For example the 'marching ants' display might be usefully applied to prevent masking of lines in motion, while adding pixel noise might be useful in situation where high-spatial frequency noise is already a component of the scene. The approach we outlined can also be used to represent the reliability of a source of information without significantly impairing its visibility; information from sensors prone to misregistration, drift or intrinsic noise might be presented in a way that weights its saliency in proportion of its reliability.


ACKNOWLEDGMENTS
This work was supported by NASA RTOP 506-59-65, NASA RTOP 505-64-53 and NASA grant NCC2-811 to OGI.


REFERENCES
  1. Foyle, D. C., Ahumada, A. J., Larimer, J., Sweet, B. T. (1992)
    Enhanced/ Synthetic Vision Systems: Human Factors Research and Implications for Future Systems.
    SAE Transactions: Journal of Aerospace, v. 101, pp. 1734-1741.
  2. Harmon, L. D., Julesz, B. (1973)
    Masking in Visual Recognition: Effects of 2-Dimensional Filtered Noise.
    Science, v. 180, pp. 1194-1197.
  3. G. B. Shaw (1979)
    Computer Graphics and Image Processing, v. 9, pp. 135-149.
  4. Landy, M. S., Cohen, Y and Sperling, G. (1984)
    In: HIPS: A UNIX -based Image Processing System.
    Computer Vision, Graphics and Image Processing, v. 25, pp. 331-347.
  5. Tiana, C. L. M., Lanham, J. and Pavel, M. (1994)
    Integration of edge location Information across Frequency.
    Investigative Ophthalmology and Visual Science (suppl), vol. 35, pp. 2064.
  6. Landy, M. S. (1993)
    Combining multiple cues for texture edge localization.
    SPIE Proceedings, vol. 1913, pp. 506-517.
NASA Ames Research Center, Vision Lab / carlo@vision.arc.nasa.gov