DECEMBER 2001 24 In parallel with this trend, new auto-stereoscopic techniques are now being developed that hopefully will allow true 3D stereoscopic views to be generated and viewed without the need for external aids such as spectacles and without undue strain on the part of the user. In this particular context, considerable funding for these new developments in auto-stereoscopy has been forthcoming from the entertainment industry, since many content providers perceive a large market developing for 3D stereoscopic products in the cinema, TV and video games sectors of their industry. There is also a considerable interest from the medical profession in the use of 3D stereoimages in certain types of surgical procedures and in the 3D stereo-visualization of computer tomography and magnetic resonance scans. To this can be added the further strong interest in and the support for the development of new 3D stereoscopic technology from defence agencies. This is related particularly to their training and simulation requirements as well as those for image interpretation for intelligence and mapping purposes. If all of this current activity and development of 3D stereoscopic technology is successful, then of course it will be very beneficial to the geoinformatics industry as a whole as well as these other fields of activity. 3D Stereo-Viewing for Photogrammetry & Image Interpretation Within photogrammetry, the traditional method of viewing hard-copy stereo-pairs in analogue and analytical instruments mostly involved the use of high precision photo stages and elaborate and expensive trains of high-quality optical components to allow the binocular 3D stereo-viewing of any part of the stereo-model. Often these stereo-plotting instruments featured zoom lenses or turreted optical systems with alternative lenses to allow the magnification of the stereo-model needed for high precision measurements. Some less accurate and less versatile stereo-plotting instruments based on optical projection (such as Multiplex, Balplex and Kelsh Plotters) used simpler systems such as anaglyphic (red/blue) filters or mechanically operated alternating shutters to implement the viewing of the resulting stereo-model. On the image interpretation side, generations of geologists and other field scientists and military photo-interpreters utilized simple lens or mirror stereoscopes in conjunction with stereo-pairs of hard-copy aerial photographs to carry out their interpretative work. These traditional techniques are still in widespread use. 3D Stereo-Viewing of Digital I Is Auto-Stereoscopy the Future for 3D? With the increasing availability of digital imagery for use in mapping and GIS operations, the whole matter of the 3D stereoscopic viewing of aerial and space imagery has been undergoing a radical change. The reasons are varied. On the one hand, we have seen the widespread adoption of the digital photogrammetric workstation (DPW). In association with this development, the computational power that now exists within the DPW can handle comfortably the large amounts of digital image data that are required to produce high-quality 3D stereoscopic images of the terrain interactively in real time on its monitor screen. So the implementation of the new 3D display technology is now much easier than before. Besides which, there is now an increasing realization that the 2D visualizations of 3D terrain data that are currently being labeled as "3D views" are insufficient and indeed deficient for many purposes - especially for measurement and for image interpretation and image understanding. This has also encouraged the use of truly 3D stereoscopic imagery. By Prof. Gordon Petrie Figure 1: Twin monitors viewed with a mirror stereoscope (Drawn by Mike Shand) 3D Stereo-Viewing of Digital I DECEMBER 2001Latest News? Visit www.geoinformatics.com 25 3D Stereo-Viewing in Remote Sensing & GIS Turning next to remote sensing, although digital image processing systems equipped with display monitors have been used with space images ever since the first Landsat satellite came into operation nearly thirty years ago, the data has almost always been monoscopic. Indeed many users of satellite remote sensing imagery have never viewed a 3D stereoscopic image on the display screens of their systems. Nor have they used 3D stereoscopic data in their analyses. As for the GIS community, as noted above, what is often described by them - even in GeoInformatics magazine! as a 3D image or visualization is, in fact, a 2D perspective view or projection of the terrain. This usually takes the form of a flat 2D image appearing either on a display screen or in hard-copy form, even if 3D data such as a DEM was used in its construction or formation. Introduction of Digital Imagery However, with the introduction of digital photogrammetric workstations (DPWs) in the early 1990s and their increasing acceptance and use in the late 1990s, the whole situation began to change quite rapidly. In particular, the development of the DPW eliminated the need for the expensive highprecision mechanical and optical components of the previous analogue and analytical instruments. Stereo-pairs in digital form generated from the scanning of hard-copy aerial photographs soon became commonly available and into widespread use. So the 3D stereo-viewing technology had to be changed in accordance with this development. With the advent of the new airborne digital imagers (e.g. HRSC-A, ADS40) and spaceborne digital imagers (e.g. those mounted on the IKONOS, EROS-A1, OrbView-4, QuickBird-2 and SPOT-5 satellites) that can generate overlapping pushbroom scanner images in the along-track direction, the situation can be expected to change still further. Certainly more 3D stereo-image data will become available in the future. Thus it does seem appropriate to review the current situation regarding the 3D stereo-viewing of digital imagery acquired from both airborne and spaceborne platforms and to speculate on its future development. Why 3D Stereo-Viewing? Based on the extensive experience of the mapping community over many years, there is no doubt that the use of 3D stereo-viewing improves the interpretation and measurement of imagery of almost all types of terrain. Thus the provision of a 3D stereoviewing facility is regarded by most photogrammetrists as being a complete necessity for their operations. In the first place, this facility is required for the accurate identification and measurement of the ground control points needed for the absolute orientation or geo-referencing of the stereo-model. After which, it is then needed for the subsequent measurement of the terrain detail (nowadays called feature extraction!) that is required for topographic map compilation and the acquisition of the 3D digital data needed for use in a GIS/LIS environment. A 3D stereo-viewing facility is also a vital element in carrying out map revision and the editing of the digital elevation data (DEMs) produced by automatic image matching techniques. Last, but not least, the availability of such a facility also permits the stereo-superimposition of 3D vector data over the stereo-model for quality checks of the accuracy and completeness of the 3D data. Thus virtually all DPWs feature a stereo-viewing and measuring capability - except for one or two systems produced by companies that specialize primarily in the remote sensing field and appear to be less aware of the attributes of 3D stereo-viewing. Current Methods Six methods of 3D viewing are currently in use within the general area of photogrammetry and geoinformatics. However many others are possible - the number of patents for 3D stereo-viewing systems run into the many thousands. Whatever the method actually used, the basic requirement for 3D stereo-viewing is that the observer's left eye must see the left image only of the two overlapping images and the right eye, the corresponding part of the right image only. • Binocular Viewing (1) The first possibility is to utilize two flatscreen display monitors displaying the left and right images of the stereo-pair respectively. These can be viewed using a mirror stereoscope or a more complex optical train (Fig. 1). This arrangement has been magery Figure 2: Single monitor display with split screen viewing (Mike Shand) Figure 3: Twin monitors viewed with polarizing spectacles (Mike Shand) magery DECEMBER 2001 26 used in the early I2S DPW, which has now disappeared from the market. The matter of ensuring the correct positioning and alignment of the twin monitors and the exact correspondence of the left and right images displayed separately on the two monitors appears to have discouraged the use of this approach. (2) An alternative and more popular approach is to display the left and right images side-by-side on a single display monitor and to view these through a simple lens or mirror stereoscope (Fig. 2) - the so-called split-screen stereo method. This was the method used in the pioneering Kern DSP1 DPW from the late 1980s. It is currently in quite extensive use on many of the DPWs produced by the Canadian company, DVP Geomatic Systems. Other DPWs employing this arrangement include LH Systems' 600 series DPWs running the SOCET SET software suite, GeoSystems' Delta Workstation and some examples of the KLT Associates Atlas DPW. The increasing availability of the wide-screen television sets that have been developed recently for use with HDTV would appear to be very suitable for the display of side-by-side images for 3D stereo-viewing. • Use of Complementary Filters (3) Another possibility is to use two flatscreen display monitors set at right angles to each other, the one with its axis pointing horizontally, the other with its axis pointing vertically upwards. One has a horizontal polarization sheet placed in front of its screen; the other has a vertical polarization sheet placed over it. A large semireflecting mirror set at a 45° angle between the two monitors acts as a simple beam splitter. This allows the two component images of the stereo-pair to be superimposed on each other, while the observer wears appropriate spectacles fitted with the corresponding horizontally and vertically polarizing filters to allow stereo-viewing (Fig. 3). This arrangement was used originally by Matra in its Traster analytical plotters employing hard-copy photographs. The same basic arrangement has been used in the Topcon PI-1000 DPW and in Galileo/Siscam's Stereodigit and Microdigit DPWs. In each case, quite small monitors have been used for the image display which rather limits their resolution; on the other hand, the method does not involve dynamic alternating imaging which carries the possibility of flickering. The Microdigit DPW utilises twin liquid crystal (LC) displays; however the other two DPWs in this group both utilize conventional CRT-based monitors. But it must be said that these DPWs have only had a very limited penetration of the market. (4) A low-cost solution that provides a 3D stereo-viewing capability is simply to superimpose the two component images of the stereo-pair on the screen of a single colour monitor with the one image displayed in red and the other in blue or green using the well-known anaglyphic technique. As noted above, this method is familiar to most photogrammetrists from its use on analogue stereo-plotting instruments such as the Multiplex, Balplex, Kelsh Plotter, Zeiss DP-1, etc. based on optical projection. With DPWs, users view the resulting 3D stereo-model appearing on the screen of the display monitor wearing spectacles equipped with the corresponding red/blue or red/green filters (Fig. 4). This method has been used quite a lot with the R-WEL DMS system. Previously the method has been limited to the use of monochrome (black and white) stereoimagery. However recent developments have shown that some limited colour rendition of the stereo-model from colour aerial photographs is possible using the anaglyphic technique. • Alternating Images (5) Probably the most commonly used 3D stereo-viewing system on DPWs is to display the left and right component images alternately at high speed, e.g. 50 to 60 Hz per image, on a single monitor screen. This makes use of the capabilities of the human visual system to store and merge the two component images of a stereopair across a time gap of 50ms. The actual 3D stereo-viewing is then carried out using so-called "active" spectacles equipped with alternating shutters. These are synchronised with the alternate left and right images being displayed on the monitor screen so that the left eye sees the left image only and the right eye, the right image only (Fig. 5). Again, as noted above, the basic idea is familiar to those photogrammetrists who have used systems such as the Stereo-Image Alternator (SIA) on older optical projection instruments such as the Balplex and Kelsh Plotter. However these systems used mechanically operated shutters. The modern versions that are in use with DPWs use pairs of liquid crystal (LC) alternating shutters mounted in spectacle frames. The open/shut operations of these shutters are synchronized electronically with the display image. Thus the spectacles need power and some control circuitry, together with a timing signal from the display monitor that triggers the operation of the appropriate shutter. This signal is provided either by direct wiring to the display controller or using a wireless technique employing an infra-red (IR) emitter mounted on top of the display monitor. Most of the suppliers of DPWs that utilize this type of alternating shutter technology have adopted the CrystalEyes system from the StereoGraphics Corporation based on the use of LC shutters and an IR emitter. However the rival MacNaughton company which is best known for its NuVision system (see below) - also offers this type of system via its 60GX wireless spectacles, again used in conjunction with an IR emitter. Many much cheaper alternatives can now be sourced from various suppliers in the Far East and the U.S.A. that are serving the huge 3D video games market. Formerly the method was mainly in use on those DPWs based on up-market SGI graphics work stations which were "stereo-ready", i.e. they did not need additional hardware to implement the method. However this no longer applies since PCs are readily adapted for the purpose using "stereo-ready" cards that are produced in large numbers for use with 3D video games. Thus the technology is now in widespread use by almost all DPW suppliers. Figure 4: 3D stereo-viewing with anaglyphic spectacles (Mike Shand) DECEMBER 2001Latest News? Visit www.geoinformatics.com 27 (6) The final method that is in current use on DPWs also uses alternating (left/right) overlapping images displayed on a single monitor screen. However, in this case, each successive image displayed on the screen has a different (clockwise/anti-clockwise) polarisation pattern induced by a LC-based modulator - sometimes described as an electronic "prism" or "panel" - that is bezel-mounted on the front of the display monitor (Fig. 6). Users viewing the 3D stereo-model wear spectacles equipped with the corresponding polarising filters. Again these ensure that the left eye only sees the left image and the right eye, the right image only. Moreover these simple and inexpensive spectacles do not need power to activate them. Thus they are usually described as being "passive" spectacles to distinguish them from the so-called "active" spectacles equipped with alternating shutters described in the previous paragraphs in (5) above. The principal supplier of this type of viewing system has been the MacNaughton company with its NuVision system that was developed originally by the Tektronix company. This is available both in the form of an integrated monitor or as kits that can be fitted over the screen of an existing display monitor. Recently the rival StereoGraphics company has also adopted this technology with the introduction of its so-called ZScreen 2000 monitor series. Multiple Viewing It will be noted that those 3D stereo-viewing systems that use spectacles equipped with filters or alternating shutters - as described in (3) to (6) above - all allow a number of users to view the stereo-images simultaneously. Whereas those systems employing binocular mirror or lens stereoscopes - types (1) and (2) above - restrict the viewing to a single observer. However doubtless, if required, this restriction could be overcome to some extent through the use of dual binocular systems, as was done in the past with a number of analogue or analytical stereo-plotting instru- ments. Hard Copy Prints & Transparencies For many years, the principal method of producing 3D hard-copy prints and transparencies for publication or widespread use has been to use the anaglyphic method. The two component images of the stereopair are printed in red and blue respectively, superimposed on a common base sheet. Usually this printing is done using a photographic or lithographic process. 3D viewing of the printed stereo-pair is then achieved using spectacles equipped with complementary red/blue filters. Quite recently, the StereoJet process has been introduced which allows 3D stereo-viewing of fullcolour hard-copy images produced from overlapping digital imagery. With this process, the left and right images of the stereo-pair are registered and printed on the opposite sides of a transparent base. This is done using an inkjet printer applying proprietary polarizing inks on to a proprietary substrate. The two printed images have polarizing patterns that are oriented at right angles to one another. As with the corresponding screen displays described above, the observer obtains the 3D stereoscopic image by viewing the prints or transparencies with spectacles equipped with the appropriate polarizing filters. Examples of this new process were shown at the recent ASPRS 2001 Annual Conference held in St. Louis. Several service bureaux have also been set up in the U.S.A. (e.g. San Francisco Imaging Services) and Canada (e.g. Aero Geometrics) to supply this product to the mapping community in North America. Auto-Stereoscopic Displays At the present time, an enormous research effort is being undertaken and huge sums of money are being spent on the development of auto-stereoscopic displays using digital image data. These devices display (or attempt to display) the 3D spatial image of an object to an observer without the use of spectacles, filters or other external viewing aids. Thus commentators now talk and write about having the possibility of having "free viewing" systems in the future instead of the "aided viewing" systems employing spectacles and filters that are currently being used on DPWs and in other applications utilizing 3D stereoscopic viewing. Much of this research effort is directed specifically towards medical, CAD, industrial inspection, flight simulator and multi-media applications. However the results will be of considerable importance for the geoinformatics industry. At the present time, there are a wide variety of alternative techniques being investigated or developed to form the basis of auto-stereoscopic displays. These are being carried out both by a number of small enterprising start-up companies and by some large and well-established companies such as Philips, Zeiss, Sharp, Sony, Sanyo, etc., that are well known in the fields of optics and consumer electronics. Besides which, a number of research groups in various universities in the U.S.A., Japan and Europe (especially in Germany and the U.K.) are also involved in this development. Even attempting to sumFigure 5: Alternating images on the monitor screen with alternating shutters for 3D stereo-viewing (Mike Shand) Figure 6: Alternating images on the monitor screen with an electronic prism set in front of the screen: 3D viewing using polarizing spectacles (Mike Shand) DECEMBER 2001 28 marize briefly all the various different approaches would fill a whole issue of GeoInformatics magazine: it really is a wild and somewhat confusing scene! Besides which, many of the concepts are quite complex and far from easy to understand. Thus only a couple of simple examples based on the use of gratings and/or lenticular sheets will be outlined here to illustrate the basic idea of auto-stereoscopic displays. Many more complex approaches are being worked on at the moment. • Parallax Stereograms The starting point may be taken as the socalled parallax-barrier method, well known for nearly a century and forming the basis of the rather crude 3D viewing cards that come free in packets of breakfast cereals. A much more refined version of this basic idea is being used with digital image data. This utilizes a very finely ruled grating that is placed in front of a specially produced image. The spacings in the grating form a series of parallel transparent slits or gaps. Each slit in the grating acts as a window to a specific vertical column or strip of the image placed behind it. This image comprises two sets of interlaced and alternating vertical columns derived from the two images of the stereo-pair. This striped interlaced image and the vertical grating with its slits are arranged so that the observer's left eye can only see the oddnumbered columns or strips derived from the left image; while the right eye can only see the even-numbered columns or strips from the right image. The opaque lines of the grating block the unwanted images from being viewed. Thus both images of the stereo-pair are displayed simultaneously and reach the appropriate eyes of the observer. The result is that the observer sees a 3D stereoscopic image. • Lenticular Sheets An extension to this basic concept involves replacing the grating with its set of parallel slits by a lenticular sheet. This comprises an array of long parallel cylindrical lenses or lenslets that are formed from a transparent plastic material using a special mould. Each cylindrical lens in the array takes the place of an individual slit. The result is a much improved set of images with a considerable reduction in the loss of light caused by the opaque lines of the grating used in the previous method. • DTI 3D Auto-Stereoscopic Displays These concepts and technologies have been used as the basis for a series of 15 and 18 inch flat-panel colour LCD displays produced by a small American company, Dimension Technologies Inc. (DTI) based in Rochester, New York. Instead of the grating being placed in front of the image, it takes the form of an array or lattice of very thin but very bright vertical lines of light placed behind the transparent liquid crystal display (LCD) panel on which the strip images are being displayed (Fig. 8). This illumination lattice is generated by a lenticular sheet which focuses the light from fluorescent lamps and forms the parallel series of thin light lines on a translucent diffuser. These illuminate the alternating image columns so that each eye receives only the appropriate strip images. In this way, a 3D stereoscopic impression is formed - though it can only be viewed by a single person from a fixed position in front of the display panel. Thus they are single user displays. These DTI 3D auto-stereoscopic displays have been supplied both to NASA and to various U.S. defence agencies concerned with intelligence gathering and mapping. The displays are in use with Recon Optical, the American manufacturer of airborne digital reconnaissance cameras, and have also been evaluated recently by Z/I Imaging. • Philips 3D Auto-Stereoscopic Displays In the case of the Philips 3D-LCD autostereoscopic displays, the sheet of finepitched cylindrical lenticular lenses is placed in front of a high-resolution LC colour display panel in such a position that the LCD image plane lies in the focal plane of the lenses (Fig. 8). If the stereo-pair of images of the terrain are divided up into alternating vertical strips or columns from the left and right images respectively, then, once again, all the columns of the left image are seen by the left eye only, while the columns of the right image are only seen by the right eye. Combining the two sets of image data, the observer obtains a full 3D stereoscopic impression of the terrain. Recent further developments have seen the technology enhanced to allow several persons to view the 3D stereoscopic image at the same time. Philips appear to be concentrating their efforts on selling this technology and its accompanying software on an OEM basis to companies that are already established in specific application fields. This could well suit the system suppliers to the geoinformatics industry. DEMs & Orthoimages With the advent of overlapping airborne and spaceborne digital imagery has come the highly automated production of DEMs and orthoimages. The DEMs are produced using automatic image matching (correlation) techniques. This operation is then followed by the ortho-rectification of one of the overlapping images using the 3D coordinate data produced by the DEM to generate the orthoimage. This capability is provided on almost all modern DPWs. For some time now, this procedure has been implemented both for stereo-pairs of aerial photographs and with the overlapping cross-track scanner imagery acquired by the SPOT and IRS-1C/D satellites. With the current introduction of overlapping imagery acquired along-track by pushbroom scanners operating both from airborne platforms (e.g. the HRSC-A and ADS40 imagers) and spaceborne platforms (e.g. the IKONOS and EROS-A1 imagers), this type of automated operation can be expected to become still more widespread. Indeed already both Space Imaging with its IKONOS space image data and ISTAR with its HRSC-A airborne image data offer DEMs and orthoimages as standard products from their overlapping pushbroom scanner imagery. Stereomates Of course, the generation and supply of fully rectified individual orthoimages or orthoimage mosaics means that users canFigure 7: (a) The basis of the DTI 3D auto-stereoscopic display. (b) DTI's 15 inch and (c) 18 inch display monitors (Source: Dimension Technologies Inc.) DECEMBER 2001Latest News? Visit www.geoinformatics.com 29 not then enjoy the benefits of 3D stereoviewing. In the past, this led to the development in Canada, France and Austria of stereomates. These used the DEM elevation data to re-introduce parallaxes in a controlled manner into the rectified imagery, so creating a second image (the stereomate) that could be utilised together with the orthoimage to create and view a 3D stereo-image of the terrain. However only one or two DPWs offer this possibili- ty. Chromo-Stereoscopy Over the last few years, an alternative approach has been to use the chromostereoscopic technique originally developed by Chromatek Technologies in the U.S.A. This method has been developed still further by Dr. Toutin of the Canada Centre for Remote Sensing (CCRS) for use with remotely sensed image data. The basis of the method is completely different to all of those discussed above. The different elevation values given by the DEM are encoded into the orthoimage as varying colour values using the wide range of colours available across the whole of the visible spectrum. Thus it uses only a single image (the orthoimage) instead of utilizing two images as required by all the other stereo-viewing methods discussed above. This colour encoding of the orthoimage in accordance with the corresponding elevation value for each pixel position is carried out in a precise and highly structured manner through a preliminary radiometric processing step prior to stereo-viewing. Once this step has been completed, the resulting colour-encoded image is viewed by the observer when it is then decoded using quite simple and inexpensive prismatic spectacles to give a strong 3D impression of the terrain covered by the ortho-image. Thus the chromo-stereoscopic approach is very different to that of the conventional 3D stereoscopic techniques - which are based on the use of two images containing disparities or parallaxes (resulting from relief displacements) related to the terrain elevation to produce the required depth. By contrast, the chromo-stereoscopic orthoimage retains its map geometry but still gives a 3D stereoscopic impression of the terrain to an observer wearing the appropriate ChromoDepth spectacles. The chromostereoscopic images can either be printed in hard-copy form or displayed on colour monitor screens. Readers of GeoInformatics have already had the opportunity to view the various examples of chromo-stereoscopic images that were included in the September 2001 issue of the magazine. • ChromoDepth 3D Stereo-Viewing The basis of the actual stereoscopic viewing is to use the so-called ChromoDepth spectacles. These are equipped with pairs of very thin double prisms made from plastic film. One of the two prisms in each double prism is made of a highly dispersive material with a refractive index that is strongly wavelength (i.e. colour) dependent (Fig. 9b). The second prism in the double prism has a low refractive index. When combined together, the resulting double prism acts like a thick glass prism (Fig. 9a): however they are actually produced in the form of a thin plastic film. When the pair of double prisms are mounted in the appropriate orientation in the frame of the viewing spectacles, the colour encoded image will then appear to the observer in stereoscopic 3D giving depth to the scene (Fig. 9c). Thus the terrain objects with the highest elevation (encoded in red) will appear to be nearer (i.e. higher) to the observer than those at the lowest elevations (encoded in blue) that appear to lie further away (i.e. at a lower elevation). Terrain objects at intermediate elevation values (encoded in orange, yellow, green, etc.) will appear to lie at the appropriate depth (or elevation) between the highest and lowest elevation values. Conclusion With the ever-increasing availability of overlapping stereo-imagery acquired in digital form from airborne and spaceborne platforms, the benefits of 3D stereo-viewing can be experienced by ever more people working in the geoinformatics field. Thus the current developments in 3D stereo-viewing technology - such as the auto-stereoscopic, StereoJet and chromostereoscopic techniques - are of considerable interest. They promise to make it easier for users to utilize 3D stereo-viewing of the terrain with all the advantages that go with it. Professor G. Petrie (g.petrie@geog.gla.ac.uk), Department of Geography & Topographic Science, University of Glasgow, Glasgow, G12 8QQ, Scotland, U.K. Web Pages - http://www.geog.gla.ac.uk/~gpetrie Figure 9: The basis of chro- mo-stereoscopic viewing. (a) The refraction of white light through a glass prism. (b) The superchromatic double prism. (c) ChromaDepth viewing spectacles using a pair of double prisms (Drawn by Mike Shand) Figure 8: The basis of the Philips 3D auto-stereoscopic displays using a lenticular sheet (Source: Philips Research)