CHAPTER 1
INTRODUCTION
In certain application fields where image processing is involved, the use of spectral information in the visible (VIS) and near infrared (NIR) range is critical. The systems for spectral measuring can be classified according to their spectral resolution or their spatial resolution. Narrowband hyper spectral systems are usually based on point spectrometers or linear spectrometers. These systems can measure many precise narrow spectral bands of a point light source or a line source, in the VIS and NIR ranges and beyond. On the other hand, Red, Green and Blue (RGB) cameras provide a very high 2-D spatial resolution while the spectral information is badly separated into three overlapping bands in the VIS range. Some systems can acquire narrowband spectral information with high 2-D spatial resolution, but at the cost of a very long acquisition time.
Smartspectra is a smart multi spectral system that fits somewhere between these two approaches. The Smartspectra camera provides six bands with fully configurable spectral shape from snapshot to snapshot, in the range 400“1000 nm. Each band can be as narrow as 5 nm or as wide as an RGB band. At the same time, the limited number of bands reduces the acquisition time of the system yet maintaining the relevant spectral information. The small number of bands also reduces the subsequent processing time, encouraging the use of the system in real-time applications. As a variety of classification and quantification problems demonstrate, six bands is a good trade-off between resolving power and response time of the system.
For some applications, each band can be configured to acquire a linear combination of narrow bands. This functionality is useful in order to estimate convenient parameters such as the optimum multiple narrowband reflectance (OMNR) index with a single Smart Spectra band. The Smart Spectra camera acts like a RGB camera in terms of spatial resolution and integration time. Furthermore, the flexibility of band configuration allows the system to adapt to a wide variety of problems and changing situations in real-time.
Smart Spectra technology is affordable while assuring robustness. The system is intended to make multi spectral techniques accessible to industrial, environmental, and commercial applications. Although the system is not fully operative yet, a first prototype called autonomous tunable filter system (ATFS) is finished and is functional.
CHAPTER 2
MULTISPECTRAL DEVICES
The Smart Spectra system is different from other spectral systems on the market. The most similar devices available are spectrometers. These devices are used to analyze the spectrum of a point, a line or a bi dimensional image. As a result, the user gets a plot (1-D data), a matrix (2-D data) or a hypercube (3-D data), respectively, that represent the source in the spectral domain.
Point spectrometers are based on the spectral dispersion of a light beam using a reflection grating or prism. The spectrum is projected onto a linear array of sensors (CCD or photodiodes). Each element of the array is exposed to a single wavelength of light. The combination of the array elements responses forms the spectrum of the beam. The 1-D output data of these systems represents the radiance value for each wavelength. Linear spectrometers analyze a linear image by dispersing the light into a rectangular area that is projected to a 2-D array sensor. The output of these systems is a 2-D data matrix representing spatial information in one dimension and spectral information in the other.
Finally, multi band systems can capture 2-D spectral images by the use of a spectral filter in front of a 2-D sensor array. The band selection can be performed by the use of a wheel of filters or by the use of a single tunable filter. There are two main types of electronically tunable filters: Acousto-optic tunable filters (AOTF), and liquid crystal tunable filters (LCTF). Smart spectra uses acousto-optic tunable filter as the band selection filter.
2.1. ACOUSTO-OPTIC TUNABLE FILTER
AOTFs are solid state devices that act as electronically tunable spectral band pass filters. An AOTF consists of a properly oriented birefringent uniaxial crystal to which a piezoelectric transducer is bonded. The application of a radio frequency (RF) signal to the transducer produces an acoustic wave that propagates inside the crystal. The traveling acoustic wave modulates the refraction index of the material periodically, due to the elasto-optic effect. The process acts like a volume phase grating, leading to the diffraction of a particular wavelength that satisfies a specific momentum- matching condition inside the birefringent medium. The wavelength filtered by the crystal can be rapidly tuned across a wide spectral range by changing the applied RF signal. The acousto-optic interaction also changes the polarization state of a single wavelength. The structure of an anisotropic uniaxial medium permits two normal modes (ordinary and extraordinary) to propagate in any direction with mutually orthogonal polarizations and different velocities. If the AOTF input light beam at the tuned wavelength is extraordinarily polarized, then it emerges from the AOTF ordinarily polarized, and vice versa.
Fig 2.1 AOTF Design
The acousto-optic filter design uses two basic geometric configurations, collinear and non- collinear. In the non-collinear configuration depicted in fig.2.1, the directions of the optical and acoustic waves are different, causing diffracted and non-diffracted rays to be angularly separated at the filter output.
2.2. MULTISINE AOTF DRIVER
When a pure sinusoidal RF signal is applied to the AOTF, the crystal diffracts a single wavelength with a very narrow bandwidth. The diffraction efficiency is directly proportional to the RF power. Moreover, if a linear combination of pure sinusoids is applied to the crystal, the corresponding combination of wavelengths is diffracted. This characteristic can be used to increase the bandwidth of the filtered pass band by using a set of very close sine signals.
Fig 2.2Multisine AOTF Driver
The first prototype of the Smart Spectra optical tunable filter used this multi-sine approach to excite the AOTF, implemented as a custom RF driver design. The concept of this RF driver is shown in fig 2.2. A DSP is used to generate a multi-sine signal in base band, which is up-converted to the proper frequency range with an RF mixer circuit. This driver was tested on an AOTF showing a moderate performance. The response was not very good because the mixer added some inter modulation products to the RF signal. These products spread the power density and therefore reduced the diffraction efficiency in the selected band and increased the out-of-band diffracted light.
2.3. DIGITAL DIRECT SYNTHESISERS
Another approach takes advantage of the fast dynamic response of the acousto-optic devices and the fast switching capabilities of modern digital direct synthesizers (DDS). This technique drives the AOTF with a sweeping acoustic frequency, i.e., the frequency applied to the crystal varies cyclically from an initial frequency to a final frequency in constant steps. By taking an integration time that is a multiple of the complete sweeping time, the acquired image is proportional to the intensity of the spectral range diffracted by the different frequencies.
Fig 2.3.Block Diagram of sweeping frequency driver
A new RF driver was developed using this approach. Fig 2.3 shows the block diagram of the driver, which was implemented in the ATFS prototype. The core of the driver is the AD9858, a 1 Giga Sample per second DDS, capable of generating a frequency-agile analogue output sine wave at up to 400MHz. The DDS is clocked by a 1GHz signal generated by the AD4360-2 integrated synthesizer and voltage-controlled oscillator. Both integrated circuits are controlled by an ADuC842 microconverter. The RF signal amplitude is controlled by an AD8367 variable gain amplifier. The low-level RF signal output is amplified by the AP5300-2 power amplifier.
Fig 2.4.Diffraction Frequency of the sweeping frequency driver
The DDS has been programmed to generate a complete sweep within a 5 ms period. The initial and final frequencies are calculated in real-time from the bandwidth requirements, as well as the frequency step. Fig 2.4 shows the diffraction efficiency (D.E.) of the sweeping frequency driver when the RF generator was configured to filter a pass band centered at 800 nm with a variable bandwidth. Although the DE decreases as the bandwidth increases, the total energy contained in the pass band is kept almost constant, that is the efficiency of the system is not altered.
Table 2.1.Diffraction efficiency of sweeping frequency driver
Table 2.1 shows the diffraction efficiency of the sweeping frequency driver. The DDS can be switched instantly among four frequency profiles. This feature allows the generation of an arbitrary four sine signal in which each sine is used one-fourth of the time. For an integration time that is multiple of the signal period, the result is equivalent to the multi-sine approach with four sines.
CHAPTER 3
IMAGING SENSORS
The Smart Spectra camera uses a commercial 2-D sensor array. Three different sensor technologies can be applied to the system. They are
1. CCD Sensors
2. CMOS Sensors
3. Intensified Technologies
3.1. CCD SENSORS
CCDs are composed of a matrix of photogates on which the light is collected. The photons are converted into electrons (charge) and transferred to the output. At the output stage charge is converted to voltage. The bandwidth of the output limits the reading speed of the sensor. Conventional CCD sensors have a single output, but some CCD sensors have 2, 4 or even 32 outputs. A conventional CCD can read as many as 30 full images per second (for 640 X 480 sensor elements). A high-speed CCD with 32 outputs can read 300 full images per second. Multiple outputs may decrease the uniformity of the image, because of the differences among the output stages.
CCD characteristics greatly depend on the field of application. Applications in astronomy and science demand the highest performance, such as very low noise and high responsivity. The reduction of the readout noise is achieved with long time exposure and low readout speed. To increase the responsivity, CCDs for astronomy are normally back-thinned. Alternatively, CCDs for science have a large full-well capacity, allowing for 14 or 16 bits output digitalization. CCDs for astronomy and science use Full Frame architecture, which is the slowest but most sensitive architecture.
By contrast, industrial applications demand lower spatial resolutions and responsivity but higher data rates. CCD sensors for machine vision do not demand resolutions higher than 10 bits, while the required spatial resolution is very dependent on the application. Normally sensitivity is not an issue since lighting can be controlled.
3.2. CMOS SENSORS
CMOS sensors are formed by a matrix of photodiodes. Unlike CCD sensors, each photodiode is connected to a charge-to-voltage converter next to it. Therefore, the charge is not read sequentially at the sensor output, but in parallel inside each sensor element. Consequently, the readout speed can be much higher than in a CCD but the uniformity is much lower. CMOS sensors use the same manufacturing technology as other CMOS integrated circuits and are accordingly cheaper to produce than CCDs.
There are two main groups of CMOS sensors depending on the application. We can find low cost CMOS for cost-sensitive applications and high performance CMOS for high-speed imaging.
3.3. INTENSIFIED TECHNOLOGIES
Intensified technologies are used to boost the CCD responsivity in low light level conditions. The main technologies are:
1. Intensified CCD (ICCD)
2. Electron bombarded CCD (EBCCD)
3. Low light level CCD (L3CCD)
The ICCD uses an intensifier tube to increase the number of photons that hit the CCD by several orders of magnitude. The intensifier tube contains an electron multiplier and a phosphorescent output screen. Therefore, ICCDs can perform a wavelength shifting if the spectral response of the multiplier differs from the emitting wavelength of the phosphorus screen.
The EBCCD is a hybrid of the intensified and the conventional CCD. In this device, photons are detected by a photocathode similar to that in an image intensifier. However, the released electrons are accelerated across a gap and impact directly on the back side of a back thinned CCD instead of a phosphorus screen. These energetic electrons generate multiple charges in the CCD resulting in a modest gain of a few hundred.
Currently, a new technology called L3CCD can surpass the performance of the other intensified technologies. L3CCD technology improves the performance of conventional CCDs by inserting an on-chip multiplication stage prior to the output conversion. This stage increases the signal gain while the noise is not amplified. L3CCD technology resembles an image intensifier implemented in solid state.
CMOS based sensors match the requirements of many real problems at a lower cost and simplicity than CCD counterparts so they have been preferred for the development of the first prototypes of our system. Intensified technologies are an emerging market with breakthrough products being launched every few months with significant price reductions. These latter technologies can be the optimum choice in certain demanding applications but may not be in main stream products.
CHAPTER 4
SYSTEM DESCRIPTION
The system can be divided into two main blocks, the sensor and the host computer, as depicted in fig 4.1. The sensor part mainly involves optics and electronics, while the host computer comprises the driver for sensor configuration and image acquisition, image processing software, and the development of specific algorithms for the application fields of fruit quality assessment, agriculture, and environmental monitoring. The camera acquires six bands that may be located in the VIS and NIR spectra, and simulates two common Firewire RGB cameras. The ATFS prototype shown by dotted lines in fig 4.1 is a reduced version of the Smart Spectra system. It uses an off-the-shelf digital camera to acquire multispectral images but it does not include the toolkit software and the user utility program.
Fig 4.1.Block diagram of Smart Spectra
The input lens is used to focus the object inside the optical tunable filter. The optical tunable filter provides a wide working range of 400-1000nm due to the use of two transducers optimized for different frequencies. The cross over wavelength is 580nm corresponding to 130MHz acoustic frequency. The RF output is switched from one transducer to the other depending on the selected range. The optical tunable filter is excited by a sweeping frequency generator. The camera is used to acquire multispectral images.
The Smart Spectra host computer software presents three main blocks (see fig 4.1). The first is the driver that controls the multi spectral sensor and manages data acquisition; the second is the toolkit that provides the hyper spectral possibilities of the system to the end user, and finally the user utility software.
4.1. AUTONOMOUS TUNABLE FILTER SYSTEM
The ATFS is a finished prototype of the Smart Spectra with complete multi spectral capabilities. It is composed by the following hardware components:
Input lens: The function of this lens is to focus the object inside the optical tunable filter. Telecentric lenses have shown to be the best choice for AOTF imaging systems because they reduce chromatic aberration and stray light.
Optical tunable filter: The optical filter of the ATFS presents a wide working range of 400“1000 nm, due to the use of two transducers optimized for different frequencies. The crossover wavelength is 580nm (which corresponds to a 130MHz acoustic frequency). The RF output is switched from one transducer to the other, depending on the selected range.
RF generation circuit: The AOTF is excited by the above mentioned sweeping frequency generator. The acoustic frequency range is 218“130MHz for one transducer and 130“70MHz for the other.
Fire wire camera: The ATFS uses a commercial Fire wire camera to acquire the multi spectral images. The software driver on the host computer synchronizes the camera trigger with the RF generation circuit, and is responsible for the image data flow from the camera to the upper software layers. Currently, this driver supports different models of Fire wire cameras. In the ATFS prototype a high performance CCD camera was used providing 1360 X 1036 pixel images with 12-bit resolution.
.
CHAPTER 5
HOST SOFTWARE
The Smart Spectra host computer software presents three main blocks. The first is the driver that controls the multi spectral sensor and manages data acquisition; the second is the toolkit that provides the hyper spectral possibilities of the system to the end user, and finally the user utility software. The ATFS prototype uses the same low-level driver as the Smart Spectra system, but it does not require either the toolkit or the user utility software.
5.1. DRIVER FOR CAMERA AND FILTER CONTROL
The driver for sensor control of the Smart Spectra multi spectral sensor is a set of low-level functions that allow the toolkit in the host computer to communicate with the sensor configuration port and set up all its functionalities. This driver is used for both the ATFS prototype and the Smart Spectra toolkit.
The driver of the camera is divided into two parts, one is used to configure and perform the image acquisition, while the other is used to configure the filter. The synchronization between the RF generation and the camera acquisition is guaranteed by using this driver. The filter control block is responsible for sending the filter parameters to the RF generator (initial and final RF frequencies, frequency step, number of steps and RF power). These parameters are calculated by the driver from the wavelength and bandwidth specified by the upper software layer.
The driver for sensor control acts as an interface between the Smart Spectra toolkit and the multi spectral sensor. Given any acquisition request, the driver implements the camera configuration, the filter configuration and a synchronized acquisition when the filter is ready.
5.2. ATFS APPLICATION
The ATFS application provides a graphical user interface (GUI) to the sensor driver. This application allows the user to easily acquire multi spectral images. The user must specify a list of bands to acquire, with their central wavelengths, desired bandwidths and, optionally, integration times. If no integration time is provided, the ATFS application performs a smart auto-exposure procedure that estimates the best integration time for each band in order to maximize the dynamic range. Once the bands have been specified, the system sequentially takes the images for each band configuration and saves the whole set of images
5.3. TOOLKIT
The Toolkit is the high-level software layer to be used by the system integrators for accessing and programming the sensor. The main purpose of this software layer is to provide basic data structures and processing tools of the multi spectral images to extract useful information. The rationale of the toolkit is that system integrators and developers only need the basic tools to interface the low-level layer of the system.
Therefore, the main features of the toolkit are oriented to provide programming tools to achieve the above-mentioned connection of the system with the application integrator, and it focuses on the specific functionalities related with the features of the system, rather than providing general-purpose image processing techniques. The features of the currently developed toolkit can be summarized as follows:
¢ It allows using an object oriented programming language, CPP, widely used for application development in image analysis and machine vision.
¢ The user may easily extend the set of functions and abstract data types defined in the toolkit.
¢ The toolkit works on PC-based architectures under the most common operating systems: MS-Windows and Linux.
¢ The toolkit provides high-level functions to access to low-level drivers for sensor configuration, acquisition, etc. It also provides basic functionality for image representation and access, storage and other common multi spectral image manipulation functions.
¢ There also exist novel specific functions for multi spectral image processing.
5.4. USER UTILITY SOFTWARE
The User Utility software is a software application that provides potential users with a quick insight into the technical features of the Smart Spectra camera in an easy and effective way. This software application uses a graphical environment to display all the information extracted from acquired images (using the Smart Spectra camera), stored images (using the most extended graphic file formats) or the result of processing images using the available functions in the toolkit. The Smart Spectra User Utility software runs in MS Windows environment. The features of the User Utility software include:
¢ Enquiries to find the Smart Spectra cameras present and select them.
¢ Display and configuration of the Smart Spectra sensor parameters.
¢ Set acquisition mode and image acquisition format.
¢ Display acquired or stored multispectral images, showing particular multispectral features.
¢ It allows performing in a friendly environment most of the image processing functions available in the toolkit, including the specific multispectral processing data like band selection, multispectral invariant representations, etc.
CHAPTER 6
REAL TIME IMPLICATIONS
Contrary to other multi spectral systems, the Smart Spectra system is more suitable for real-time applications since it acquires a small amount of bands with fast and flexible band configuration. The six spectral bands provided by the system are enough for most applications.
Although theoretically, for a classification-oriented application, the more bands, the better the classification rate, the improvement in the classification rate is not significant from a certain number of bands. When adding more bands, the amount of information grows considerably, while the classification rate remains almost constant. In addition, in hyper spectral imaging, most of the information present in contiguous bands is redundant, since they tend to be highly correlated. A reduced number of spectral bands, around six, is enough to characterize most of the classification tasks. The system can use less than six bands when not required by the application, reducing further the acquisition and processing time.
Another advantage of the Smart Spectra system with respect to colour RGB images or any usual trichromatic representation is that the calculation of spectral invariants is more feasible. The fact of having narrow band representations allows calculating invariants with simple operations like subtracting and dividing band values of a pixel, speeding up further image processing steps. Therefore, many image processing tasks become easier and computationally efficient when invariant representations to illuminant intensity, colour, shades, highlights etc. can be obtained, improving the computational performance, for instance, of image segmentation and object recognition processes, and contributing to meet real time constraints in some demanding applications.
Regarding hardware aspects, the synchronization between the RF generator and the digital camera allows a fast multispectral acquisition. The total acquisition time is the summing of the sensor configuration and settling time, the camera integration time and the image readout and transfer time. The response time of the AOTF when switching on the acoustic frequency in the Smart Spectra RF generator is smaller than 5 ms. Likewise, the readout of the digital camera takes less than 70 ms and the image transmission about 56 ms. Those times can be normally neglected when compared to the integration time, in the range 200ms to 2s for the camera with indoor illumination.
CHAPTER 7
APPLICATIONS
Fruit and vegetable quality assessment: Present commercial applications on fruit and vegetable sorting include some special capabilities for estimating quality factors by means of spectral information. The trend for these systems is the use of spectral and spatial information in order to estimate and analyse the presence of pathologies in the skin, the amount of certain quality components like acids and sugar, the detection of chemical substances like pesticides that could be dangerous for health, water-core, ripeness, and so on.
Agriculture: The trends in agriculture are nowadays directed to rural monitoring of the effects of industrial and other aggressive environmental practices. The use of spatial information and multispectral imaging provides the assessment of vegetation pathologies, plant stress states, and the presence of certain substance in the crop, which allows a qualitative and quantitative evaluation of these features. This has got many implications in environmental monitoring and control. Another trend in this field is precision farming techniques in which crop management is performed locally. This requires the ability to detect and identify spatial distribution of the types of vegetation in crops. Once identified, using multi- or hyper-spectral imaging techniques, local treatment may be applied (for instance, irrigation, fertilization, insecticide or herbicide). The approach has broad implications on production costs and the environment management.
Remote sensing and aerial spectral imaging: Remote sensing was one of the first application areas where spectral imaging was used, in order to identify and monitor the natural resources and areas on earth surface. Aerial spectral imaging is being developed with the aim of monitoring natural resources like coastal areas, forestry and extensive crops. Spectral imaging is used to recognize different type of terrains and vegetation, the detection of aggressive phenomena like fire, pollution, etc. and the assessment of global levels of substances in water and vegetation like acid rain.
Medical spectral imaging: Biological tissues exhibit unique spectra in induced or auto fluorescence, and in transmission or reflection. Spectral differences in tissue pathology may be spatially resolved using imaging spectroscopy. An example of the application of this fact is the use of multispectral imagery of dental samples (hard tissue) for the study and diagnosis of caries. Other examples in soft tissues are the enhancement of certain features, for instance, blood vessels in organs for assistance in surgery operations.
Archaeology and art: In these applications spectral analysis is used for the recognition and detection of archaeological material, study of composition of products, or the enhancement of faded colours or writing. Another example is how the analysis of paintings for restoration can be supported by visible and near IR spectral imaging.
Other examples of target areas of application are quality assessment of industrial components, like semiconductors, electronic circuits, glass, etc. and all possible fields where spatial and spectral information is needed. In many of these fields, there are currently different applications and commercial products using spectral data. These applications now use ad-hoc sensors or devices made using cameras combined with filters.
CHAPTER 8
FUTURE IMPROVEMENTS
The Smart Spectra system is an extended version of the ATFS prototype. The Smart Spectra system will include some improvements over the ATFS hardware
Smart Spectra camera: The Smart Spectra system will include a Firewire camera based on the CMOS sensor. This camera will take full advantage of the optical filter. The camera and RF generator will share the same controller in order to configure the correct gain and integration time for each band, and maintain perfect synchronization.
Smart Spectra toolkit: The Smart Spectra toolkit has already been developed, but it has been tested only on the ATFS hardware. Some tests and minor corrections will be necessary for final smart spectra system
Smart Spectra hardware: In the final Smart Spectra system two different AOTFs covering the VIS and NIR parts of the spectrum will be used, instead of one AOTF with two transducers, since it has shown very bad performance in one transducer region. The RF generator will include an integrated 1W power amplifier and a RF power control loop.
Another AOTF excitation technique is also being investigated based on the use of a very high-speed digital to analog converter (DAC) circuit. This circuit will allow the generation of a multi-sine RF signal in the proper frequency range without the need of an up converting circuitry. This technique overcomes the drawbacks of the base band multi-sine generator while retaining all of its benefits. As the different RF frequencies will be applied simultaneously to the AOTF instead of sequentially (DDS driver), a great reduction of the integration times is expected.
.
CHAPTER 9
CONCLUSION
The Smart Spectra system proposes a new concept for multispectral imaging. It is a hybrid of common colour cameras (three broad overlapping bands) and spectrometers (many very narrow bands). It uses six bands that are configurable in central wavelength and bandwidth from snapshot to snapshot. Stress is placed on robustness, flexibility, and affordable cost in order to make it accessible to final users. A software platform is built to simplify the use of this camera. The use of a reduced number of quickly configurable bands makes the system especially suited for real-time applications.
The authors have developed the hardware for the first prototype using a commercial digital camera. This prototype has been used success fully for the estimation of the chlorophyll content on plant leaves. Real-time implications of the system have been analyzed throughout the paper.
REFERENCES
[1] sciencedirect.com
[2] elsevierlocate/rti
[3] smartspectra.com