[back]

CMOS Imager with Embedded Analog Early Image Processor
Christophe Basset, Bedabrata Pain (JPL), Pietro Perona

Abstract. We are developing a computational CMOS imager with integrated early image processing general-purpose filter. The goal of this collaborative work with the Jet Propulsion Laboratory is to produce a single chip serving as a camera able to pre-process the image in real-time through a filter chosen by the user, allowing an efficient implementation of a variety of computationally intensive applications such as autonomous navigation, object avoidance or intercept, real-time target tracking and recognition.

Motivation. A system capable of tracking any target in real-time in an unknown environment finds numerous applications: object avoidance or interception, autonomous navigation (machine vision, nano-rovers, robots, docking...), recognition (tracking of eyes, nose...), etc. A hardware implementation is very well suited for such real-time vision systems. A software-based approach would lack miniaturization (need of a camera, a computer with a frame grabber) and would run too slowly (~1 frame/second) for most of these applications requiring a real-time flow of data.

The on-going collaboration with the Jet Propulsion Laboratory has brought to this project expertise in Active Pixel Sensors. This CMOS technology for building imager chips allows on-focal plane signal processing (as opposed to their CCD counterparts that need to serially output the flow of pixels to an external processing chip). The filtering can therefore be implemented as a fast, low-power analog circuit.

Research and Achievements. Tracking is achieved by matching a template to an image using a convolution unit, allowing generic filters to be used. The chip has an integrated imager array and a 7x7 pixel digital memory to store the kernel. For tracking, the kernel represents the template, chosen off-chip through a separate learning process. This part will therefore not be discussed here. Filtering is performed through a column-parallel architecture of computing units, so real-time computation can be achieved.

When tracking a target, the initial "best match" is located during a slow scan of the entire imager array (anticipated size of 1024x1024 pixel) allowing us to restrict the search for the following frames to a smaller, 25x25 pixel sub-window.

At each frame, the kernel is applied to a window of interest. In the case of tracking applications, the highest correlation response is determined in a decision-maker circuit. This feeds back to an addressing unit that updates the search window location, ensuring that it remains centered on the target. Figure 1 shows the block diagram of the feature-tracking chip.

The core of the filtering system relies on a 7x7 pixel cell which task is to perform convolution. The image provided by the array reaches the convolution block one row at a time. Hence, the 25x25 pixel search window can be processed by only 25 convolution units operating in parallel. Each of these cells computes the matching between the analog image (I) and the template (T):

The image is fed to the convolution unit one row at a time (making the system parallel in this direction). By implementing a series of delay lines, we can reconstruct the complete 25x25 convolution map in 25 cycles (one image line processed in parallel per cycle).

The imager provides the pixels as analog currents to the
convolution while the template comes as 8-bit digital lines. Convolution is obtained by mirroring and scaling the image currents (binary-scaled: by x2, x4, x8...). The scaled currents are dumped, through switches controlled by the template bits, into capacitors for read-out. To keep the transistors from getting too large (the most significant bit would be 128 times bigger than the least significant bit), the template is divided into two halves (mirrors scaled x1 to x8 each) and dumped into the accumulating capacitors with different integration times (50ns and 800ns respectively).

Figure 2. Simplified schematics of a pixel convolution unit

A first prototype of the convolution chip was built and tested. It consists of a 64x64 pixel imager array, a 49-byte digital memory to store the kernel, and a single 7x7 pixel convolution cell tied to the center of the image. Test results demonstrated a convolution was indeed performed and allowed identification of adjustments to be made for the next fabrication run.

Figure 3. Test setup

The second generation of the convolution chip has been tested and presented at the 2003 Workshop on CCDs and Advanced Image Sensors, May 2003 in Elmau, Germany. From the first generation, this run incorporated design modifications (we demonstrated better matching in computation cells and in the pixels) as well as added features (sum of all pixels necessary to perform normalization so targets can be tracked accurately, etc.). Figure 3 shows the chip on its test interface board.

Figure 4. Convolution linearity. Fixed image, template of varying intensity. The line in (a) shows the ideal response while the dots show the measured response.(b) is the difference plot between the two.

A new chip generation is being developped which will allows scanning of the entire image array at each frame. The computation core has been re-designed to address mostly timing performance. It will be able to output a filtered-version of the image in real-time, making it fully functional for a number of application requiring preprocessing of the image.

References
R.H. Nixon, et al, "256x256 CMOS Active Pixel Sensor Camera On A Chip", pp. 178-179, Proc. IEEE International Solid-State Conference, San Francisco, Ca. Feb. 1996

L.G. McIlrath, et al, "Design and analysis of a 512_768 current-mediated active pixel array image sensor", IEEE Trans. on Electron Devices, vol. 44, pp. 1706-1715, Oct. 1997.

C.Clark, et al, "Application of APS arrays to star and feature tracking systems", Proc SPIE, Vol. 2810, pp 116-120, 1996

T. Komuro, et al, "A digital vision chip specialized for high-speed target tracking", IEEE Trans. on Electron Devices, vol. 50, pp. 191-199, Jan. 2003.
A. Graupner, et al, "CMOS image sensor with mixed-signal processor array" IEEE Journal of Solid-State Circuits, vol. 38, pp. 948-957, June 2003.
A.A. Biyabani, L. R. Carley, T. Kanade, "An analog CMOS IC for template matching", proc. IEEE Int. Solid-State Circuits Conference, pp. 82-83, Feb. 1999