Spatio-temporal Compressive sensing Active Pixel Sensor (SCAPS)


Project Personnel

Jie (Jack) Zhang, Tao Xiong, Chetan Singh Thakur


Many modern systems require the use of small and power efficient image sensors. These applications often demand image sensors to simultaneously satisfy many specifications such as size, spatial and temporal resolution, light sensitivity and motion blurring. For example, cameras for endoscopic procedures must not only be small to fit within the catheter, but also have high light sensitivity to acquire images within the body. Cameras mounted on unmanned air vehicles (UAV) for surveillance and mapping tasks must be power efficient and capable of providing videos with both high spatial and temporal resolution. Designing an image sensor that could satisfy these specifications simultaneously while satisfying Size Weight and Power (SWaP) requirement is not trivial. Furthermore, there are also direct trade-offs between many of the image sensor specifications. For example, high frame rate imaging limits the each frame’s exposure time. Short exposure time then leads to low light sensitivity and low signal to noise ratio (SNR). On the other hand, longer exposure can improve SNR but at the cost of increased motion blur.


We developed a camera to address these SWaP design challenges. We named it the Spatio-temporal Compressed Sensing Active Pixel Sensor (SCAPS). SCAPS is an all-CMOS implementation of temporal compressive sensing with pixel-wise coded exposure (PCE). This sensor can increase video pixel resolution and frame rate simultaneously while reducing data readout speed. Compared to previous architectures, this system modulates pixel exposure at the individual photo-diode electronically without external optical components. Thus, the system provides reduction in size and power compare to previous optics based implementations. The prototype image sensor (127 × 90 pixels) can reconstruct 100 fps videos from coded images sampled at 5 fps. With 20× reduction in readout speed, our CMOS image sensor only consumes 14μW to provide 100 fps videos.


Fig.1. Conventional Global Exposure vs. Pixel-wise Coded Exposure


Fig. 2. (A) An Optical Implementation of PCE (B) all-CMOS PCE implementation at the image sensor focal plane.


Fig. 3. Pixel and System architecture of our SCAPS image sensor


We implemented first version of SCAPS using 180nm standard CMOS. Fig. 4. shows an example that SCAPS can be used to reduce readout speed, enhance video SNR and reduce motion blur at the same time. ROW1 shows a video of a blinking eye captured by the image sensor at full rate without pixel wise coded exposure. This is a 100 fps video with 10ms frame exposure time between frames. The SNR and contrast of the scene is low as the signal level is weak. On the other extreme, ROW2 shows a video captured with 20 fps with 50ms exposure time between frames. The scene SNR increases but at the cost of motion blur and reduced framerate. SCAPS is then used to capture a similar scene using Te = 5, which allows 50ms of pixel exposure time. The coded image and recovered frames are shown in ROW3. Due to extra exposure time, SNR of the coded image improves. Since an overcomplete dictionary trained using blur-free 100 fps videos is used for sparse reconstruction, the reconstructed video’s blurring is reduced. In comparison, video in ROW3 has higher SNR than ROW1 with lower motion blur compared to ROW2. ROW3’s readout rate is also 14× less than ROW1.


Fig. 6. Enhancing scene SNR and reducing motion blur simultaneously using SCAPS


Fig. 5. A 100 fps video reconstructed using coded image in Fig. 4.



Insect-based Wireless Sensor Node 

Mobile wireless sensor nodes (MWSN) mounted with low power cameras have been utilized for surveillance, emergency management and monitoring of hazardous areas. In recent year, we have seen growing interests on converting insects such as beetles or dragonflies into the carriers for low power image sensors. The insects have many advantages in term of aerodynamics, maneuvering capabilities and the ability to carry payloads heavier than their mechanical counterparts. In order to provide unobstructed flight for the insect and high quality videos at all conditions, the image sensor must be light, low power, fast and adaptive: Given the flying speed of these insects (e.g., up to 15 meters/s for dragonflies) and relative fast scene change due to small field of view, the camera needs to be able to sample the scene at fast frame rate to reduce motion blurring. As wireless transmission accounts for a majority of the system power, the image sensor must limit its data transmission bandwidth. Finally, given the extreme applications, the image sensor needs to be adaptive and capable of providing high quality videos regardless of the insect flight speed and illumination intensity of the scene.

We are currently working on mounting the SCAPS sensor integrated with Ultra-Wide Band (UWB) radio onto a small dragonfly. Fig. 7 below show a conceptual illustration of such system. Fig. 8 shows the small lens mount and the integrated circuits.


Fig. 7. An conceptual illustration of an insect-based wireless camera sensor


Fig. 8. Small lens mount and system integration of the SCAPS image sensor.

Fast High Dynamic Range (HDR) Imaging

One of the advantages of SCAPS is that it allows exposure across multiple frames. The exposure time for every pixel can also be controlled independently. We think that these features can be used to acquire HDR videos efficiently with only single shots. Long exposure can used to capture details at dark illumination. Short exposure can be used to capture motion. We are working on this topic now, please check back for more details!

In The Media

Image Sensor World : Compressive Sensing Camera with Pixel-wise Coded Exposure


Hitomi, Y., Gu, J., Gupta, M., Mitsunaga, T., & Nayar, S. K. (2011, November). Video from a single coded exposure photograph using a learned over-complete dictionary. In Computer Vision (ICCV), 2011 IEEE International Conference on (pp. 287-294). IEEE.

Zhang, J., Suo, Y., Zhao, C., Tran, T. D., & Etienne-Cummings, R. (2015, October). CMOS implementation of pixel-wise coded exposure imaging for insect-based sensor node. In Biomedical Circuits and Systems Conference (BioCAS), 2015 IEEE (pp. 1-4). IEEE.

Zhang, J., Xiong, T., Tran, T., Chin, S., & Etienne-Cummings, R. (2016). Compact all-CMOS spatiotemporal compressive sensing video camera with pixel-wise coded exposure. Optics Express24(8), 9013-9024.