1
Transient Imaging

Adrian JARABO

Graphics and Imaging Lab, University of Zaragoza, Spain

1.1. Introduction

In 1878, English photographer Eadweard Muybridge captured his famous Horse in Motion series, the first sequence of photographs able to capture a dynamic scene by using stop motion (Leslie 2001). His invention revolutionized photography, leading to ground-breaking innovations that ended up with the invention of the Lumière Cinématographe that gave birth to the cinema (Lumière 1936). But beyond that, this first sequence of photographs allowed people to analyze the dynamics of the horse at run, which was followed by many (up to 100,000 sequences) images of animals and humans in motion that allowed for better understanding on the dynamics of locomotion.

Being able to capture how the world evolves with time was a powerful tool for understanding the physical laws governing the world and how to carefully analyze experiments. Almost 90 years after Muybridge’s invention, Harold Edgerton pushed these photographic analyses of the world dynamics further, capturing them at 10,000 frames per second. A famous example of Edgerton’s captures is the Bullet Through Apple photograph, freezing in time ultrafast events such as the bullet penetrating through the apple (Bedi and Collins 1994). Seeing the world at that speed helped with understanding the mechanical behavior of fast dynamic systems such as liquid flows or explosions. However, while impressive on their own, all these high-speed photography techniques were not even close to capturing the fastest events in nature: the propagation of light.

Only recently, with the emergence of so-called transient imaging, it has been possible to capture events at a temporal resolution in the order of the speed of light, up to five orders of magnitude faster than Edgerton’s stroboscope. While the seminal work by Abramson introduced the first light-in-flights visualization using holography (Abramson 1978, 1983) for coherent light in small scenes, it was not until 40 years later when the first macroscopic light-in-flight animations were demonstrated with the introduction of femto-photography (Velten et al. 2012b, 2013); for the first time, we were able to see light propagating at a scene at roughly a trillion frames per second (see Figure 1.1).

Figure 1.1. Example capture of a light pulse propagating through a bottle filled with water using femto-photography (Velten et al. 2013), with an effective frame rate of 1 trillion frames per seconds. The large split image is a composite of the three complete frames shown in the insets

(source: Jarabo et al. (2017)).

The complete videos of this and other scenes captured with femto-photography can be downloaded from Velten et al. (2013a).

Capturing light transport at such temporal resolution has made it possible to see the invisible: from impressive animations of light in motion to a myriad of new computer vision and scene understanding techniques that leverage the enormous amount of information encoded in the temporal domain of light, enabling us, for example, to see around corners or through turbid media. In the rest of this chapter, we will introduce the mathematical properties of time-resolved light transport, the techniques developed for capturing light in the transient state, and some of the numerous scene understanding applications enabled by capturing light in motion. For a further in-depth review of transient imaging, we refer to monographs on the topic (Jarabo et al. 2017).

1.2. Mathematical formulation

One of the most general assumptions in computer vision and scene understanding is to consider the speed of light to be infinite, therefore assuming light transport in the steady state. This is a reasonable assumption since most of the existing imaging hardware is very slow compared to the speed of light. Even in the case of high-speed cameras, capable of capturing up to a few thousand frames per second, light travels tens of kilometers in each frame (Figure 1.2, left). With the emergence of ultrafast imaging devices, effective exposure times of nano- or pico-seconds have been made possible. At such exposure times, light travels a few centimeters per frame, effectively breaking the assumption of a steady state of light, requiring viewers to account for the time-resolved nature of light transport. The incoming light Lt(x, t) at the camera’s pixel x is a function defined on the temporal domain t (Figure 1.2). This time-resolved radiance relates to its steady-state counterpart Ls(x) as

Figure 1.2. Synthetic example of time-resolved light transport (data from Jarabo et al. (2014)). In the steady state (left), all light has finished propagating along the scene. However, in the transient state, the incoming radiance at the pixel is a function of time, and the light takes several nanoseconds to complete the propagation along the scene. The plot shows the temporal profile of the radiance in pixels (a), while the individual frames in the plot are shown as blue dashed lines.

Mathematically speaking, the time-resolved radiance Lt(x, t) relates to the impulse response of the scene T (x, t − τ) and the temporal profile of the illumination Li(t) as

with ∗t being the convolution operator on the temporal domain. In general, when we talk about transient imaging, we are interested on the impulse response of the scene. From equation [1.2], we can see that continuous illumination essentially reduces to equation [1.1] (see Figure 1.3); this illustrates the need for a temporally resolved illumination to capture transient light transport, ideally an ultrafast pulsed illumination that makes it possible to directly capture the impulse response of the scene, although as we will see in section 1.3, there are methods that make it possible to use continuous illumination for lower-cost transient imaging by coding the illumination in time.

Figure 1.3. Time-resolved incoming radiance (bottom) at the ground plane for different types of emission in a simple scene (top, the temporal profile of the emission is on the inset). The frequency in either the temporal or spatio-angular domain affects the frequency of the transient illumination. (a) A pulsed point light source generates a sharp signal on the temporal domain. By reducing the frequency in the spatial (b) or temporal (c) domains, by means of a pulsed area light or a non-Delta emission, respectively, the incoming light is no longer a differential pulse, and the high frequencies in the impulse response are lost.

Transient light transport, as encoded in the impulse response matrix, also presents a cross-dimensional information transfer between the temporal and angular domains, as demonstrated by Wu et al. (2012) by using a Fourier analysis on the spatio-directional time-resolved light transport. This property has a myriad of potential applications, including, for example, bare-sensor imaging. In addition, it also shows that a loss on angular frequency translates into a loss on the temporal frequency of the integrated time-resolved signal, as illustrated in Figure 1.3 (c). Therefore, in order to get a narrow-banded impulse response of the scene, a light source with a very small surface is desired.

Figure 1.4. Left: ray diagram illustrating the two-planes light field parametrization for an impulse point light source illumination. As the light propagates to distance d, the light field suffers from a hyperbolic curvature in the temporal domain. Right: overview of the operators for forward and inverse propagation of time-resolved light fields in both the primal and the frequency domains

(source: adapted from Wu et al. (2012)).

1.2.1. Analysis of transient light transport propagation

We have already seen one important property of the transient light transport encoded in T (x, t), which is the cross-dimensional transfer between the angular and the temporal domains. To study it further, and following Wu et al. (2012), let us first define the time-resolved light field as a time-dependent spatio-angular function Lt(x, υ, t), where υ encodes the direction of propagation of the light field. We can now define how a source light field (x0, υ0, t0) changes as it propagates through the scene to a parallel plane at distance d (see Figure 1.4, left), as

with x0 = x − υ d, υ0 = υ and t0 = t − , and c being the speed of light. Physically, light rays at more slanted angles (i.e. higher υ) take longer to propagate. For a pulsed point light source at position x0, this means that the time-resolved light field has a hyperbolic curvature in the spatio-temporal domain. In essence, this means that the spatio-temporal light field suffers two transformations as it propagates: a shear on x and a hyperbolic...