In-Depth Notes on Image Formation

Formation of Images: Projection, Illumination, and Sensors
Understanding image formation is essential for extracting information from the world, particularly in fields like computer vision, photography, and imaging technology. This process involves several components and principles that govern how we perceive and analyze images.
Key Questions in Image Formation

1. What determines where a point on the image plane will depict a point on a 3D object?

   • This is influenced by the geometric relationship between the camera, the object, and the image plane, as well as the camera’s intrinsic and extrinsic parameters.

2. What determines the light intensity hitting a specific location on the image plane corresponding to a point on the 3D object?

   • Factors include the illumination of the scene, reflective properties of the object surfaces, and the spatial distribution of light sources.

3. How is the light intensity at a specific point converted into a discrete value of a pixel (RGB) in a digital image?

   • This conversion process involves the use of sensors that capture light and translate it into electrical signals, which are then processed and quantized into digital values representing color and intensity.

Projection Geometry from 3D World to 2D Image
Image projection geometry is dictated by the camera model, which defines how 3D points are mapped onto a 2D image plane. The general model is central projection, where each point of a 3D scene is connected to the camera's center through rays that intersect the image plane.

• The mathematics underpinning this projection is crucial for accurate image representation and is foundational in developing imaging systems.

• We begin with an ideal camera which has a pinhole (or aperture) without a lens, known as a pinhole camera. This model simplifies the projection process, illustrating basic principles of light travel.

Perspective Projection and Properties
Perspective Projection: This forms the basis of most image formation processes, providing a realistic visual representation. The relationship between points in 3D space and their 2D projections is described mathematically by:
$(x, y) = (f \frac{X}{Z}, f \frac{Y}{Z})$
where ( f ) is the focal length, and ( X, Y, ) and ( Z ) are the coordinates of the 3D object.

• Properties: Images formed via perspective projection are inherently inverted—objects closer to the camera appear larger than those further away, establishing critical spatial relationships that are vital for depth perception. The size of objects in the image is inversely proportional to their distance from the camera, which is a fundamental aspect of visual interpretation.

Affine and Orthographic Projection

• Affine Projection: Models scenarios where projection occurs at infinity. This is particularly useful in computer graphics for rendering distant objects, where perspective is less significant, allowing for simplified calculations.

• Orthographic Projection: A simplified model for structures that are far from the camera. Under certain conditions, the projection equations simplify to linear transformations in 2D, which maintains the dimensions of objects without distortion and is beneficial in technical drawings and architectural projections.

Lighting
Effective optical systems not only capture images but also provide adequate brightness for the scenes being represented.

• Brightness: The intensity of light in a scene is measured in luminance (typically in candela per square meter), which relates to the energy flux incident on the image plane. Proper lighting is fundamental to ensure clear and discernible images.

• Photometric Concepts: Various quantities are measured, including luminous flux, illuminance, and brightness. Each of these depends on the wavelength of light and human visual perception, which is critical for applications like photography and film where color rendering and clarity are paramount.

Sensors and Digitization of Images
During image formation, light sensors convert light energy into electrical energy for subsequent processing. This involves a series of steps:

1. Sampling: The process of converting continuous luminance into discrete values occurs according to the Nyquist-Shannon theorem, which ensures proper representation of frequencies without loss of information.

2. Quantization: This step involves mapping continuous luminance values to discrete levels, often in binary formats (commonly 8 bits per pixel), which allows for the representation of a wide range of tones and colors in digital images.

   • The choice of bit depth directly influences the image quality, color representation, and detail in the final digital image.

Effects of Noise on Image Formation
Noise significantly impacts image quality and can obscure important details. It is typically modeled as random variables, with common types including Gaussian and Poisson noise.

• Key sources of noise include:

  • Dark current noise: Arises from thermal activity within the sensor, producing unwanted signals even when no light is present.

  • Shot noise: This type varies based on the rates of electron release events, being more pronounced in low light conditions due to the statistical nature of photon arrival.

  • Read noise: Associated with the readout process of signals by electronic sensors, this source of noise is critical in evaluating sensor performance and image fidelity.

Statistical Models of Images for Sensors
Models incorporate advanced techniques to evaluate lighting conditions and the effects of sensor imperfections on captured images. The output signal can be stylized as:
$V(x, y) = G[\beta(x, y) I(x, y) + N<em>{dark}(x, y) + N</em>{shot}(x, y) + W_{read}(x, y)]$
where ( V(x, y) ) is the output signal, ( G ) is a gain factor, ( \beta ) is a modulation function representing varying lighting, and the individual noise components accounted for help refine image quality.

Summary
This text delves into multifaceted aspects of image formation, emphasizing the intricacies of projection techniques, the role of sensor technology, and the impact of noise on image clarity. By linking geometric principles and photometric concepts, a comprehensive understanding of visual representations within computational vision systems is constructed, highlighting its relevance in technology-driven environments.