Lecture 6 Notes: Image Warping and Morphing

Transformations in 3D

• Local Space: Objects are created individually with their own local coordinates.

• Model Matrix: The first transformation, converting objects from local to world coordinates.

• The lecture focuses on the pipeline of transformations to take objects from their initial space to the screen.

• Key matrices: Projection matrix, Model matrix, and View matrix.

Coordinate Systems and Transformations

• Local Coordinates: Each object has its own origin.

• World Coordinates: Provide a consistent environment for assembling objects.

• View Space: Depends on camera location and viewpoint; the view matrix transforms world coordinates into the view space.

• Projection Matrix: Transforms the view space to the clip space (normalized space where x, y, and z are between -1 and 1), based on parallel or perspective projection.

• Viewport Transformation: Transforms the clip space to the screen space.

• The workshop will cover PVM matrices and transformations (Projection, View, Model).

Image Space Transformations: Warping and Morphing

• Warping: The geometric transformation of graphical objects from one coordinate to another.

• Attributes like color, texture, and normals remain unchanged during warping.

• Morphing: A more dramatic change where one object gradually transforms into a different object, affecting both shape and attributes.

Morphing: In-Between Spaces

• Morphing involves transitioning one object into another over time such that:

  • The intermediate images must represent the transformation of the actual 'object' and not simply the mixture of the input images.

  • It is necessary to perform extrapolation between the points from the start object to the end object as an average over time.

• At $t=0$, the image is the original one. At $t=1$, the image is the destination one. In between, the image is a combination of both with a weighted average.

Weighted Average and Interpolation

• To achieve morphing, the object’s geometry and shape between two states needs to be determined.

• Vector Definition: Define a vector $V = Q - P$, where $P$ is one point and $Q$ is another.

• Midpoint Calculation: Adding half of the vector $V$ to $P$ results in the midpoint.

  • $P + 0.5V = P + 0.5(Q - P) = 0.5P + 0.5Q$

• Extrapolation: Adding 1.5 times the vector $V$ to $P$ extrapolates beyond the line segment

  • $P + 1.5V = P + 1.5(Q - P) = -0.5P + 1.5Q$

• Coefficients can be adjusted for specific point estimation via interpolation or extrapolation.

Pixel-wise Interpolation

• One method involves interpolating each pixel value between the starting and destination images independently.

• Problem: misalignment of features between images.

  • Simple pixel-wise averaging can lead to issues when faces are not aligned because of unrelated pixel characteristics being averaged.

• Linear Interpolation / Cross Dissolve: A weighted interpolation over time to transition from one image to another.

  • The method is computationally efficient, but becomes problematic and introduces noise if the images are not well-aligned.

  • Averaging of the intensity values for the pixels is expressed as follow: $Image<em>{final} = w</em>1 * Image<em>1 + w</em>2 * Image<em>2$, where $w</em>1$ is the weight associated to the first image and $w_2$ the weight associated to the second image.

Warping for Alignment

• One solution to the alignment problem is to warp the images to align them before applying cross-dissolve or weighted averaging.

• Procedure:

  • Images are warped to align corresponding features.

  • Cross-dissolve or weighted average is then applied to transition from one aligned image to the other.

Non-Linearity

• Linear interpolation may not be the best approach; non-linear methods can also be considered to model gradual changes, offering a smoother dynamic.

Parametric Warping

• Considers the image space as a function mapping inputs (domain) to outputs (range).

• Changes can be applied either to the input (domain) or the output (range) of the image function.

• Changes to Range (affecting pixel values/intensity): Brightening an image by adding a value to each pixel.

• Changes to Domain (affecting input values): Low pass filtering or limiting certain values of $x$.

Transformations in Image Domain

• Common Transformations include translation, rotation and scaling.

• Translation: Applying translation to create more samples for data augmentation.

• Rotation: Rotating images to generate additional samples.

• Aspect Ratio: Changing the aspect ratio.

• Affine Transformation: Combining rotations, scaling, and translations.

• Perspective Transformation: Applying out-of-plane rotation, which can introduce perspective effects.

Global vs. Local Transformations

• Linear transformations can be used to model warping.

• Global Transformation: The same transformation is applied uniformly to all pixels.

• Local Transformation: The transformation varies based on the spatial location of pixels in the image.

Linear Transformations

• Modeled by matrices, such as scaling (uniform or non-uniform) and rotation.

• Scaling, whether uniform or non-uniform, can be modeled by a matrix with two scaling factors for $x$ and $y$ directions.

• Rotation can be parameterized using an angle $\theta$, where the transformation matrix depends directly on this angle.

Inverse of Rotation Matrix

• The inverse of a rotation matrix is equal to its transpose, simplifying calculations.

Forward and Inverse Warping

• If a transformation matrix is unavailable, forward or inverse warping can be used.

• Forward Warping:

  • Starts with a pixel in the original image and maps it to its corresponding location in the destination image.

  • The value of the original pixel is then transferred to the new location.

  • Problem:

    • The destination pixel may overlap multiple pixels, with values that need to be distributed resulting in a 'splotching' effect.

    • Holes and gaps may appear in the destination image.

    • Interpolation with neighboring pixels may be performed to fill empty spots.

• Inverse Warping:

  • Starts with a pixel in the destination image and maps it back to its corresponding location in the original image.

  • The value of the original pixel is assigned to the new pixel.

  • Problem:

    • If the location is between two pixels of the original image, then averages (interpolation) are taken to get the final value.

    • Repeated averaging may remove high frequencies and causes blurring.

Feature-Based Warping

• Uses line segments to determine pixel correspondence between original and destination images.

• These distances include:

  • perpendicular distance from the point to the line

  • distance from where the perpendicular touches on the line, to the starting point of the line segment.

• These line segments decide how to map pixels from one image to another.

• Uses the following formula to determine how a pixel $P$ maps to $P'$:

  • Calculates parameters $U$ and $V$ based on pixel $P$ and line segment endpoints.

  • Uses $U$ and $V$ to find the corresponding point $P'$ in the source image.

  • The formulas being:

    • $u = \frac{(P-X) \cdot (Y-X)}{||Y-X||^2}$

    • $v = \frac{(P-X) \cdot perp(Y-X)}{||Y-X||}$

• An extension include to use multiple line segments and calculates a weighted combination of their effects.

• The multiple line segments uses the following parameters that tune precision and smoothness:
  * $A$:
  * If $A$ is large reduces the effect of the distance.
  * If $A$ is small increases the effect of the distance.
  * $p$:
  * If length of the line segment increases, potentially creates high weight in that weighted combination of all the lines.
  * If the ditance is smaller, contributes again to a larger weight, means that the line is a larger weight.

Non-Rigid Transformation

• Uses control points to define shapes and curves, with displacement vectors indicating their movement.

• A convex hull that are made of triangles is created from the all control points.

• Calculating the New Point Location:

  • For all these points, apply the displacement vectors to update to a new state.

  • After, the point that started sitting inside of a triangle is moved (after control point movement). Then a linear interpolation is applied to find the value for this new point.

Linear Interpolation

• Uses the formula:

  • $x = a<em>1x</em>1 + a<em>2x</em>2 + a<em>3x</em>3$

• Subject to:

  • $a<em>1 + a</em>2 + a_3 = 1$

• The coefficients depend on distance, with closer points having higher coefficients.

• Partial or Piecewise Transformation: If parts of the image or piecewise transformation happen, the points will be moved. So piecewise transformation always has a chance that you might lose the connectivity and the straight line.

Topics Covered

• Definitions of warping and morphing.

• Simple method for morphing: cross dissolve.

• Parametric morphing.

• Feature-based morphing using line segments.

• Nonlinear transformation using control points and estimating new point locations.