cosc428 final exam

Describe Homography, H

Relates relative pose of 2 cameras viewing a planar scene. Estimate from feature correspondences using RANSAC.

Describe the Essential Matrix, E

Relates relative pose of 2 cameras viewing a 3D scene. Estimate from feature correspondences using RANSAC.

Describe Bundle Adjustment, BA

Minimises reprojection error:

1. Initialise using RANSAC (for E)
2. Then estimate a set of 3D points and camera poses which minimises reprojection error in a 3D scene

What are the four steps in a Fiducial Marker Algorithm

1. Find Marker Outline
2. Estimate the homography (3DOF orientations/angles) using the corner positions.
3. Estimate the extrinsics (3DOF location/x,y,z in 3D space) using the focal length and marker size.
4. Extract the pattern internal to the quad and compare against a database for the marker ID

How does ArUco find the marker outline?

1. Adaptive thresholding to binary
2. Contour Extraction
3. Quad extraction and threshold on size

How does AprilTags find the marker outline?

1. Edge Detection (not binary)
2. Line segment detection
3. Quad extraction and threshold on size

Name the stages of a CNN in the order that they operate from an input image

1. Input image
2. Convolutional stage
3. Non-linear stage
4. Pooling stage

Name 3 commonly used deep learning framworks

1. Torch
2. Caffe
3. DeepDream

(had a deep dream holding a torch in a cafe)

List eight common image transformations and distortions

1. Warping
2. Blur
3. Translation
4. Rotation
5. Partial Occlusion
6. Illumination
7. Noise
8. Radial Distortion

List the five steps of the natural feature registration algorithm

1. Find all the feature points in the image
2. In order to differentiate each feature point, create a descriptor of a local window using a function
3. Repeat steps 1 & 2 for the source and current frame.
4. Compare all features in the source to all features in the current frame to find the closest matches.
5. Use matches to calculate the transormation

Name three natural feature registration algorithms

1. Vuforia
2. OPIRA
3. Ferns
4. MetalIO SDK

How do pixels in the camera differ from the photoreceptors in the human eye - colour space, distribution of colour, sensitivity and resolution.

Cameras use the RGB colour space and have evenly distributed CCD square pixel elements that approximate equal sensitivity to red, green and blue (25% red, 50% green, 25% blue) but with a much lower dynamic range and a much wider spectral resolution (can detect IR and ultraviolet). Cameras have a much higher frame rate.

Humans resemble the CIE colour space. Foveal 6.5 Mpixel 3 colour camera with a narrow angle lens as well as a peripheral sensitive 100 Mpixel monochrome camera with a wide angle lens. Has a limited spatial resolution of 1-3cm at a distance of 20 metres (with a blind spot. Cognitive processing ends up limiting the 10^8 :1 dynamic range to distinguishing approximately 100 colours and 16-32 shades of black and white. Graph of the colour response of the human photopic vision - red, green and blue cones after correction for absorption by the lens and other inert pigments. Hraph of the spectral response of the human foveal and peripheral vision

Draw the graph of the colour response of the human photopic vision - red, green and blue cones after correction for absorption by the lens and other inert pigments. Also draw the spectral response of the foveal/photopic vision and the peripheral/scotopic vision.

Write the formula for the gradient direction of an edge and the gradient strength of an edge in an image

angle = arctan((df/dy)/(df/dx))
magnitude = sqrt((df/dx)^2 + (df/dy)^2)

Describe how the Canny edge detection algorithm accomplishes Good Detection (filter responds to an edge), Good localisation (detect an edge near a true edge) and single response (one per edge).

Canny is susceptible to noise so it is first convolved with a first order Gaussian filter, the result is a slightly blurred image that is not affected by a single noisy pixel. Edges can occur in a variety of directions so Canny uses four filters to detect horizontal, vertical and diagonal edges using operators such as Sobel or Prewitt which returns the first derivative in each direction. The edge gradient and direction can then be determined where the angle is rounded to one of the four angles representing the vertical, horizontal and diagonal directions.

0. edge map from blurred image
1. norm of the gradient
2. thresholding to respond to edges not noise
3. thinning for good localization & one response per edge using non-maximum surpression, predict the next edge point and then construct the tangent to the edge curve
4. hysteresis to improve localization - use a high threshold to start edge curves and a lower threshold to continue them

How does the choice of Gaussian Kernel size affect the behaviour of the Canny edge detector?

Large sigma detects large scale edges and small sigma detects small scale edges

Describe how the Harris Detector works.

Harris captures the structure of the local neighbourhood using an Autocorrelation matrix where 2 strong eigenvalues indicate a good local feature, 1 indicates a contour and none indicate a uniform region. Harris gives a measure of the quality of a feature because the best feature points can be thresholded on the eigenvalues.

Describe how SIFT works

1. Thresholded image gradients are sampled over a 16x16 array of locations in scale space at 8 different scales
2. An array of orientation histograms are created at each location creating 128 dimesions (8 orientations x 4x4 histogram)
3. SIFT is based of vector angles so is computationally efficient

Compare the SIFT and the Harris Detector

SIFT and Harris are both illumination and rotation invariant because they are based on operators of gradient but they are not deformation invariant because deformations can change gradients. SIFT is scale invariant because it is sampled at different scales but harris is not, they are also both translation invariant for x & y motion perpendicular to the camera.

Describe how correctly matched points in two images enable finding depth values in a stereo pair of images.

1. One image is rectified with respect to the other using the essential matrix.
2. Points lying on a horizontal line in one image are matched with corresponding points on the same line in the other image using least squares of pixel values over a region around each point.
3. The "x" distance between a matching pair of points is called the disparity, the larger the disparity the closer that point is to the camera based on triangulation.

Describe how correctly matched points in two images enable finding optical flow in two successive frames of video using the Lucas-Kanade algorithm.

1. Lucas-Kanade method integrates gradients over a patch to find features good enough to track using the Harris detector.
2. A constant velocity is assumed for all pixels within an image patch to assist with matching features in successive frames and remove outliers.
3. Optical flow is the measure of the movement that feature points undergo in successive frames.

Describe how depth can be calculated from optical flow using a single camera.

Relative depth can be calculated from the velocity of optical flow points (which is larger when the depth is less). So absolute depth can be determined if the camera velocity and pose is known and the intrinsic camera parameters are known.

What are the advantages and disadvantages of the following fro obtaining depth values:
(a) structured light camera
(b) time-of-flight camera
(c) stereo camera
(d) LiDAR

(a) can provide very accurate results, sensitive to lighting conditions, sensitive to movement, very fast
(b) efficient distance algorithm, sensitive to background lighting, multiple reflections can mean measured distances are inaccurate
(c) complex to set up, computationally expensive, cheap
(d) high accuracy, weather and light independent, generate large datasets that are hard to interpret.

State the spectral resolution, dynamic range, spatial resolution at 20m and radiometric resolution (shades of colour and grey) of human vision.

400-700nm, 10^8:1, 1-3cm, 16-32 shades of grey and

How can the Hough Transform be generalised to detect curved lines in an image even when the curve doesnt have a simple analytic form?

1. Pick a reference point (Xc, Yc) and draw segments from that point to the boundary, measure the length and orientation (r, alpha) of that segment and record the gradient at that point.
2. form an A accumulator array of possible reference points, scaling factor and rotation angle.
3. For each pixel in the image compute psi
4. for each (r, alpha) corresponding to thi(x,y) do
   a. Xc - Xi + r(thi)Scos(alpha(thi) + theta)
   b. same for Y
   c. A(Xc,Yc,S,theta)++
5. Find the maxima of A

Describe the following three steps of TextonBoost:

1. convolution and clustering
2. boosted texture layout filters
3. alpha-expansion graph cut

1. Input image is convolved with a filter bank of 17 filters. Clustering then finds common combinations of filter responses where each pixel has 17 responses. After thousands of iterations of K-means clustering each pixel is assigned to its closest texton.
2. Recognizes the texture from the textons by using texture layout filters. Texton uses 5000 texture layout filters to distinguish between 21 classes of texture. Each texture is voted for based on the weight of the filter and the number of matching textons, the texture with the most votes will be assigned to that pixel.
3. a. Image converted to a graph
   b. Initial random labelling occurs
   c. randomly choose the next class to expand, this is called class alpha
   c. perform a graph cut where the two class are the previous class and the class alpha
   d. back to step 2 - always converges on local maxima.

What are the characteristics of a good local feature to track?

1. Satisfy brightness consistency
2. Have sufficient texture variation
3. Not have too much texture variation
4. Correspond to a "real" surface patch
5. Not deform too much over time

A fiducial marker can be used to find the 6-DOF pose of a camera. What are the five steps to finding this pose? Describe each of them.

1. Fiducial Marker Detection - Steps are taken to locate the marker such as the use of thresholding and edge detection.
2. Rectangle fitting - Rectangle is fitted to the fiducial marker using contours.
3. Pattern Checking - the pattern is extracted from the marker and checked against a database.
4. Lens undistortion - performed using the cameras intrinsic parameters
5. Pose estimation - estimates the x,y,z co-ordinates of the camera using the focal length and marker size as well as the 3DOF angles using the corner locations.

Briefly describe each of the four morphological operations and explain what effect they have on an image and why.

Erosion - places the structuring element on the pixel of an object and removes that pixel if the structuring element overlaps a non-object pixel shrinking the object.
Dilation - places the structuring element on an object pixel, if the neighbouring pixels are not part of the object it makes it part of the object - this grows the object.
Opening - A dilation operation followed by erosion. The dilation first fills gaps and holes in the image and then closing maintains the overall object size.
Closing - An erosion operation followed by dilation. Erosion removes noise and breaks narrow lines, the dilation operation then returns the remaining objects to their original size.

What does it mean to say the Fourier transform is a self-inverting transform?

This means that the function is its own inverse - if you apply it twice you will get back to your original input.

What does it mean if the centre of a Fourier transform is blacked out? What if only the centre wasn't blacked out?

(a) High-pass filter the inverse fourier transform of this would result in edge detection
(b) Low-pass filter, the inverse fourier transform of this would result in image blur

Describe the bundle adjustment algorithm.

1. The essential matrix (E) is initialised using the RANSAC algorithm.
2. The essential matrix estimates the relative pose of two cameras viewing a 3D scene.
3. The bundle adjustment algorithm then finds the set of camera positions and 3D points that minimise reprojection error.
4. Reprojection error is the distance between where a 3D point is projected into an image and where it is measured to lie.
5. Bundle adjustment does this using a steepest descent algorithm by estimating the gradient of the error with respect to the position and then follows the gradient downhill until the error is within a tolerance.

What are the features of a good edge detector?

1. Good Detection: filter responds to an edge not noise
2. Good Localization: detect edge near true edge
3. Single Response: One response per edge

If a virtual 3D model of a body can be kept aligned with a person moving in a 2D image, then it is possible to find all the joint angles for each frame of video, from that 3D model. So describe how three chained homogeneous transformation matrices can project a point b on the ith body part of a virtual 3D model onto a pixel p in a 2D image.

Homography estimates the relative pose of two cameras viewing a planar scene, using correspondences found using RANSAC. Each homogenous transform matrix will represent a plane being x-y, x-z and y-z. Using the 3 Dimensional co-ordinates the three homogenous matrices can map that virtual point onto the image by using the homography matrices for each respective plane.
p(x,b) = I(x,Ci(x,Bi(x,b))) - 3D body part frame of reference to 3D person frame of reference to 3D camera world frame of reference to 2D image.

Give a strength, weakness and an application for the CIE colour space as well as sketch the space.

Strength: percepptually easy to understand and mix colours.
Weakness: Based on human perception so it is not easily applied to computer vision as it was originally intended for humans to subjectively compare colours
Application: Colour temperature lighting for photographers.

Give a strength, weakness and an application for the RGB colour space as well as sketch the space.

Strength: Designed to represent the colours used in media such as displays or cameras so is immediately available to computer vision algorithms.
Weakness: Not all colours are perceptually uniform so it is hard to determine colour differences in the RGB colour space. This space is also device specific.
Applications: Computer graphics

Give a strength, weakness and an application for the HSV colour space as well as sketch the space.

Strength: More useful than RGB for analysing colours for example performing colour range checking.
Weakness: Not perceptually uniform and is device specific.
Application: Used by artists in photoshop, used in light bulbs.

What is meant by background subtraction, differencing, ghosting and foreground aperture?

1. Refers to the first frame or some derivative of it being the reference frame.
2. algorithm usually refers to the difference between two adjacent frames where the previous frame is the reference frame.
3. Refers to a second image of the moving object as an artifact of a difference algorithm.
4. Refers to a hole appearing in a moving object as an artifact of a difference algorithm.

How does the Laplacian of a Gaussian filter sharpen an image?

The Laplacian of a Gaussian filter subtracts the low frequencies (blurred image) from the original image - leaving the high frequencies (edges) remaining as a sharpened image with higher contrast.

Why does a sharpened image appear to have more content than its original image?

Although there is actually less content from the subtraction of the low frequencies. The accentuated high frequency edges gives the illusion of more content because there appear to be more edges and human perception is sensitive to edges.

Name five object recognition methods.

Pose clustering
Geometric hashing
Generalised Hough Transform
Template matching
Direction histogram

What is an advantage and a disadvantage of Pose Clustering?

1. Robust to partial occlusion
2. Clutter may lead receive more votes than the target

What is an advantage and a disadvantage of Geometric Hashing?

1. Fast but a relatively sparse hash table is critical for good performance.
2. Difficult to choose the size of the buckets

What is an advantage and a disadvantage of Generalised Hough Transform?

1. Computationally efficient for minimal parameters such as straight lines in a noise free image.
2. Does not scale well to multi-object scenes

What is an advantage and a disadvantage of Template Matching?

1. Good for finding small parts of an image which match a template image.
2. Does not handle partial occlusion well and other variations without a large increase in template numbers.

What is an advantage and a disadvantage of Direction histogram?

1. Features are invariant to illumation, 3D rotation and object variation.
2. Uses many small sub-templates to increase robustness to partial occlusion and other variations

What are the three main issues in tracking?

1. Prediction
2. Data Association
3. Correction

Describe the three main issues in tracking: Prediction, Data Association, Correction in the context of the Kalman filter.

Prediction: predict the state of the current frame based of the state in previous frames. The new state is multiplied by a known constant and some zero mean noise is added. Thus, the predicted state is a constant multiplied by the old state.
Data Association: Calculate the state from the current frame considering kinematic models and error minimisation.
Correction: If the measurement error is low (Gaussian Noise), used the measurement state from the current frame. Otherwise place a higher weighting on the predicted state.

How can an improved and smoothed estimate be made using a Kalman filter?

By running two Kalman filters, one in the forwards direction and one in the backwards direction and then combining the estimates. Here the backwards estimates are used as yet another measurement for the forward filter.

What are the advantages of a particle filter over a kalman filter?

1. Predicts multiple positions
2. It is multi-modal and non-Gaussian

What is the primary goal of Computer Vision?

To recognise objects and their motion.

What is the optic nerve? and what limitations does it place on our vision?

The optic nerve is the part of the thalamus that relays signals from the eyes to the visual cortex.
The optic nerve contains approximately 800,000 nerve fibers meaning rods and cones are interconnected giving approximately 10 output channels from the eye to the brain.

How would you measure a colour using a colour matching paradigm?

1. Pick a set of three primary colours.
2. Measure the amount of primary needed to match monochromatic light, t(lambda) at each spectral wavelength lambda.

How does the Hough transform work?

map a point in hough space to a line in the image (d = xcos(theta) + ysin(theta))

1. initialize H[d,theta]
2. for each edge point l[x,y] in the image
   for theta = 0 to 180
   d = xcos(theta) + ysin(theta)
   H[d,thetha] += 1
3. Find the values of (d, theta) where H[d, theta] is maximum
4. the detected line in the image is given by d = xcos(theta) + ysin(theta)

What is the general method for detecting corners?

1. Filter the image
2. Compute the magnitude and gradient everywhere
3. Construct a C window
4. Use linear algebra to find the two eigenvalues
5. If they are both big we have a corner

What is a local invariant feature? List the 5 advantages of local invariant features.

Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale and other imaging parameters.

1. Locality: robust to occlusion and clutter
2. Distinctiveness: individual features can be mapped to a large DB of objects.
3. Quantity: many featuresd can be generated for small objects
4. Efficiency: close to real time performance
5. Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness

Name two annoying pheonomena associated with lenses and describe each.

Chromatic Abberation: light at different wavelengths follows different paths, hence some wavelengths are defocused, for machines we can coat the lens.
Scattering at the lens surface: some light entering the lens system is reflected off each surface it encountered (Fresnel's Law), coat the lens interior.

Define geometric calibration

The relationship between coordinates in the real world and coordinates in the image are found. This can correct for lens distortion, focal length and the image centre.

Name two segmentation algorithms.

Mean Shfit segmentation: better when only pixel colour is being used.
Graph-cut segmentation: Better when a derived feature is being used, such as texture.

Outline how K-means clustering works

1. Choose how many clusters we want and place them randomly (in the case of TextonBoost 400)
2. Assign each pixel to its closest cluster center.
3. Move the cluster centers to the mean of their pixels.
4. Go back to step 2.

What three other features of TextonBoost help it achieve accurate results?

Location: Sky normally at the top of an image, grass at the bottom. Increase the probability when a texture occurs in its usual locaiton.
Colour: Gaussian mixture model, use K-means clustering to find common colours in a texture and increase the probability when it is its usual colour.
Edges: Decrease the probability if the neighboring pixel is different.

What are the advantages of passive vision systems (images)?

1. advantageous as they rely on capturing images
2. provide an ideal framework for capturing subjects in their natural environment.
3. advancement in the technology.
4. Provide a qualitative analysis

What are the disadvantages to marker tracking?

1. Requires markers to be worn and set-up procedures are extensive.
2. Rarely used in clinical or coaching environments
3. Unable to analyse existing video tapes.

Describe the following steps in particle filtering: Sample, Predict and Measure

1. Sample set contains possible alternate values for parameters. When tracking through backroud clutter or occlusion a joint angle may have N possible alternate values with respective weights.
2. Predict the sample set from the process density, oredict joint angles from the next frame considering the kinematic model, body model and error minimisation.
3. Measure and weight the new position in terms of the observation density. Estimate, normalize and smooth the weights.

What is a dyneme?

The smallest contrastive unit of movement. Human movement can be constructed from an alphabet of 35 dynemes

What is transfer learning?

Basically learning when building on existing knowledge. Can throw away the final classification layer or re-assemble the pre-trained parts to be part of a larger model. We can also only train new parts or use low global learning rates.

What is the Iterative Closest point algorithm

• Used to align two point clouds.
• For each point on the source surface find the closest point on the destination surface
• Measure the error between clouds using least means squared and calculate the optimal transformation to reduce it.

How can RANSAC (Random Sampling and Consensus) be used with point clouds?

Allows us to fit models to the data (cylinders, planes…), picks the minimum number of points to define the model, then tests the model against the remaining points to verify it. Left with the model parameters, inliers and outliers.

What are some common incorrect matches that occur during feature matching?

1. Self-similar environments: objects look the same but are different.
2. Corner features on occlusion boundaires which do not correspond to actual 3D points.
   3.

Why are homogenous coordinates used in feature tracking?

1 extra dimension is added, this means both rotation and translation can be represented using matrix multiplication and projection by normalization.

How does RANSAC work?

1. randomly choose the minimum number of datapoints needed to generate a model
2. Hypothesise: compute a model from the hypothesis set, if this model only contains inliers then the model will be correct.
3. Test: count how many datapoints would be inliers if the model was correct.
   Repeat until amodel compatible with a large number of data points is found, these datapoints are inliers.