Numerical Basics and Data Fundamentals

  • The IEEE 754 standard is a widely used representation for floating-point arithmetic, specifying how numbers are represented and manipulated in computers.

  • It defines numbers through three main components: a sign bit indicating positive or negative, an exponent which allows for scaling the magnitude, and a mantissa (or significand) representing the precision bits of the number.

  • The complete value of a floating-point number is computed as: value=sign×(1.mantissa)×(2exponent)value = sign \times (1.mantissa) \times (2^{exponent}), where the mantissa is assumed to be normalized.

  • The exponent is stored in a biased format to accommodate both positive and negative values by using a bias value that depends on the total number of bits allocated for the exponent, which simplifies comparisons of numbers.

IEEE 754 Standard Types

  • The standard outlines several formats for floating-point representation, including binary16 (half precision), binary32 (single precision), binary64 (double precision), binary128 (quadruple precision), and binary256.

  • Among these, binary32 (float32) and binary64 (float64) are the most prevalent in modern computing applications, providing a good trade-off between range and accuracy.

import numpy as np

# Example of creating float32 and float64 arrays in NumPy
float32_array = np.array([1.0, 2.0, 3.0], dtype=np.float32)
float64_array = np.array([1.0, 2.0, 3.0], dtype=np.float64)
print(float32_array.dtype)  # Output: float32
print(float64_array.dtype)  # Output: float64
Special Float Features

  • Zero: Both positive and negative zeros exist, represented as (+0.0) and (-0.0), showcasing the precision of the representation.

  • Infinity: Positive and negative infinity are represented with special bit patterns where all exponent bits are ones and mantissa bits are zeros, denoted as (+\text{∞}) and (-\text{∞}).

  • NaN (Not a Number): This signifies undefined or unrepresentable values such as the result of dividing zero by zero and is also represented by specific bit patterns.

# Example of generating NaN and Infinity
nan_value = float('nan')
infinity_value = float('inf')
print(nan_value)  # Output: nan
print(infinity_value)  # Output: inf
Float Exceptions

  • The IEEE 754 standard defines several exceptional conditions in floating-point operations:

    • Invalid Operation: e.g., operations resulting in NaN, such as ( \frac{0.0}{0.0} ) or ( \text{sqrt}(-1.0)).

    • Division by Zero: This occurs when a non-zero number is divided by zero, yielding either +∞ or -∞ based on the sign of the numerator.

    • Overflow: This happens when a computation produces a result that exceeds the largest representable floating-point number.

    • Underflow: Occurs when a result is too small to be represented as a normalized floating-point number, leading to rounding toward zero.

    • Inexact: This flag is raised when a result cannot be exactly represented due to rounding errors.

# Example of division by zero
result = 1.0 / 0.0
print(result)  # Output: inf
NumPy and Float Exceptions

  • The popular Python library NumPy handles floating-point exceptions by default; it catches all floating-point exceptions except for inexact results, typically issuing a warning while allowing the computation to continue.

  • Users can customize behavior to throw exceptions for specific cases using settings in NumPy to enable stricter error handling appropriate for sensitive computational tasks.

# Example of managing floating point errors in NumPy
np.seterr(all='warn')  # Set warning for all errors
result = np.array([1.0, 0.0]) / np.array([0.0, 0.0])  # Warning will be shown
Roundoff and Precision Tips

  • Floating-point arithmetic introduces roundoff errors, which arise from the inability to represent certain decimal fractions accurately in binary format.

  • The distance (gaps) between representable floating-point numbers increases as the magnitudes of the values increase, potentially leading to significant errors in calculations with large numbers.

  • To maintain accuracy, avoid using the comparison operator (==) for floating-point numbers; instead, use methods that check for equality within a tolerance range, either absolute or relative, to account for potential roundoff errors.

# Checking float equality within a tolerance
def are_close(a, b, tol=1e-9):
    return abs(a - b) < tol

a = 1.0
b = 1.0 + 1e-10
print(are_close(a, b))  # Output: True
Machine Precision

  • The relative error in floating-point representation is bound by the machine epsilon: epsilon≤12×2−t\text{epsilon} \le \frac{1}{2} \times 2^{-t}, where $t$ represents the number of bits used for the mantissa, highlighting the intrinsic limits of floating-point accuracy.

NDArray Layout

  • In NumPy, arrays are stored as contiguous blocks of memory, with a header that describes their properties including dimensions, the total number of elements, the size of each element, the shape of the array, the strides (the number of bytes to step in memory for each dimension), and a pointer to the raw data.

Strides and Shape

  • Strides are crucial for efficient memory access, indicating the byte offset to move in memory to reach the next element along each dimension of a multi-dimensional array.

  • Understanding strides is key to leveraging efficient multidimensional indexing, allowing for optimized data manipulation.

# Example of checking strides in a NumPy array
array = np.array([[1, 2], [3, 4]])
print(array.strides)  # Output: (8, 4) depending on array type
Array Transformations

  • Transpose: Changing the shape of the data by switching rows and columns; this is performed as an (O(1)) operation, posing no computational overhead since it only changes the strides rather than the actual data.

  • Flipping/Rotation: Similar to transposing, but involves altering strides without needing to reorder the elements themselves, also an (O(1)) operation.

# Example of transposing a NumPy array
transposed_array = np.transpose(array)
print(transposed_array)
C and Fortran Order

  • C order (row-major): In this layout, the last index of a multi-dimensional array varies the fastest in memory layout, suitable for languages like C.

  • Fortran order (column-major): Here, the first index changes fastest, evident in languages like Fortran; this differentiation impacts performance due to how data is accessed in memory layouts.

# Example of specifying order in NumPy
array_c = np.array([[1, 2], [3, 4]], order='C')  # C order
array_f = np.array([[1, 2], [3, 4]], order='F')  # Fortran order
NumPy Types

  • By default, NumPy assigns float64 as the datatype for floating-point numbers and int32 for integers unless specified otherwise.

  • The .astype() method allows for efficient conversion between different array element types, optimizing memory usage.

# Example of changing data type in NumPy
int_array = np.array([1, 2, 3], dtype=np.int32)
float_array = int_array.astype(np.float64)
print(float_array.dtype)  # Output: float64
Rank Promotion

  • This occurs when new dimensions are added to an array, enhancing the dimensionality to facilitate operations in higher dimensions.

  • Rank promotion can be achieved through methods like broadcasting, replicating data with np.tile, or using indexing with None or np.newaxis.

# Example of adding a new axis
array_with_new_axis = int_array[:, np.newaxis]
print(array_with_new_axis.shape)  # Output: (3, 1)
Rank Reduction

  • Reducing dimensionality involves collapsing one or more dimensions from an array, streamlining data representation in simpler forms.

  • Methods to achieve rank reduction include indexing, utilizing functions like reductions such as np.sum(x), or invoking ravel() to flatten the structure.

# Example of reducing array rank
flattened_array = int_array.ravel()
print(flattened_array)  # Output: [1 2 3]
Squeezing

  • This operation removes dimensions of size one from the shape of the array using the np.squeeze() function, which helps in simplifying array dimensions.

# Example of squeezing an array
dim_one_array = np.array([[[1], [2], [3]]])
squeezed_array = np.squeeze(dim_one_array)
print(squeezed_array.shape)  # Output: (3,)
Elided Axes

  • In NumPy, utilizing an ellipsis (...) allows for replacement of repeated colons (:) in indexing, providing a concise notation for multi-dimensional indexing.

# Example of using ellipsis in indexing
array = np.array([[1, 2], [3, 4], [5, 6]])
print(array[..., 1])  # Output: [2 4 6]
Swapping and Rearranging Axes

  • The function np.swapaxes(a, axis1, axis2) facilitates the rearrangement of axes in a multi-dimensional array, providing flexibility for reshaping data for analysis.

# Example of swapping axes in a 3D array
three_d_array = np.array([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
swapped_axes = np.swapaxes(three_d_array, 0, 1)
print(swapped_axes)
Einstein Summation

  • Using np.einsum provides a powerful notational method for reordering higher-rank tensors. This function allows for concise and efficient computation of sums and products over specific axes of arrays, essential for many linear algebra operations.

# Example of using einsum for matrix multiplication
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
result = np.einsum('ik,kj->ij', A, B)
print(result)