Multivariate Differentiation and Integration Course Notes

Basics of Vector Spaces and Multivariate Functions

  • Multivariate Functions: A multivariate function f:Rn→Rf: \mathbb{R}^n \rightarrow \mathbb{R} (with n>1n > 1) takes nn input arguments and returns one output value.

    • Examples:

      • The fuel consumption of a car depending on speed, road conditions, and weather.

      • Expression using specific variables: f(x,y,z)=x2+2yzf(x, y, z) = x^2 + 2yz.

      • Expression using a vector input: f(x)=f(x1,x2,x3)=(2x1+x2)x3f(\mathbf{x}) = f(x_1, x_2, x_3) = (2x_1 + x_2)x_3.

  • Arithmetic Operations: The sum, product, and ratio are defined similarly to single-variable functions:

    • (f+g)(x)=f(x)+g(x)(f + g)(\mathbf{x}) = f(\mathbf{x}) + g(\mathbf{x})

    • (f⋅g)(x)=f(x)⋅g(x)(f \cdot g)(\mathbf{x}) = f(\mathbf{x}) \cdot g(\mathbf{x})

    • (fg)(x)=f(x)g(x)(\frac{f}{g})(\mathbf{x}) = \frac{f(\mathbf{x})}{g(\mathbf{x})} (where g(x)≠0g(\mathbf{x}) \neq 0)

  • Vector-Valued Functions: Functions f:Rn→Rmf: \mathbb{R}^n \rightarrow \mathbb{R}^m have multiple inputs and multiple outputs. They are a combination of mm multivariate functions:

    • f(x)=(f1(x)f2(x)...fm(x))f(\mathbf{x}) = \begin{pmatrix} f_1(\mathbf{x}) \\ f_2(\mathbf{x}) \\ ... \\ f_m(\mathbf{x}) \end{pmatrix}

    • A simple example is matrix multiplication: f(x)=Axf(\mathbf{x}) = \mathbf{A}\mathbf{x}.

  • Definition 8.3: Matrices and Vectors:

    • Vector: A list of n∈Nn \in \mathbb{N} values xk∈Rx_k \in \mathbb{R}.

    • Column vector: x=(x1x2...xn)\mathbf{x} = \begin{pmatrix} x_1 \\ x_2 \\ ... \\ x_n \end{pmatrix}.

    • Row vector: x=(x1x2…xn)\mathbf{x} = \begin{pmatrix} x_1 & x_2 & \dots & x_n \end{pmatrix}.

    • Vectors are considered column vectors by default.

    • Matrix: A table A∈Rm×n\mathbf{A} \in \mathbb{R}^{m \times n} with mm rows and nn columns containing values Aj,kA_{j,k}.

  • Definition 8.4: Addition and Scalar Multiplication:

    • x+y=(x1+y1x2+y2...xn+yn)\mathbf{x} + \mathbf{y} = \begin{pmatrix} x_1 + y_1 \\ x_2 + y_2 \\ ... \\ x_n + yn \end{pmatrix}

    • ωx=(ωx1ωx2...ωxn)\omega \mathbf{x} = \begin{pmatrix} \omega x_1 \\ \omega x_2 \\ ... \\ \omega x_n \end{pmatrix}

  • Definition 8.5: Matrix and Vector Products:

    • A∈Rm×n\mathbf{A} \in \mathbb{R}^{m \times n} and B∈Rn×p\mathbf{B} \in \mathbb{R}^{n \times p} can be multiplied resulting in C=AB∈Rm×p\mathbf{C} = \mathbf{AB} \in \mathbb{R}^{m \times p}, where Cj,k=∑l=1nAj,lBl,kC_{j,k} = \sum_{l=1}^n A_{j,l}B_{l,k}.

    • Matrix-Vector Product: Ax=(A1,1x1+A1,2x2+⋯+A1,nxnA2,1x1+A2,2x2+⋯+A2,nxn…Am,1x1+Am,2x2+⋯+Am,nxn)\mathbf{A}\mathbf{x} = \begin{pmatrix} A_{1,1}x_1 + A_{1,2}x_2 + \dots + A_{1,n}x_n \\ A_{2,1}x_1 + A_{2,2}x_2 + \dots + A_{2,n}x_n \\ \dots \\ A_{m,1}x_1 + A_{m,2}x_2 + \dots + A_{m,n}x_n \end{pmatrix}.

  • Definition 8.7: Inner Product and Norm:

    • Inner Product: For x,y∈Rn\mathbf{x}, \mathbf{y} \in \mathbb{R}^n, ⟨x,y⟩=yTx=∑k=1nxkyk\langle \mathbf{x}, \mathbf{y} \rangle = \mathbf{y}^T \mathbf{x} = \sum_{k=1}^n x_k y_k.

    • Norm Properties:

      1. ∥x∥≥0\lVert \mathbf{x} \rVert \geq 0

      2. ∥x∥=0  ⟺  x=0\lVert \mathbf{x} \rVert = 0 \iff \mathbf{x} = \mathbf{0}

      3. ∥ωx∥=∣ω∣∥x∥\lVert \omega \mathbf{x} \rVert = |\omega| \lVert \mathbf{x} \rVert

      4. ∥x+y∥≤∥x∥+∥y∥\lVert \mathbf{x} + \mathbf{y} \rVert \leq \lVert \mathbf{x} \rVert + \lVert \mathbf{y} \rVert

    • Euclidean Norm: ∥x∥2=⟨x,x⟩=∑k=1nxk2\lVert \mathbf{x} \rVert_2 = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle} = \sqrt{\sum_{k=1}^n x_k^2}. Distance between vectors is measured as ∥x−y∥\lVert \mathbf{x} - \mathbf{y} \rVert.

  • Corollary 8.8: Cauchy-Schwarz Inequality: For vectors x,y∈Rn\mathbf{x}, \mathbf{y} \in \mathbb{R}^n, the inequality ∣⟨x,y⟩∣≤∥x∥2∥y∥2|\langle \mathbf{x}, \mathbf{y} \rangle| \leq \lVert \mathbf{x} \rVert_2 \lVert \mathbf{y} \rVert_2 holds.

Limits and Continuity in Multivariate Functions

  • Definition 8.9: Convergence and Continuity:

    • Convergence: A sequence of vectors xk∈Rn\mathbf{x}_k \in \mathbb{R}^n converges to x∈Rn\mathbf{x} \in \mathbb{R}^n if lim⁡k→∞∥xk−x∥=0\lim_{k \rightarrow \infty} \lVert \mathbf{x}_k - \mathbf{x} \rVert = 0.

    • Continuity: f:Rn→Rf: \mathbb{R}^n \rightarrow \mathbb{R} is continuous at x0\mathbf{x}_0 if for every ϵ>0\epsilon > 0 there exists δ=δ(ϵ,x0)\delta = \delta(\epsilon, \mathbf{x}_0) such that ∥x0−x∥<δ  ⟹  ∣f(x0)−f(x)∣<ϵ\lVert \mathbf{x}_0 - \mathbf{x} \rVert < \delta \implies |f(\mathbf{x}_0) - f(\mathbf{x})| < \epsilon.

    • Equivalently: lim⁡k→∞xk=x0  ⟹  lim⁡k→∞f(xk)=f(x0)\lim_{k \rightarrow \infty} \mathbf{x}_k = \mathbf{x}_0 \implies \lim_{k \rightarrow \infty} f(\mathbf{x}_k) = f(\mathbf{x}_0).

  • Directional Limits: Multivariate functions can approach a point from infinitely many directions. A function is only continuous if the limit is identical regardless of the direction chosen.

  • Example 8.10: The function f(x,y)=x2x2+y2f(x, y) = \frac{x^2}{x^2 + y^2} with f(0,0)=0f(0,0) = 0 is discontinuous at (0,0)(0,0).

    • Limit along x-axis: lim⁡x→0f(x,0)=x2x2=1\lim_{x \rightarrow 0} f(x, 0) = \frac{x^2}{x^2} = 1.

    • Limit along y-axis: lim⁡y→0f(0,y)=0y2=0\lim_{y \rightarrow 0} f(0, y) = \frac{0}{y^2} = 0.

    • Limit along diagonal: lim⁡x→0f(x,x)=x22x2=12\lim_{x \rightarrow 0} f(x, x) = \frac{x^2}{2x^2} = \frac{1}{2}.

    • Since limits differ by direction, the function is discontinuous at (0,0)(0,0).

Partial and Directional Derivatives

  • Definition 8.11: Partial Derivative and Gradient:

    • Partial derivative involves differentiating with respect to one variable while holding others constant.

    • Let gk(t)=f(x1,…,xk−1,xk+t,xk+1,…,xn)g_k(t) = f(x_1, \dots, x_{k-1}, x_k + t, x_{k+1}, \dots, x_n).

    • The kk-th partial derivative: ∂f∂xk(x)=fxk(x)=gk′(0)\frac{\partial f}{\partial x_k}(\mathbf{x}) = f_{x_k}(\mathbf{x}) = g'_k(0).

    • Gradient: If all partial derivatives exist, the vector containing them is the gradient: ∇f(x)=grad f(x)=(∂f∂x1(x)…∂f∂xn(x))\nabla f(\mathbf{x}) = \text{grad } f(\mathbf{x}) = \begin{pmatrix} \frac{\partial f}{\partial x_1}(\mathbf{x}) \\ \dots \\ \frac{\partial f}{\partial x_n}(\mathbf{x}) \end{pmatrix}.

  • Example 8.12: Critical Analysis of ∣x∣y|x|^y:

    • f(x,y)=∣x∣yf(x, y) = |x|^y for x≠0x \neq 0 and f(0,y)=0f(0, y) = 0 for y≥0y \geq 0.

    • Partial derivative fx(x,y)=yx∣x∣y−2f_x(x, y) = yx|x|^{y-2} for x≠0x \neq 0 or y>1y > 1.

    • Partial derivative fy(x,y)=∣x∣yln⁡(∣x∣)f_y(x, y) = |x|^y \ln(|x|) for x≠0x \neq 0 or y>0y > 0.

    • At (0,0)(0, 0), the limit along the x-axis is 11 while along the y-axis it is 00, showing jump discontinuity.

  • Example 8.13: Gradients of Norm-based Functions:

    • g(x)=∥x∥22  ⟹  ∇g(x)=xg(\mathbf{x}) = \frac{\lVert \mathbf{x} \rVert^2}{2} \implies \nabla g(\mathbf{x}) = \mathbf{x}.

    • h(x)=∥x∥  ⟹  ∇h(x)=x∥x∥h(\mathbf{x}) = \lVert \mathbf{x} \rVert \implies \nabla h(\mathbf{x}) = \frac{\mathbf{x}}{\lVert \mathbf{x} \rVert} for x≠0\mathbf{x} \neq 0. Note that it is not differentiable at 0\mathbf{0}, similar to ∣x∣|x| in 1D.

  • Definition 8.14: Directional Derivative:

    • Calculates change along a unit direction vector h\mathbf{h} (∥h∥2=1\lVert \mathbf{h} \rVert_2 = 1).

    • Formula: Dhf(x)=∂f∂h(x)=lim⁡t→0f(x+th)−f(x)tD_{\mathbf{h}}f(\mathbf{x}) = \frac{\partial f}{\partial \mathbf{h}}(\mathbf{x}) = \lim_{t \rightarrow 0} \frac{f(\mathbf{x} + t\mathbf{h}) - f(\mathbf{x})}{t}.

    • The kk-th partial derivative is a directional derivative where h\mathbf{h} is the kk-th unit vector eke_k.

  • Theorem 8.16: Calculating with Gradient: If partial derivatives are continuous, then the directional derivative is the dot product of the gradient and the direction vector:

    • fh(x)=(∇f(x))Th=⟨∇f(x),h⟩=∑k=1nhk∂f∂xk(x)f_{\mathbf{h}}(\mathbf{x}) = (\nabla f(\mathbf{x}))^T \mathbf{h} = \langle \nabla f(\mathbf{x}), \mathbf{h} \rangle = \sum_{k=1}^n h_k \frac{\partial f}{\partial x_k}(\mathbf{x}).

Multivariate Differentiation and Linear Approximation

  • Total Differentiability: Unlike single-variable calculus, division by a vector is undefined, so the derivative is defined using a linear approximation.

  • Definition 8.19: f:Rn→Rf: \mathbb{R}^n \rightarrow \mathbb{R} is differentiable at x0\mathbf{x}_0 if there exists a unique linear function f′(x0):Rn→Rf'(\mathbf{x}_0): \mathbb{R}^n \rightarrow \mathbb{R} such that:

    • f(x)−f(x0)=f′(x0)(x−x0)+r(x)f(\mathbf{x}) - f(\mathbf{x}_0) = f'(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0) + r(\mathbf{x})

    • Where lim⁡x→x0r(x)∥x−x0∥=0\lim_{\mathbf{x} \rightarrow \mathbf{x}_0} \frac{r(\mathbf{x})}{\lVert \mathbf{x} - \mathbf{x}_0 \rVert} = 0.

  • Theorem 8.21: Properties and Connections:

    • If all partial derivatives exist and are continuous at x0\mathbf{x}_0, then ff is differentiable.

    • If ff is differentiable, then all partial derivatives exist, and f′(x0)(x−x0)=⟨∇f(x0),x−x0⟩f'(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0) = \langle \nabla f(\mathbf{x}_0), \mathbf{x} - \mathbf{x}_0 \rangle.

    • Differentiability implies continuity.

  • Example 8.23: Counterexamples:

    • Case 1: Inverse of Theorem 8.21 — Function with a gradient that is NOT differentiable:

      • f(x,y)=xyx2+y2f(x, y) = \frac{xy}{x^2 + y^2} at (0,0)(0,0). Gradient is 0\mathbf{0}, but linear approximation error r(x,y)r(x, y) does not decrease fast enough.

    • Case 2: Differentiable function with non-continuous partial derivatives:

      • g(x,y)=(x2+y2)sin⁡(1x2+y2)g(x, y) = (x^2 + y^2) \sin(\frac{1}{\sqrt{x^2 + y^2}}) at (0,0)(0,0).

  • Theorem 8.24: Operational Rules:

    • Sum: (f+g)′=f′+g′(f + g)' = f' + g'

    • Product: (f⋅g)′(x0)=f(x0)g′(x0)+g(x0)f′(x0)(f \cdot g)'(\mathbf{x}_0) = f(\mathbf{x}_0)g'(\mathbf{x}_0) + g(\mathbf{x}_0)f'(\mathbf{x}_0)

    • Quotient: (fg)′=gf′−fg′g2(\frac{f}{g})' = \frac{g f' - f g'}{g^2}

  • Theorem 8.26: Multivariate Chain Rule: If f:R→Rf: \mathbb{R} \rightarrow \mathbb{R} and g:Rn→Rg: \mathbb{R}^n \rightarrow \mathbb{R}, then:

    • (f∘g)′(x0)=f′(g(x0))⋅g′(x0)(f \circ g)'(\mathbf{x}_0) = f'(g(\mathbf{x}_0)) \cdot g'(\mathbf{x}_0)

    • ∇(f∘g)(x0)=f′(g(x0))∇g(x0)\nabla (f \circ g)(\mathbf{x}_0) = f'(g(\mathbf{x}_0)) \nabla g(\mathbf{x}_0)

  • Extra 8.27: Jacobi Matrix: For a map f:Rn→Rpf: \mathbb{R}^n \rightarrow \mathbb{R}^p, the derivative is an n×pn \times p matrix called the Jacobi matrix, where entry (j,k)(j, k) is the kk-th partial derivative of the jj-th output.

Radial Functions and Mean Value Theorem

  • Definition 8.29: Radial Functions: Functions that depend only on the distance from the origin: f(x)=g(∥x∥)f(\mathbf{x}) = g(\lVert \mathbf{x} \rVert) or f(x)=g(∥x∥2)f(\mathbf{x}) = g(\lVert \mathbf{x} \rVert^2).

    • Gradient Formula: ∇f(x)=g′(∥x∥)x∥x∥\nabla f(\mathbf{x}) = g'(\lVert \mathbf{x} \rVert) \frac{\mathbf{x}}{\lVert \mathbf{x} \rVert} or ∇f(x)=2g′(∥x∥2)x\nabla f(\mathbf{x}) = 2g'(\lVert \mathbf{x} \rVert^2)\mathbf{x}.

  • Theorem 8.30: Multivariate Mean Value Theorem: If ff is differentiable on the segment from a\mathbf{a} to b\mathbf{b}, there exists ω∈(0,1)\omega \in (0, 1) such that:

    • f(b)−f(a)=(∇f(a+ω(b−a)))T(b−a)f(\mathbf{b}) - f(\mathbf{a}) = (\nabla f(\mathbf{a} + \omega(\mathbf{b} - \mathbf{a})))^T (\mathbf{b} - \mathbf{a}).

  • Lipschitz Continuity (Example 8.31): A function is Lipschitz continuous (∣f(x)−f(y)∣≤C∥x−y∥|f(x) - f(y)| \leq C \lVert x - y \rVert) if its gradient is bounded.

    • f(x)=∥x∥f(\mathbf{x}) = \lVert \mathbf{x} \rVert is Lipschitz continuous because ∥∇f∥=1\lVert \nabla f \rVert = 1.

    • g(x)=arcsin⁡(∥x∥)g(\mathbf{x}) = \arcsin(\lVert \mathbf{x} \rVert) is not Lipschitz continuous because the gradient is unbounded as ∥x∥→1\lVert \mathbf{x} \rVert \rightarrow 1.

Higher Order Derivatives and Schwarz's Theorem

  • Definition 8.32: Higher Order Partial Derivatives:

    • ∂2f∂xi∂xj(x)=fxixj(x)=∂∂xj(∂f∂xi)\frac{\partial^2 f}{\partial x_i \partial x_j}(\mathbf{x}) = f_{x_i x_j}(\mathbf{x}) = \frac{\partial}{\partial x_j} (\frac{\partial f}{\partial x_i}).

    • Multi-derivatives are read from left to right.

    • f∈Ck(Rn)f \in C^k(\mathbb{R}^n) means the function is kk-times continuously partially differentiable.

  • Theorem 8.34: Schwarz's Theorem: If f∈Ck(Rn)f \in C^k(\mathbb{R}^n), the order of partial derivatives does not matter:

    • ∂2f∂xi∂xj=∂2f∂xj∂xi\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial^2 f}{\partial x_j \partial x_i}.

  • Counterexample 8.35: Failure of Schwarz's Theorem: For f(x,y)=xy(x2−y2)x2+y2f(x, y) = \frac{xy(x^2 - y^2)}{x^2 + y^2} at (0,0)(0,0), fxy=−1f_{xy} = -1 and fyx=1f_{yx} = 1. The second order partial derivatives are not continuous at the origin.

  • Definition 8.36: Multi-index Notation:

    • A multi-index is a vector k=(k1,…,kn)\mathbf{k} = (k_1, \dots, k_n) where kj∈N0k_j \in \mathbb{N}_0.

    • Order: ∣k∣=∑j=1nkj|\mathbf{k}| = \sum_{j=1}^n k_j.

    • Factorial: k!=k1!⋅k2!⋅⋯⋅kn!\mathbf{k}! = k_1! \cdot k_2! \cdot \dots \cdot k_n!.

    • Derivative: ∂kf=∂∣k∣f∂x1k1…∂xnkn\partial^\mathbf{k} f = \frac{\partial^{|\mathbf{k}|} f}{\partial x_1^{k_1} \dots \partial x_n^{k_n}}.

Multivariate Taylor Polynomials and Hessian Matrices

  • Theorem 8.37: Multivariate Taylor Polynomial: For f∈Cm(Rn)f \in C^m(\mathbb{R}^n), there exists y0y_0 on the line between x0\mathbf{x}_0 and x\mathbf{x} such that:

    • f(x)=∑∣k∣≤m−1∂kf(x0)k!(x−x0)k+∑∣k∣=m∂kf(y0)k!(x−x0)kf(\mathbf{x}) = \sum_{|\mathbf{k}| \leq m-1} \frac{\partial^\mathbf{k} f(\mathbf{x}_0)}{\mathbf{k}!} (\mathbf{x} - \mathbf{x}_0)^\mathbf{k} + \sum_{|\mathbf{k}| = m} \frac{\partial^\mathbf{k} f(y_0)}{\mathbf{k}!} (\mathbf{x} - \mathbf{x}_0)^\mathbf{k}.

  • Taylor degree definitions:

    • Degree 1: Tangent (hyper)plane. T1(f,x0)(x)=f(x0)+⟨∇f(x0),x−x0⟩T_1(f, \mathbf{x}_0)(\mathbf{x}) = f(\mathbf{x}_0) + \langle \nabla f(\mathbf{x}_0), \mathbf{x} - \mathbf{x}_0 \rangle.

    • Degree 2: Quadratic approximation. Includes the Hessian matrix.

  • Definition 8.41: Hessian Matrix: The symmetric matrix of second order partial derivatives:

    • Hf(x0)=(∂2f∂x1∂x1…∂2f∂x1∂xn………∂2f∂xn∂x1…∂2f∂xn∂xn)H_f(\mathbf{x}_0) = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1 \partial x_1} & \dots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\ \dots & \dots & \dots \\ \frac{\partial^2 f}{\partial x_n \partial x_1} & \dots & \frac{\partial^2 f}{\partial x_n \partial x_n} \end{pmatrix}.

    • Quadratic Form: T2(f,x0)(x)=f(x0)+⟨∇f(x0),x−x0⟩+12(x−x0)THf(x0)(x−x0)T_2(f, \mathbf{x}_0)(\mathbf{x}) = f(\mathbf{x}_0) + \langle \nabla f(\mathbf{x}_0), \mathbf{x} - \mathbf{x}_0 \rangle + \frac{1}{2}(\mathbf{x} - \mathbf{x}_0)^T H_f(\mathbf{x}_0) (\mathbf{x} - \mathbf{x}_0).

    • Every multivariate quadratic polynomial can be written as 12xTHx+bTx+c\frac{1}{2}\mathbf{x}^T\mathbf{H}\mathbf{x} + \mathbf{b}^T\mathbf{x} + c.

Optimization: Local Minima and Maxima

  • Definition 8.42: Local Extrema:

    • Local Maximum: f(x)≤f(x0)f(\mathbf{x}) \leq f(\mathbf{x}_0) for ∥x−x0∥<ϵ\lVert \mathbf{x} - \mathbf{x}_0 \rVert < \epsilon.

    • Local Minimum: f(x)≥f(x0)f(\mathbf{x}) \geq f(\mathbf{x}_0) for ∥x−x0∥<ϵ\lVert \mathbf{x} - \mathbf{x}_0 \rVert < \epsilon.

    • Extremum is strict if the inequality is strict for x≠x0\mathbf{x} \neq \mathbf{x}_0.

  • Theorem 8.43: Necessary Condition: If x0\mathbf{x}_0 is a local extremum, then ∇f(x0)=0\nabla f(\mathbf{x}_0) = \mathbf{0}.

    • Points where the gradient is zero are candidates for extrema.

    • Gradient Descent Application: To find a minimum, move in direction −∇f-\nabla f: xk=xk−1−ω∇f(xk−1)\mathbf{x}_k = \mathbf{x}_{k-1} - \omega \nabla f(\mathbf{x}_{k-1}).

  • Matrix Definiteness (Definition 8.44):

    • Positive Definite: xTHx>0\mathbf{x}^T\mathbf{H}\mathbf{x} > 0 for all x≠0\mathbf{x} \neq 0.

    • Positive Semidefinite: xTHx≥0\mathbf{x}^T\mathbf{H}\mathbf{x} \geq 0.

    • Negative Definite: xTHx<0\mathbf{x}^T\mathbf{H}\mathbf{x} < 0 for all x≠0\mathbf{x} \neq 0.

    • Indefinite: Matrix satisfies none of the above.

  • Test 8.46: Eigenvalues: A symmetric matrix is:

    • Positive definite if all eigenvalues ωk>0\omega_k > 0.

    • Negative definite if all eigenvalues ωk<0\omega_k < 0.

    • Indefinite if it has both positive and negative eigenvalues.

  • Test 8.48: Sylvester's Criterion: Let HkH_k be the upper left k×kk \times k sub-matrix.

    • Positive definite if det⁡(Hk)>0\det(H_k) > 0 for all kk.

    • Negative definite if det⁡(Hk)<0\det(H_k) < 0 for odd kk and det⁡(Hk)>0\det(H_k) > 0 for even kk.

  • Theorem 8.50: Sufficient Conditions for Extrema:

    1. If Hf(x0)H_f(\mathbf{x}_0) is positive definite, x0\mathbf{x}_0 is a strict local minimum.

    2. If Hf(x0)H_f(\mathbf{x}_0) is negative definite, x0\mathbf{x}_0 is a strict local maximum.

    3. If Hf(x0)H_f(\mathbf{x}_0) is indefinite, x0\mathbf{x}_0 is an inflection point (saddle point).

    4. If semidefinite, the case is inconclusive.

Convex and Concave Multivariate Functions

  • Definition 8.53: Convexity: f(wx1+(1−w)x2)≤wf(x1)+(1−w)f(x2)f(w\mathbf{x}_1 + (1-w)\mathbf{x}_2) \leq w f(\mathbf{x}_1) + (1-w)f(\mathbf{x}_2) for all w∈(0,1)w \in (0, 1).

  • Theorem 8.54: Hessian and Convexity:

    • ff is convex   ⟺  Hf\iff H_f is positive semidefinite everywhere.

    • ff is strictly convex if HfH_f is positive definite everywhere.

    • ff is concave   ⟺  −f\iff -f is convex.

Basics of Multivariate Integration

  • Definition 9.1: n-dimensional interval: An interval I⊂RnI \subset \mathbb{R}^n is defined as I=[a1,b1]×[a2,b2]×⋯×[an,bn]I = [a_1, b_1] \times [a_2, b_2] \times \dots \times [a_n, b_n].

    • Area/Volume: ∣I∣=∏k=1n(bk−ak)|I| = \prod_{k=1}^n (b_k - a_k).

  • Multivariate Riemann Integral (Definition 9.2): Defined via Riemann sums over hyperrectangles. ff is integrable if the sum RS(f, Xn,TnX_n, T_n) converges to a constant cc as the partition size tends to zero.

  • Example 9.4: Riemann Sum of xyxy: For f(x,y)=xyf(x, y) = xy on [0,1]×[0,1][0,1] \times [0,1], the integral is 14\frac{1}{4}.

  • Theorem 9.6: Fubini's Theorem: If ff is Riemann integrable on [a,b]×[c,d][a, b] \times [c, d], then the double integral can be calculated as iterated single integrals:

    • ∬If(x,y)dxdy=∫cd(∫abf(x,y)dx)dy\iint_I f(x, y) dx dy = \int_c^d \left( \int_a^b f(x, y) dx \right) dy.

    • Counterexample 9.5: f(x,y)=x2−y2(x2+y2)2f(x, y) = \frac{x^2 - y^2}{(x^2 + y^2)^2}. Iterating in different orders yields −π4-\frac{\pi}{4} and π4\frac{\pi}{4}. This function is not Riemann integrable.

  • Null Sets (Definition 9.11): A set X⊂RnX \subset \mathbb{R}^n has measure 0 if it can be covered by intervals whose total size is less than any arbitrary ϵ>0\epsilon > 0.

    • Examples: Point sets, lines in R2\mathbb{R}^2, planes in R3\mathbb{R}^3, and graphs of continuous functions.

  • Theorem 9.14: Lebesgue Integrability Condition: A bounded function is Riemann integrable if and only if it is continuous almost everywhere (i.e., its discontinuities form a null set).

Integration on Normal Domains

  • Domain Boundaries (Definition 9.16): Boundary points ∂X\partial X are points where any neighborhood contains points both inside and outside XX. If ∂X\partial X is a null set, and ff is continuous on XX, ff is integrable.

  • Simple Regions (Definition 9.19):

    • x-simple: X={(x,y)∣a≤x≤b,ρ1(x)≤y≤ρ2(x)}X = \{ (x, y) | a \leq x \leq b, \rho_1(x) \leq y \leq \rho_2(x) \}.

    • y-simple: X={(x,y)∣a≤y≤b,ρ1(y)≤x≤ρ2(y)}X = \{ (x, y) | a \leq y \leq b, \rho_1(y) \leq x \leq \rho_2(y) \}.

  • Theorem 9.20: Iterated Integral on Simple Regions:

    • For x-simple: ∬Xf(x,y)dA=∫ab(∫ρ1(x)ρ2(x)f(x,y)dy)dx\iint_X f(x, y) dA = \int_a^b \left( \int_{\rho_1(x)}^{\rho_2(x)} f(x, y) dy \right) dx.

  • Example 9.18: Ellipse Integration: Integrating x3y+12yx2+4x^3y + 12yx^2 + 4 over x2+2y2≤1x^2 + 2y^2 \leq 1 result in 22π2\sqrt{2}\pi. It requires splitting the constraint into y=±1−x22y = \pm \sqrt{\frac{1 - x^2}{2}}.

  • Example 9.21: trigonometric bounds: For f(x,y)=x+1f(x, y) = x + 1 on 0≤x≤π0 \leq x \leq \pi, cos⁡(x)≤y≤sin⁡(x)+2\cos(x) \leq y \leq \sin(x) + 2, the volume is π2+3π+4\pi^2 + 3\pi + 4.