Lecture01 Recap unconstrained optimization(update)

Lecture 01: Recap of Unconstrained Optimization

Mathematical Optimization Basics

  • Objective Function: The function we aim to minimize or maximize.

  • Optimization Variables: Denoted as n for the optimization variables (or simply the variable).

  • Constraints: May include inequality constraints and equality constraints:

    • Inequality Constraints: c_i ≤ 0 for i in I

    • Equality Constraints: c_i = 0 for i in E

Assumptions

  • Functions involved: S, c_i : R^n -> R must be smooth (C² functions) and twice differentiable.

  • Finite number of constraints (considering only continuous optimization problems).

Examples of Optimization Problems

  • Portfolio Optimization: Example involving risk and returns optimization,

    • Objective: Minimize risk subject to constraints on budget, returns, etc.

Basic Definitions

  1. Feasible Set: Set of points satisfying all constraints.

    • Feasibility set is expressed as:S = {x in R^n | c_i(x) ≤ 0 for i in I, c_i(x) = 0 for i in E}

  2. Active Constraints: A point x is called active if at this point c_i(x) = 0 for some i.

  3. Optimality Conditions:

    • A solution x* is a global minimizer if for all x, f(x*) ≤ f(x).

    • A local minimizer if there exists a neighborhood M such that:f(x*) < f(x) for all x in M.

Classifications of Optimization Problems

  1. Unconstrained Optimization: No constraints where sets E and I are empty.

  2. Quadratic Optimization Problems (QP): when the objective function is quadratic and constraints are affine linear.

  3. Nonlinear Optimization Problems (NLP): generally refer to optimization problems that are not quadratic.

Unconstrained Optimization

  • Definition: Involves minimizing an objective function without any restrictions on the variables.

  • Necessary Optimality Conditions (First-order):

    • If x* is a local minimizer, then the gradient at that point must be zero: ∇f(x*) = 0.

  • Taylor Expansion: Key method used to demonstrate necessary conditions for local minima.

Theorems on Necessary and Sufficient Conditions

  1. Second-order Necessary Optimality Condition (SONC):

    • If x* is a local minimizer and f is twice differentiable, then the Hessian D²f(x*) should be positive semidefinite.

  2. Second-order Sufficient Optimality Condition (SOSC):

    • If ∇f(x*) = 0 and D²f(x*) is positive definite, x* is a strict local minimizer.

Algorithmic Approaches

  • Steepest Descent Method: A method used for finding local minima by moving in the steepest descent direction (negative gradient direction).

  • Line Search Strategy: Finding the minimum value along the direction of descent; involves minimizing the objective function iteratively.

  • Backtracking and Armijo Rules: Procedures to ensure sufficient decrease in function values while iterating.

  • Newton's Method: Utilizes second derivative (Hessian) to achieve faster convergence, known for its quadratic convergence rate.

  • Quasi-Newton Methods: For approximating Hessians to avoid direct computation, e.g., BFGS method.

Convergence Analysis

  • Rate of Convergence: Assessed using the quotient convergence rate, which indicates how quickly a sequence converges to a limit, categorized as:

    • Q-linearly: ||x_{k+1} - x^*|| ≤ k||x_k - x^*||

    • Q-superlinearly and Quadratically, where convergence rates are faster under certain conditions.