Numerical Optimization

Unit 1: Introduction

• Mathematical Formulation using an example
• Continuous versus Discrete Optimization
• Constrained and Unconstrained Optimization
• Global and Local Optimization
• Stochastic and Deterministic Optimization
• Convexity
• Optimization Algorithms

Unit 2: Fundamentals of Unconstrained Optimization

• Concept of a Solution – Recognizing a Local Minimum
• Nonsmooth Problems
• Overview of Algorithms
• Two Strategies: Line Search and Trust Region
• Search Directions for Line Search Methods
• Models for Trust-Region Methods
• What is Scaling
• Line Search – Convergence of Line Search Methods
• Rate of Convergence – Convergence Rate of Steepest Descent
• Newton’s Method
• Quasi-Newton Methods
• Trust Region
• The Cauchy Point algorithm
• Global Convergence – Reduction Obtained by the Cauchy Point
• Convergence to Stationary Points

Unit 3: Conjugate Gradient Methods

• Basic Properties of the Conjugate Gradient Method
• A Practical Form of the Conjugate Gradient Method
• Rate of convergence

Unit 4: Calculating Derivatives

• Finite-Difference Derivative Approximations
• Approximating the Gradient
• Approximating a Sparse Jacobian
• Approximating the Hessian
• Approximating a Sparse Hessian

Unit 5: Theory of Constrained Optimization

• Local and Global Solutions
• Smoothness
• Examples – A Single Equality Constraint
• A Single Inequality Constraint
• Two Inequality Constraints
• Tangent Cone and Constraint Qualifications
• First-Order Optimality Condition
• Second-Order Conditions – Second-Order Conditions and Projected Hessians
• Linear and Non-linear Constrained Optimizatio
• Augmented Lagran
• ngian Methods