Hauptnavigation

Numerical Optimization

Veranstaltung Wochentag Termin Ort
042527 (Übung: 042528) Montag (Übung: Mittwoch) 10.15 - 12.00 Otto-Hahn-Str. 12 - Raum 1.056,
Campus Nord

Lecture Information

  • Module description: pdf
  • Language: English
  • Lecturer: Dr. Sangkyun Lee
  • TA: Sibylle Hess
  • Attendance to Übung is REQUIRED.
  • Office Hour (lecturer): Thursday 13:00-14:00, LS8, OH 12, Raum 4.023 (or by appointments)

Final Exam:

04.03.2016 (Friday) 10:00-12:00, OH12 R4.013

  • Use this form to register for the exam pdf.
  • Exam questions will be from lecture notes, discussions in class, and homeworks including bonus questions.
  • You can bring an A4 paper with your own summary note to the exam, but using ONLY ONE SIDE of the paper.


Content:

In this lecture we will learn theories and algorithms of numerical optimization. We study the mathematical structure of various optimization problems to design efficient algorithms. The structure is investigated by accessing the zero-th order (function values), the first order (derivatives), and the second order information (Hessians) about the objective function, as well as by looking into the geometry of constraints. We discuss constrained and unconstrained optimization problems in continuous spaces, focusing on understanding motivations behind technical details and analyzing convergence rate / algorithm complexity algorithms. Fundamental concepts such as optimality and duality will be discussed in detail, which also become popular tools to better understand algorithms in many areas including machine learning, data mining, and statistics. The importance of smoothness and convexity will be elaborated, especially in connection to regularization problems in high dimensions. Some advanced topics from non-smooth, large-scale, or matrix optimization will be included if time permits. Homework assignments will be given to check theoretical and practical understanding of techniques.


Aims:

The aim of this lecture is to provide students with understanding of fundamental concepts and techniques in optimization in an advanced level, so that students can understand and see how to use and design efficient numerical optimization algorithms for their own research problems.


References:

  • Numerical Optimization, J. Nocedal and S. Wright, 2nd Ed, Springer, 2006
  • Introductory Lectures on Convex Optimization, Y. Nesterov, Springer, 2004
  • Nonlinear Programming, D. P. Bertsekas, 2nd Ed., Athena Scientific, 2003 (2nd printing)
  • Convex Optimization, S. Boyd and L. Vandenberghe, Cambridge, 2004 [pdf]

Lecture Notes:

The lecture notes will be updated: check before each lecture. Questions / correcting errors will be welcomed. See the calendar below for lecture topics and schedule.
  • Lecture note from last semester [pdf]
  • Current version (incremental update): [pdf]
  • (19.10) Lecture 1: Introduction
  • > (optional reading) Intro to compressed sensing pdf

Homeworks:

  • A homework will be given approx. every two weeks. Some will require implementing algorithms in R or Matlab.
  • Exercises are at the end of each chapter in the lecture notes.
  • Submit your handwritten (must be readable)or TEX-typesetted answer at the BEGINNING of the lecture on the due date. On emergency, you can submit by email to the lecturer, before the lecture begins.
  • Dicussion with other students are encouraged, but copying is STRICTLY prohibited (you will not be allowed for the final exam). A rule of thumb: never share your written solutions to other studnets.
> Homework 1 (due 2.11, beginning of the class): exercises 1.1, 1.2 (a), 1.3, 1.4, and 1.5 (1.1 has been updated on 21.11, so please check the recent version)
> Homework 2 (due 16.11, beginning of the class): exercises 2.5, 3.1, 3.2, 4.1, **4.3, 4.4. (numbers marked with ** are optional for extra points)
> Homework 3 (due 30.11, 10:15am. Submit by email to the TA, Sibylle): exercises 5.2[10], 5.3[10], 6.1[10], 6.2 (1)[5], (2)[20], (3.a)[10], (3.b)[5], **5.1[10], **6.2(3.c)[5] (6.2 requres Matlab/Octave coding). Numbers in square brackets are the scores. Submit your latex typesetted solution, program code, a script to demo your code (zip everthing into a single achieve, firstname.lastname.zip or .tar.gz) MNIST data set: download
> Homework 4 (due 11.01.16 10:15am. Submit by email to the TA, Sibylle): exercise 8.1 (requires Matlab/Octave coding). Submit your latex typesetted solution, program code, a script to demo your code (zip everthing into a single achieve, firstname.lastname.zip or .tar.gz)
> Homework 5 (due 25.01.16 10:15am. Submit by email to the TA, Sibylle): exercise 9.1, 9.2(a) and *9.2(b). 9.2(a) requires coding, and 9.2(b) is optional for bonus points. Submit your latex typesetted solution, program code, a script to demo your code (zip everthing into a single achieve, firstname.lastname.zip or .tar.gz)
> Homework 6 (due 8.2.16. Submit in the beginning of the class at 10:15am): exercise 10.1, 10.3, 10.5, 10.6, 11.1, and *11.2. Problems marked with * are for bonus points.


Class Message Board:



Lecture Schedule:

The schedule may change, so please check often. Please ignore the lecture times on the calendar.