next up previous
Next: Advantages and disadvantages of Up: Newton's method Previous: Proof of quadratic convergence

Newton's method for unconstrained minimization

Since Newton's method for solving

\begin{displaymath}\min_{x\in{\bf {\rm R}}^n}f(x)
\end{displaymath}

is nothing more than Newton's method applied to the nonlinear system

\begin{displaymath}\nabla f(x)=0,
\end{displaymath}

the following theorem is a corollary to Theorem 2.2:

Theorem 3.1   Suppose $f:{\bf {\rm R}}^n\rightarrow{\bf {\rm R}}$ is twice continuously differentiable, and $x^*\in{\bf {\rm R}}^n$ satisfies
1.
$\nabla f(x^*)=0$;
2.
$\nabla^2f(x^*)$ is positive definite (and hence, in particular, nonsingular);
3.
$\nabla^2f$ is Lipschitz continuous on a neighborhood of x*.
Then x* is a strict local minimizer of f and, for any x(0)sufficiently close to x*, Newton's method defines a sequence that converges quadratically to x*.

I now give an example that shows that Newton's method can still converge if the hypotheses of the above theorem fail, specifically, if $\nabla^2f(x^*)$ is positive semidefinite and singular. I define $f:{\bf {\rm R}}^2\rightarrow{\bf {\rm R}}$ by

f(x)=(x1+x2-3)2+(x1-x2+1)4.

Then

\begin{eqnarray*}\nabla f(x)&=&\left[\begin{array}{c}
2(x_1+x_2-3)+4(x_1-x_2+1)^...
...2+1)^2\\
2-12(x_1-x_2+1)^2&2+12(x_1-x_2+1)^2\end{array}\right].
\end{eqnarray*}


An easy calculation show that, with x*=(1,2),

\begin{displaymath}f(x^*)=0,\ \nabla f(x^*)=\left[\begin{array}{c}0\\ 0\end{arra...
...la^2f(x^*)=\left[\begin{array}{cc}2&2\\ 2&2\end{array}\right],
\end{displaymath}

and x* is the unique global minimizer of f. However, the eigenvalues of $\nabla^2f(x^*)$ are 0 and 4, and so $\nabla^2f(x^*)$ is positive semidefinite and singular.

Begining with x(0)=(2,2), Newton's method produces the results shown in Table 2. (To save space, I only show iterates $x^{(16)},
\ldots,x^{(20)}$.)

 
Table 2: Results of applying Newton's method minimize a function of two variables.
k $\Vert x^*-x^{(k)}\Vert$ $\Vert\nabla f(x^{(k)})\Vert$ x1(k) x2(k)
16 $1.0765\cdot 10^{-3}$ $1.9962\cdot 10^{-8}$ 1.0007612194201734 1.9992387805798266
17 $7.1768\cdot 10^{-4}$ $5.9145\cdot 10^{-9}$ 1.0005074796134474 1.9994925203865526
18 $4.7846\cdot 10^{-4}$ $1.7525\cdot 10^{-9}$ 1.0003383197422977 1.9996616802577023
19 $3.1897\cdot 10^{-4}$ $5.1925\cdot 10^{-10}$ 1.0002255464948688 1.9997744535051312
20 $2.1265\cdot 10^{-4}$ $1.5385\cdot 10^{-10}$ 1.0001503643299121 1.9998496356700879
 

The results suggest that Newton's method produces a sequence that converges to x*. However, the convergence is definitely not quadratic. Indeed, the ratios

\begin{eqnarray*}\frac{\Vert x^{(17)}-x^*\Vert}{\Vert x^{(16)}-x^*\Vert}&\doteq&...
...{\Vert x^{(17)}-x^*\Vert}{\Vert x^{(16)}-x^*\Vert}&\doteq&0.6667
\end{eqnarray*}


strongly suggest that $x^{(k)}\rightarrow x^*$ linearly. A comparison between Tables 1 and 2 shows the desirability of quadratic (or at least superlinear) convergence.


next up previous
Next: Advantages and disadvantages of Up: Newton's method Previous: Proof of quadratic convergence
Mark S. Gockenbach
2003-01-23