Next: Proof of quadratic convergence Up: Newton's method for nonlinear Previous: Newton's method for nonlinear

## An example of the convergence of Newton's method

As a concrete example, I define by

The solutions are points of intersection of the circle x12+x22=1 and the parabola x2=x12. A simple graph shows that there are two solutions, and some simple algebra shows that the solution lying in the first quadrant is

(the other solution is the reflection of x* across the x2-axis).

Applying Newton's method with x(0)=(0.5,0.5), I obtain the results shown in Table 1. Several comments about these results are in order. First of all, the computations were carried out in IEEE double precision arithmetic, which translates to about 16 decimal digits of precision. Therefore, for example, where the results show that and , the apparent discrepancy is due to round-off error. Five iterations of Newton's method were sufficient to compute the solution exactly to the given precision, but when F(x(5)) is computed in floating point arithmetic, round-off error caused the result to differ from zero by a very small amount.

Table: Results of applying Newton's method to a nonlinear system.
 k x1(k) x2(k) 0 0.50000000000000000 5.0000000000000000 1 0.87500000000000000 0.62500000000000000 2 0.79067460317460314 0.61805555555555558 3 0.78616431593458214 0.61803398895790196 4 0.78615137786388734 0.61803398874989490 5 0.0000 0.78615137775742328 0.61803398874989490

Second, the convergence of to zero follows a definite pattern:

which suggests that the ratio

is asymptotically constant as . It is difficult to verify this conjecture numerically, since the error so quickly falls below round-off level. For example, I would predict that

but in fact this error is below the precision of the machine, and all I can verify is that is less than about 10-16. However, I will prove below that the conjecture is correct. In this regard, the following definitions are relevant.

Definition 2.1   Suppose is a sequence in that converges to x*.
1.
The sequence is said to converge linearly (or q-linearly) if there exists such that

2.
The sequence is said to converge superlinearly (or q-superlinearly) if

3.
The sequence is said to converge quadratically (or q-quadratically) if there exists such that

Cubic convergence, quartic convergence, and so on, are defined analogously (but are rarely used in analysis of optimization algorithms).

It is not difficult to show that quadratic convergence implies superlinear convergence, which in turn implies linear convergence.

Below I will prove the following theorem:

Theorem 2.2   Suppose is continuously differentiable and F(x*)=0. If
1.
the Jacobian J(x*) of F at x* is nonsingular, and
2.
J is Lipschitz continuous on a neighborhood of x*,
then, for all x(0) sufficiently close to x*, Newton's method produces a sequence that converges quadratically to x*.

Lipschitz continuity is a technical condition that is stronger than the mere continuity of J but weaker than the condition that F be twice continuously differentiable. Lipschitz continuity will be defined carefully below.

Quadratic convergence has two important consequences. First of all, it guarantees that if a point close to the solution can be found, then Newton's method will rapidly home in on the exact solution. Second, it provides a stopping test for Newton's method. Since Newton's method is an iterative algorithm, it is necessary to have some criteria for deciding whether the current approximation x(k) is sufficiently close to the solution x* that the algorithm can be halted. There is a simple stopping test for any superlinearly convergent sequence, which I will now derive.

I assume that it is desired to find x(k) such that , where is a given error tolerance, and I also assume that superlinearly. I will now show that

 (2)

This implies that (which is a computable quantity) is a good estimate of when x(k-1) is close to x*, and so it is reasonable to stop the iteration when is satisfied. This is not guaranteed to produce an estimate with an error less than , since it cannot be known for sure that x(k-1) is close enough to x* that

However, it works well because Newton's method usually does not take small steps unless it is close to the solution. Moreover, having verified that holds, so that is expected to hold, the algorithm then returns x(k), which should be much closer to x* than x(k-1) is. For these reasons, the stopping test

 (3)

is quite reliable.

To prove (2), I simply use the triangle inequality,1 the reverse triangle inequality,2 and the definition of superlinear convergence. First of all,

Second,

Therefore,

and so

yields (2).

Next: Proof of quadratic convergence Up: Newton's method for nonlinear Previous: Newton's method for nonlinear
Mark S. Gockenbach
2003-01-23