Conditioning

1D Equations

Suppose I want to solve 
	f[x]=b
in a simple1D case.

f[x_]=(Sqrt[x])
x0=0.1;
Plot[{f[x],f[x0], f[x0]+f'[x0](x-x0)},{x, 0, 0.2},
AxesLabel->{"x","y"}]

The slope of the tangent line tells me how carefull I need to be in x to get an accurate answer in y and vice-versa. δy = f'[x0] δx or equivalently δx = (1/(f'[x0] )) δy.  I guess, I need to be carefull if the slope of the tangent line is zero.  Of course this is way fancier than it needs to be if the function is linear.   Relative error looks a little different  ((δy )/y)=((x0 f'[x0] )/(f[x0])) ((δx)/x)

Linear Equations

Suppose I want to solve 
	A.x=y.
As always we pick a vector norm and use the induced matrix norm.
	A.(x+δx)=y+A.δx
or in other words
	A.δx=δy
so using my vector norms
	||δy||≤||A|| ||δx||
and provided A  is invertible
	||δx||≤||(A^(-1))|| ||δy||
Again relative error is more important than absolute error and the appropriate formula is 
	((||δy||)/(||y||))≤||A|| ((||x||)/(||y||))((||δx||)/(||x||))=||A|| ((||x||)/(||A.x||))((||δx||)/(||x||))≤((||A||)/(||(A^(-1))||))((||δx||)/(||x||))
The number 
	κ(A)=((||A||)/(||(A^(-1))||))
is the condition number of the matrix A in the induced norm.  It tells you how sensitive the output of A.x=y is to relative perturbations in x. If you think about it for a second you will realize that
	κ((A^(-1)))=((||(A^(-1))||)/(||A||))=(1/(κ(A)))
so the same number describes the reverse process.

For the ||.(Subscript[||, 2]) norm this the ratio of the absolute values of the largest and smallest eigenvalues.  Matlab and Mathematica frequently compute condition numbers within other functions.

In practice this means that

A= RandomReal[{-1,1},{2,2}];
{σ1,σ2}=SingularValueList[A];
κ=σ1/σ2;
ϵ=0.3;
y0 = RandomReal[{-1,1},{2}];
x0=LinearSolve[A,y0];
Δθ=π/12.0;
xs=Table[x0 + ϵ Norm[x0] {Cos[θ], Sin[θ]},{θ,Δθ, 2π,Δθ}];
ys= Table[
A.xs[[i]],
{i,Length[xs]}];

GraphicsArray[
{Graphics[{
Point[x0],
{Blue, Map[Point, xs]},
Circle[x0,ϵ Norm[x0]]
},
Frame->True,
PlotRange->Norm[x0]{{-1.3,1.3},{-1.3,1.3}},
PlotLabel->"Input"],
Graphics[{
Point[y0],
{Red, Map[Point, ys]},
Circle[y0,κ ϵ Norm[y0]]
},
Frame->True,
PlotRange->κ Norm[y0]{{-1.3,1.3},{-1.3,1.3}},
PlotLabel->"Output"]
},
PlotLabel->StringForm["κ=``",κ]
]

General Problems

Suppose I have a function
	f((OverVector[x]))=(OverVector[y])
then the linearization near a point (OverVector[x0]) is 
	f((OverVector[x0]))+∇f((OverVector[x0])).((OverVector[x])-(OverVector[x0]))≈(OverVector[y])

The condition number for this function or algorithm is κ(∇f((OverVector[x0]))).