Finite Differences are Approximate Derivatives

The simplest differences are easy to understand.

First Derivatives

The following are all approximations to the definition of the derivative

CD1[x_,h_] :=(f[x+h]-f[x-h])/(2 h)
CD2[x_,h_] :=(f[x+h/2]-f[x-h/2])/h 
CD3[x_,h_]:= ((-f[x+2h]+8 f[x+h]-8 f[x-h]+f[x-2 h])/(12 h))
FD[x_,h_] :=(f[x+h]-f[x])/h
BD[x_,h_] :=(f[x]-f[x-h])/h

h=0.001;
f[x_]:=Sin[x (E^(-(x^2)))+ Cos[x]]
LogPlot[ {
h,
(h^2),
Abs[f'[x]- CD1[x,h]],
Abs[f'[x]- CD2[x,h]],
Abs[f'[x]- CD3[x,h]],
Abs[f'[x]- FD[x,h]],
Abs[f'[x]- BD[x,h]]
},{x, -1,1},
GridLines->Automatic,
Frame->True,
FrameLabel->{"x","Error"},
PlotLabel->"Error of FD Approximations TO f'[x]",
PlotLegends->{"h","(h^2)","CD1","CD2","CD3","FD","BD"}]

x0=0.32;
LogLogPlot[ {
h,
(h^2),
Abs[f'[x0]- CD1[x0,h]],
Abs[f'[x0]- CD2[x0,h]],
Abs[f'[x0]- CD3[x0,h]],
Abs[f'[x0]- FD[x0,h]],
Abs[f'[x0]- BD[x0,h]]
},{h,0.0000000001,0.01},
GridLines->Automatic,
Frame->True,
FrameLabel->{"h","Error"},
PlotLabel->"Error of FD Approximations TO f'[x]",
PlotLegends->{"h","(h^2)","CD1","CD2","CD3","FD","BD"},
MaxRecursion->4]

Second Derivatives

The following are all approximations for the second derivative

CDTwoDer1[x_,h_] :=(f[x+h]-2f[x]+f[x-h])/(h^2)
CDTwoDer2[x_,h_] :=((-f[x+2h]+16f[x+h]-30 f[x]+16f[x-h]-f[x-2h])/(12 (h^2)))

h=0.01
LogPlot[ {
(h^2),
(h^4),
Abs[f''[x]- CDTwoDer1[x,h]],
Abs[f''[x]- CDTwoDer2[x,h]]
},{x, -1,1},
GridLines->Automatic,
Frame->True,
FrameLabel->{"h","Error"},
PlotLegends->{"(h^2)","(h^4)","CDTwoDer1","CDTwoDer2"}
]

x0=0.234;
LogLogPlot[ {
(h^2),
(h^4),
Abs[f''[x0]- CDTwoDer1[x0,h]],
Abs[f''[x0]- CDTwoDer2[x0,h]]
},{h, 0.00001,1},
GridLines->Automatic,
Frame->True,
FrameLabel->{"h","Error"},
PlotLegends->{"(h^2)","(h^4)","CDTwoDer1","CDTwoDer2"}]

Order of Approximation

The order is the first power of h in the error. It is very easy to calculate on a computer.

First Derivative Approximations

Series[ f'[x]-CD1[x,h],{h,0, 4}]
Series[ f'[x]-CD2[x,h],{h,0, 4}]
Series[ f'[x]-FD[x,h],{h,0, 4}]
Series[ f'[x]-BD[x,h],{h,0, 4}]

Second Derivative Approximations

Clear[f,h]
Series[ f''[x]-CDTwoDer1[x,h],{h,0, 4}]
Series[ f''[x]-CDTwoDer2[x,h],{h,0, 4}]