The most common way of computing numerical derivative of a function at any point is to approximate by some polynomial in the neighborhood of . It is expected that if selected neighborhood of is sufficiently small then approximates near well and we can assume that .

Let’s consider this approach in details (or go directly to the table of formulas).

At first, we sample at the ( is odd) equidistant points around :

,

where is some step.

Then we interpolate points by polynomial of degree: . Its coefficients are found as a solution of system of linear equations:

By our assumption can be approximated by the derivative of the constructed interpolating polynomial:

To illustrate the process let’s consider case when . Here interpolating parabola are defined by the system (we assume for simplicity):

After subtracting first equation from the last we get final result:

In the same way we can obtain expressions for any . Formulas for are listed below:

These expressions are very widely used in numerical analysis and commonly refered as central(finite) differences. As it can be clearly seen they have simple anti-symmetric structure and in general difference of -th order can be written as:

,

where are coefficients derived by procedure described above.

It is easy to see that if is a polynomial of a degree , then central differences of order give precise values for derivative at any point. This follows from the fact that central differences are result of approximating by polynomial. If is a polynomial itself then approximation is exact and differences give absolutely precise answer.

To differentiate a digital signal we need to use h=1/SamplingRate and replace by in the expressions above. In this case derivative of a signal is found by

Frequency response of central differences is:

Magnitude responses for are drawn below:

Red dashed line is the magnitude response of an ideal differentiator . As grows moves closer to ideal differentiator response (by increasing tangency order with at ).

In practice there is no need for ideal differentiators because usually signals contain noise at high frequencies which should be suppresed. This means that response of practical differentiator should be close to on some passband interval and should tend to zero in the upper part of Nyquist interval .

From the plot we can see that central differences don’t resemble such behavior, all they care about is to get as closer as possible to the response of ideal differentiator, without supression of noisy high frequencies. As a consequence they perform well only on exact values, which contain no noise. Different technique is needed for robust derivative estimation of noisy signals. Such approaches are considered in the next sections:

## 74 Comments

#Hi Pavel, great tutorial!

How can this be done for the second derivatives f”(x) and also for the 2D case , especially how can we approximate d^2 f(x,y) / dxdy with different orders of approximation

Thank you!

#Thanks, Kem.

1. Second order central difference is simple to derive. We use the same interpolating polynomial and assume that . Final formulas are:

3. Third order central differences are:

2. Estimation of the mixed second order derivative is a little more elaborate but still follows the same idea. In this case we should build interpolating polynomial over the 2D grid and find its mixed derivative at the point assuming that it is approximately equal to .

Here are reference formulas for mixed second order differences:

For the order of approximation:

For the order of approximation:

#Pavel,

I just wanted to say how much i enjoyed finding this resource as i am taking my first course in numerical differential equations. I am having some confusion based on the definitions for the central difference operator that i am given and the one you are using. Could you post some of the steps taken to arrive at the finite difference approximation of: d^2 f(x,y) / dxdy for O(h^2).

Thank you,

Thomas

#Hi,

I wondered whether you followed just the same approach in deriving the result for . I once wrote a Mathematica script to compute central differences and get your results except for that case. I get:

Both schemes are correct in , you just have an additional term. I’m just curious why the results are not the same (I can send you the Mathematica script if you like).

Best,

Felix

#Felix,

I have used polynomial of degree 5 – it is minimally required to get the . Also I have used least-squares instead of interpolation.

In your case – you have used polynomial of degree 6 (and not all of the combinations of ). This doesn’t change the overall order of approximation, but error terms are (as they should) be different.

#Thanks! I actually used all combinations of such that the polynomial is of degree 8. It’s probably overkill. Thanks for the advice.

Best, Felix

#Well, 5×5 grid has only 25 degrees of freedom. Full polynomial of 6th degree has 28 coefficients – which is already overkill:) Polynomial of 5th degree (21 coeffs) provides a good trade-off.

#anyone know about the derivation of the third order central differences formula?

#Hi Pavel –

Do you have a published reference for the O(h^4) approximation for the mixed 2nd partial derivatives? Thanks.

#Here is formula for the order of approximation:

#Hi Pavel -

Thanks for your quick reply. I read that you had posted that same formula earlier … in fact, it’s the earlier post that prompted me to write. I’m looking for a published reference (book or journal article) that contains the formula so that I can cite the reference in a paper I’m writing. Thanks.

Best regards,

Chris Willy

#Well, I’ve derived this formula by myself. I have no idea is it published somewhere or not.

You can reference my site – it would be of huge support for me. Several published papers already include references to my site – so I think there is no problem.

You can tell me more about your task, maybe I can derive more suitable filters for your conditions.

#Hello Pavel,

Congratulations for your good and well organised work. I’m not mathematic and I’m writing an algorithm to derivate a discrete function. My set of points are not equidistant, so they have x values that are no constant. I read your advanced work about derivatives for noisy functions and probably I will use it in a future.

For the moment, I will derivate with central differences method. I wonder if it’s easy to extend the method when points are not equidistant. Is it ok to divide by the x increment?

Thanks for your blog

#Thank you for your comment!

Central difference extension for irregular spaced data is:

So, yes, difference of function values should be divided by the increment with weight. Coefficients are from formula for uniform spacing. For instance, for and for .

Actually it is better to use from smooth noise-robust differentiators, since they are much more numerically stable even in the absence of noise. Plus they have similar approximation properties to central differences – so you won’t lose any precision and get more stability as a bonus.

Coefficients are for and for . See referenced page for other .

Let me know about the results please.

#Hello again Pavel,

Thank you very much for your fast and good answer.

Really, I have a noisy discrete function (points come from peak calculation of a laser stripe, acquired by a digital camera. I use the laser stripe to enhance profile of objects).

I need to smooth the function for display purpose and I need also to derivate the function in order to find inflexion points, local maxima and other features that I use to do measurements of the object that I’ve enhanced the profile.

At this moment I’m smoothing the function using Savitzky-Golay filter and I apply central differences to the smoothed function. I read your work and I realized that smooth noise-robust differentiators would be really useful to my work, but I don’t know how to use it to smooth the function for display purposes.

#Thank you for the description of your project – it is really interesting.

I think your application could benefit from using SNRD filters. I will help you to apply them for smoothing (for visual output) and for finding derivative.

Just let me know what polynomial degree and number of points you are using now in Savitzky-Golay filter. Knowing that info I can build SNRD filters with better properties for your task.

#Hello Pavel, I appreciate your help very much.

I’m smoothing my function with a third degree polynomial and I’m using 7 points. Once smoothed I’m applying central differences to calculate first and second derivative.

I’ve tryed to calculate first derivative by using SNRD filters on the Savitzy-Golay smoothed functions, and I’m really loosing important function detail, I’m smoothing too much, I think this is not a good way to proceed.

Thanks again for your interest and best regards from Spain.

#Congratulations on the Victory!!

Thank you for the information about your test results.

I have several comments though.

I’ve checked properties of 3rd order 7 point SG and derive some special filters with better properties. I would appreciate if you would try them and give me feedback.

1. For your application I recommend to use following filters for smoothing (for visualization):

where coefficients are:

Both of them preserve important details better than SG and have soft and guaranteed noise suppression ( is stronger).

2. For derivative estimation I recommend to use

with

Important: do not apply differentiation filters after smoothing filters. This could hurt approximation order (as you experienced with SNRD after SG smoothing).Apply derivative filter directly on noisy non-smoothed data.

#Thank you for your info on central differencing!! For a homework assignment I am given a table (similar to yours) which gives us 3 and 5 point central difference (along with 2,3, and 5 point forward and backward) formulas, but then we are asked to determine the 4 point central difference formula from the table. I’m confused because I cannot find information on this anywhere online, is it possible to do a 4 point central divided difference approximation for a first derivative?? Any help would be greatly appreciated.

#Awesome tutorial.

I’m writing a GPU raytracer and need to estimate the gradient (∂/∂x, ∂/∂y, ∂/∂z) of a distance function (for surface normals). I assume that the function is expensive to evaluate.

Now I’m using central differences – these should get me O(h²) error with seven function samples (the center is always evaluated). I was wondering if I could get the same error order with fewer samples – some kind of sample sharing between the three axes (I’ve tried the tetrahedral arrangement, but it’s even worse than forward differences because of ∂x∂y-like terms).

#Thanks.

Actually 7-points central difference has . Three point stencil is enough to achieve .

To get the lowest error with the smallest number of points in 3D one should apply 1D stencil along the corresponding axis. Only points along the required axis play major role for approximation order, others can be used for other goals (like noise suppression).

P.S.

I like your site. Some of your projects made me nostalgic about old good DOS-hacking times. Trees were taller and grass was greener back then…….

#Thanks for the quick answer and for visiting my site :-).

What I originally meant was using the three-point stencil along every axis (which makes 6 evaluations in total + one in the center for other unrelated purposes).

I’ve finally settled for 4-point forward differences (1+9 evaluations in total with error O(h³)): .

#Oh, sorry for misunderstanding. Glad you have found suitable solution.

Maybe I should include one sided differences in the tutorial.

#What would be the two dimensional 7 and 9 point formulas for (d^2/dx^2 + d^2/dy^2))f(x,y)?

#What do you need them for?

#hey!

how can one prove that the third derivative of the central difference scheme of the function

f(x)=1/x with arguments a,b,c,d is given as

f”’(1/x)=(4ab(c-d)+4cd(a-b))/abcd

#Thanks for putting up these solutions. Do you have any opinion on the usefulness of Dispersion-Relation-Preserving finite difference schemes* for time dependent problems? If so, do you know of any site where coefficients are listed for various N-point stencils?

* (ie C. K. W. Tam, and J. C. Webb, “Dispersion-Relation-Preserving Finite Difference Schemes for Computational Acoustics”, Journal of Computational Physics, Vol. 107, 1993, pp. 262-281)

#It seems that paper is not available online for free. Could you send me pdf file or scan if you have it?

I found related papers of C.K.W. Tam:

http://www.math.fsu.edu/~tam/Multi-size-mesh%20DRP.pdf

http://www.math.fsu.edu/~tam/Tam_IJCFD_04.pdf

They look interesting – I’ll read them first and then get back to your question.

Thanks for the pointer.

#Sorry for the late reply.

After having read these papers I would say that DRP are almost the same to classical central differences.

The criterion authors use to construct DRP (minimum deviation from frequency response of ideal differentiators) is actually (near-)equivalent to the exactness on polynomials condition (which is the basement of CD).

I see no particular reason to choose DRP over CD. Actually I would suggest to use CD since they have rational coefficients, so rounding error is minimized.

#Hello,

I am trying to set up a FD equation of the form:

d/dx(a(x)*d/dx f(x))

I am looking for a five point stencil in particular.

This is basically the Poisson equation where the permitivity is not a constant.

Thanks for any help!

#What is the 9 point formula for the second derivative in one dimension?

#You can try 9 point stencils from here Noise Robust Second Derivative.

#Hi Pavel,

Thanks for this post, it was very insightful. I have not seen an in depth discussion of this topic past the N = 3 case.

I am somewhat confused on the technique you used to solve for the coefficients {c_k}. You give a general expression for the central difference based derivative approximation in terms of {c_k} and say “where {c_k} are coefficients derived by the procedure described above.” But I do not see a general technique above for finding the coefficients. I can derive the coefficients for a specific h & N by setting up the linear system Ax = b and finding the 2nd row of Pinv(A). I was curious how you derived the general form in terms of the variable h.

Thanks

#Hi KS,

Yes, “procedure described above” refers to the solution of Ax=b.

Usually differentiators are considered to have anti-symmetric coefficients.

That is what I meant under “general form” without mathematical rigor.

There is a simple algorithm on how to derive without solving linear system.

Generation of Finite Difference Formulas for arbitrary Spaced Grids. Bengt Fornberg

I think you might find it interesting.

#Thanks for the reference!

#Pavel,

thanks for the nice introduction!

For mixed second derivatives it may be more efficient to use an alternative formula:

Its advantage is that evaluating the mixed derivative along with the “diagonal” derivatives (d^2 f / dx^2 and d^2 f / dy^2) requires only 2 extra function values instead of 4.

Same idea can be used to derive formula accurate to :

Only 4 extra values are needed here.

Cheers!

#Ed, thank you for the nice formulas!!

Just checked, first one has , second .

Interesting how they behave in frequency domain – I’ll take a look on the transfer functions to check isotropy (important for 2D formulas).

Maybe I should start collecting all the formulas in one big compendium.

P.S.

You can insert math formulas into comments by using LaTeX syntax:

$ ... $ – for inline formulas

\[ ... \] – for displayed.

#Hi,

I just had a comment.

Isn’t the cross derivative Ed posted asymmetric (with respect to the diagonal terms)? It would seem to me that it might introduce some anisotropy into the calculation.

cheers,

#Mixed derivative should be symmetric along diagonal (depending on order of differentiation) – and Ed’s formulas are.

If you change the order of differentiation (dxdy – > dydx), then you have to “rotate” the formula so that isotropicity (with respect to “x” and “y”) is preserved.

#Pavel

Thanks for this interesting intro. I am not mathematical in any way and am trying to argue the case over the advantages of a 5-point derivative method over say a 3-point method.

My application is in using the first and second derivative of angular displacement data to yield velocity and acceleration. I have smoothed the displacement data, collected at 100Hz, with a forward and reverse 5th order butterworth filter (1/50 ‘low’) prior to calculating velocity and acceleration. However I am confused as to the advantages and disadvantages of the 3 and 5-point methods.

Thanks

#I think results of 3 and 5-points methods won’t differ very much after smoothing you’re using.

#Hi there,

Your website was very helpful! Thank you. My scientific application is a least-squares minimization in a large parameter space where some of the parameteric derivatives are known analytically but others are unknown. I used finite difference derivatives to estimate the gradient and diagonal elements of the Hessian, and I fill in the rest of the Hessian elements using BFGS.

For my application, I checked the three-point difference result against the seven-point difference result and got agreement to within five decimal places. This means that the three-point difference is usable in the actual application.

#Hello all, i’ve wondered if anyone can help me with the following:

I have a poisson equation: (d^2/dx^2+d^2/dy^2)=-1

Let’s say that around point U(i,j) i have 8 points as follows:

U(i-1,j-1), U(i+1,j+1), U(i,j+1), U(i,j-1), U(i-1,j), U(i+1,j), U(i-1,j+1), U(i+1,j-1)

How would the solution for U(i,j) be if DeltaX and DeltaY (finite differences) are constant but different (DeltaX not equal DeltaY)?

Thanks for the help

#hello,what is the discretization of Hessian term ([(∂^2 f)/〖∂x〗^2 ][(∂^2 f)/〖∂y〗^2 ]-〖((∂^2 f)/∂x∂y)〗^2)with 9-point stencil?

#great tutorial! It helps me with a better understanding of how an approx of derivative to an arbitrary accuracy is achieved.

#Your tutorial is greadfull.I want to know how explicit central differance method is apply to the wave equation

#please can you help with the fomular for the order of o(h)^3

thanks

#Dear Mr Pavel

Let me introduce myself. my name Lutfi mulyadi I come from Indonesia. I was very interested when I saw your blog. now I’m working on my BSc thesis in the field of nuclear reactor physics in Bandung Institut Of Technology, Indonesia. My thesis title is Fast Nuclear Reactor Burnup Analysis Using Fourth Order Runge Kutta Methods. in this thesis I solve the boundary value problem with the central finite difference and then the initial value problem with 4 Order Runge Kutta Methods. I have managed to create a program for the Runge Kutta method but I have not succeeded for the central difference to the neutron diffusion equation . if you can analyze the mistakes that I do? I really hope you can help me. a have sent to you my computer code in C programming language for solving neutron diffusion equations with central difference and I attach also my thesis to your email that include the numerical methods that I use in chapter 3 Simulation Methods (Numerical Methods) .

Thank you so much for your help

#HI

i want fifth order central difference (3 or 5 point).can anyone help me,my email:ghanbari.ashkan@yahoo.com

#Hi. I am looking for a formulation of backward finite difference in 2d. Can anyone help?

#Hi ! can the central difference formula be used even when the interval between x values is not constant? if not, what formula can be applied? thanks!

#Yes, it can, check this comment for formula.

#do you have some kind of history or concept for this central difference method?

#Very nice work Pavel!

I was just wondering whether you have formulated second order accurate central difference schemes for mixed third derivatives such as and and possibly forth derivative

I would greatly appreciate your help

#By applying the second order central difference schemes in turn, e.g. , I calculated the following formulae:

where i and j refer to the nodes in the x and y directions respectively and and are the grid spacings .

Are these correct? Are these second order accurate?

#Hi Amin, at first glance your formulas seems to be all right, however I haven’t check their asymptotics yet. Where do you use such high order derivatives?

#Thanks for the quick reply.

I’m looking at implementing an ADI method to solve the unsteady convection-diffusion equation in 2D and came across a paper on the topic by Karaa and Zhang*. After expanding the formulae used I came across the following difference operators , and where and are described as

I assumed these terms represent central difference approximations to , and respectively, hence I became interested in these high order derivatives.

*High order ADI method for solving unsteady convection-diffusion problems, Journal of Computational Physics (2004)

#This is a great blog everyone. An exceptional reference book for finite difference formulas in two dimensions can be found in “modern methods of engineering computation” by Robert L, Ketter and Sgerwood P. Prawel, Jr.. This is a 1969 book but it is a jewel. I have derived the 4 partial difference “molecule” for (∂^4 w)/(∂x^3 ∂y) but I am getting some numerical overflow problems in my computer model. Does anyone have this derivation? I used the Taylor series for i+1 and i-1 and took the partial derivative wrt x and y to obtain my answer. I think it is ok but I can’t find anywhere that shows that derivation.

Thanks, Pete

#Pete, thank you for your feedback!

Common reason for overflow (in differences) is cancellation error. You might try to increase h – sampling step.

If this won’t help – post the coefficients, it will be easy to check asymptotic properties.

#hi,i want to solve this differential equation with finite differences and cetered on bu i do not have any idea how to pleas help me

U”(x)=U^2(x) U(0)=0 and U(1)=1 (0,1)

u”=(u(i+1)-2*u(i)+u(i-1))/(h^2)

u’=(u(i+1)-u(i-1)/(2*h))

but i dont know how to solve and where i should start from please help me

#Hi! I just wish to note that for high order precision formula derivation you can use some polynomial expansions, which requires only integrability of the function.

e.g. : , where, for instance, is normalized Legendre’s polynomials, scaled over interval in question. If f is in it should be noted that . The inner products may be obtained from Gauss Legendre quadratures.

#Hi George!

Thank you for your comment.

There are many possible types of discretization to compute the derivatives.

We consider uniform sampling above.

However the best one are non-uniform (e.g. derived from Chebyshev or Legendre polynomials).

I love the Trefethen’s book on the subject: Spectral Methods in MATLAB.

Another useful read: Finite Difference and Spectral Methods for Ordinary and Partial Differential Equations

#I have a smooth function of n variables. At a certain point I have the value and all the first order order derivatives (df/dxi) and all the non-mixed second order derivatives (d^2f/dxi^2) for i = 1,2,…n. My function is expensive to calculate. Are there any methods to estimate which mixed second order derivatives are likely to be the most material so I can focus on them? thank you!

#Pavel: Thank you so much for your reply. Here are the coefficients I came up with. They seem logical to me but I am not sure how to verify.

for (∂^4 w)/(∂x ∂y^3) I get

[-w(j-2,k+1)+2w(j-1,k+1)-2w(j+1.k+1)+w(j+2,k+1)+w(j-2,k-1)-2w(j-1,k-1)+2w(j+1,k-1)-w(j+2,k-1)]/4∆x∆y^3

where the j are along the x axis and the k are along the y axis. the same process was repeated for (∂^4 w)/(∂x^3 ∂y) and the form was the same with obvious diffeences in the location of the grid points. Is this correct?

The reason I am using this higher order difference is for the numerical solution of a composite anisotropic plate where the bending and twisting coupling terms of the stiffness matrix are not equal to zero. Those stiffness coefficients are mutliplied by these two derivatives and go into the final set of equations [A]{w}={q}. Where the [A] matrix has these coefficients embedded in the solution. I have solved this same problem with the same number of grid points but assuming isotropic properties and I get within 1% of the analytical solution. But, of course, these two derivatives never come into play since for isotropic materials there is no bending -twisting coupling and those terms are zero.

Thanks so much.

Pete

#Hi Pete,

This formula looks a little strange to me. It is rectangular along the x (I rather expect it to be rectangular along y).

So I have derived my own formula for the case. Here it is:

(1)

It assumes h=∆x=∆y. Also I have omitted (j,k) to save the space.

This formula is of and looks more logical. Could you try it for your problem and let me know how it works?

#Thank you Pavel. I will try your solution and see what happens. It seems to be more than O(h^2) though. Which may be exactly what I need. Also my apologies when I submitted my derivation previously. What I submitted was (∂^4 w)/(∂x^3 ∂y) which is, of course, rectangular along the x. I tried my derivation a couple of different ways and I keep coming up with the same answer so you are probably correct in that I need to change my grid size. The simplest approach I tried was simply to take the derivative of (∂^3 w)/(∂x^3 ) wrt y or ∂/∂y ((∂^3 w)/(∂x^3 )). I know that my derivation for (∂^3 w)/(∂x^3 ) is correct so simply taking the derivative of it wrt y should give me the right solution and it seems to work. I hadn’t thought about encountering truncation errors so soon though. Thanks so much for your insight.

Best Regards, Pete

#For second order vibrational perturbationtheory (PT2) as discribed in http://arxiv.org/pdf/1306.4212.pdf are required the calculation of third and forth order numerical-derivatives. Namely dxdxdy, dxdydz, dxdxdydy.

Can anyone propose a formulas for calculation this derivatives.

#Pavel:

I just wanted to thaank you for this forum and for your responses to so many other researchers. Tthere are many questions that just can’t be answered in books or papers. Many times difficulties with the mechancs of the detail derivations can put bring your work to a halt. It is always of benefit to be able to talk to colleagues with like interests. I used your derivation and I was still getting the overflow problem. I was at a complete loss and then I decided to normalize my equation solver (Gaus-Jordan Reduction) with repespect to my largest stiffness coefficient. The problem works beautifully now. Once again, thank you professor Holoborodko!

Pete

#Dear Pete,

Thank you very much for your supportive words and feedback!

And I am glad you managed to overcome numerical difficulties in your solver.

I don’t know, but maybe you can use automatic differentiation? It is usual choice when stencils lead to overflow.

(btw I have no professor degree ;)).

#Hi….

i want to derive formula of 5th order derivative for finite difference scheme..there are some issues …help me in finding this..any body knowing this please share with me…thanks for supporting me..

Best Regards,

Sami

#Dear Pavel Holoborodko

Can you find a representations for higher derivatives for the function f

f_xxxx, f_yyyy, f_xxxy, f_xyyy and f_xxyy

#Dear Amr,

Many people asked the same question in the comments. I think it is time to do this and collect everything in one reference.

I plan to derive & publish the formulas over next few days.

#Thank you Pavel Holoborodko

I try to get an expansions for the derivatives and by using Taylor series for a function in two variables .

I solve a matrix equation . But the problem is is a singular matrix. I don’t know what I do.

#Hi Pavel,

I’ve been working with the Transonic Small Disturbance Equation using a non-uniform grid, and am having issues deriving a second order central difference equation to use. Any guidance on where to go would be fantastic.

I understand that a first order is

so would that make a second order simply

Thank you very much!