Description
I have a problem with a quite simple scalar function, of which I also need the derivative together with its value. This evaluation of value and derivative is called many times (simulation). Now, I could come up with the explicit derivative of this function. But there's many such functions (so I don't want to do this for all of them) and in the end, ForwardDiff should be more efficient because it can calculate value and derivative in one pass instead of calls to two functions.
AFAIK there is no simple / straightforward way of doing something like
val, der = ForwardDiff.derivative(f, x)right? (Also see request in #391.) Here, the focus is not on syntax only, but on the fact that also the value is actually calculated, but not directly returned. To extract value and derivative of a single pass, one would have to use DiffResults. But I found that there's a huge overhead with that, expecially for very simple functions. I did some benchmarking. Using the polynomial function (similar to what I deal with):
using ForwardDiff
using DiffResults
# function and explicit derivative
f(x) = 0.1 + (0.5 + (0.7 + (0.2 - 0.05*x)*x)*x)*x
f′(x) = 0.5 + (1.4 + (0.6 - 0.20*x)*x)*xI define three versions of calculating value and derivative, actually providing the simple syntax outlined above:
# value and derivative in one pass, using DiffResults
function val_der(f, x)
res = DiffResults.DiffResult(x, (x,))
res = ForwardDiff.derivative!(res, f, x)
return (DiffResults.value(res), DiffResults.derivative(res))
end
# value by function call, derivative by ForwardDiff
val_der_sep(f, x) = (f(x), ForwardDiff.derivative(f,x))
# value and (manually derived) derivative by function calls
val_der_expl(f, f′, x) = (f(x), f′(x))Then, for the benchmarks:
using BenchmarkTools
x = 3.4
println("f(x):")
@btime f(x)
println("f′(x):")
@btime f′(x)
println("val_der(f,x):")
@btime val_der(f, x)
println("val_der_sep(f,x):")
@btime val_der_sep(f, x)
println("val_der_expl(f, f′, x):")
@btime val_der_expl(f, f′, x)On my machine, the results are (best of 3 runs for each one):
f(x):
14.407 ns (1 allocation: 16 bytes)
f′(x):
14.406 ns (1 allocation: 16 bytes)
val_der(f,x):
278.916 ns (6 allocations: 144 bytes)
val_der_sep(f,x):
15.515 ns (1 allocation: 32 bytes)
val_der_expl(f, f′, x):
16.346 ns (1 allocation: 32 bytes)
Interpreting this:
- There's quite some benchmarking overhead compared to the actual function evaluations.
- Explicit function for the derivative and using
ForwardDiff.derivativeare very close. Any idea how to more accurately benchmark those against each other? - The solution using
DiffResultsis a very bad idea for such tiny functions.
So, is there a way to also return the value (or, on request / by macro / etc., all intermediate derivatives calculated anyways) on the low-level / internal API of ForwardDiff?