Why does a function run faster the more I call it? - benchmarking

Question

I'm trying to time the execution of a function before attempting to optimize it. (The code is Elixir, but I'm using Erlang's :timer.tc.)
My general approach is "run it lots of times, then calculate the average duration." But the average decreases dramatically the more times I run it (up to a point).
An example:
some_func = fn ->
# not my actual function; it's a pure function,
# but exhibits the same speedup
:rand.uniform()
end
run_n_times = fn (count, func) ->
Enum.each(1..count, fn (_i) ->
func.()
end)
end
n = 20
{microseconds, :ok} = :timer.tc(run_n_times, [n, some_func])
IO.puts "#{microseconds / n} microseconds per call (#{microseconds} total for #{n} calls)"
Outputs for increasing values of n are like this (lightly formatted):
174.8 microseconds per call (3496 total for 20 calls )
21.505 microseconds per call (4301 total for 200 calls )
4.5755 microseconds per call (9151 total for 2000 calls )
0.543415 microseconds per call (108683 total for 200000 calls )
0.578474 microseconds per call (578474 total for 1000000 calls )
0.5502955 microseconds per call (1100591 total for 2000000 calls )
0.556457 microseconds per call (2225828 total for 4000000 calls )
0.544754125 microseconds per call (4358033 total for 8000000 calls )
Why does a function run faster the more I call it, and what does this imply for benchmarking? Eg, is there a rule of thumb like "run something >= 200k times in order to benchmark"?

Answer 1

Since your function is very fast (does nothing basically) what I think you're seeing here is the overhead of the setup and not any speedup in the runtime of the function. In this case before you start running your function you have to construct a range, construct an anonymous function and call the Enum.each function. For small numbers of repetitions these factors probably contribute more to the overall runtime of the benchmark than the actual repetitions.

Answer 2

I second what Paweł Obrok wrote in his answer. You could optimize your code by calling the function multiple times inside the loop:
run_n_times = fn (count, func) ->
Enum.each(1..count, fn (_i) ->
func.()
func.()
func.()
func.()
func.()
func.()
func.()
func.()
func.()
func.()
end)
end
That's 10 calls but you could 100 or 1000 of them. The more you do in the same loop the less will be the overhead.

Answer 3

I don't know what exactly Erlang does, but if you do the same in Javascript with a modern Javascript interpreter, then the first few calls will be interpreted (slow). Then the interpreter figures out that you are calling this function a lot and compiles it with a quick and dirty compiler. Another hundred calls, and the interpreter figures out what's happening and compiles it again, with a proper compiler this time. And another thousand calls, it gets compiled again with a highly optimising compiler. That would give exactly the kind of numbers that you found.