Set up benchmarks server

[datapythonista]

This issue is to keep track of the benchmarks project we are about to start. Adding context and some initial ideas, but we'll keep evolving the project as we make progress.

The final goal of our benchmarks system is to detect when some functionality in pandas starts taking longer for no good reason. See this fictional example:

>>> import pandas
>>> pandas.__version__
2.0
>>> data = pandas.Series([1, 2, 3])
>>> %time data.sum()
1 ms

>>> import pandas
>>> pandas.__version__
2.1
>>> data = pandas.Series([1, 2, 3])
>>> %time data.sum()
5 ms

In the example, Series.sum for a given data takes 1 millisecond in 2.0, while in 2.1 it takes 5 times that time, probably because an unexpected side effect, not because we made some changes we consider it was worth making the function 5 times slower. When we introduce a performance regression like the one above, it is likely that users will end up reporting it via a GitHub issue. But ideally, we would like to detect it much earlier, and never release versions of pandas with those performance regressions.

For this reason, we implemented many functions that run pandas functionality with some arbitrary but constant data. Our benchmark suite, the code is in asv_bench/benchmarks. Those are implemented using a framework names asv. Running the benchmark suite gives us the time it takes to run each of many pandas functions with data consistent over runs. If comparing the executions with two different pandas versions, we can detect performance regressions.

In an ideal world, what we'd like is to detect performance regressions in the CI of our PRs. So, before merging a PR we can see there is a performance regression, and we make a decision on whether the change is worth making pandas slow. This would be equivalent to what we do with tests or with linting, where we get a red CI when something is not right, and merging doesn't happen until we're happy with the PR.

In practice, running consistent benchmarks for every commit in a pull request is not feasible at this point. The main reasons are:

• Our benchmarks take around 7 hours in decent hardware
• The CI doesn't use decent hardware, it uses small workers, where it would take way longer
• To compare benchmarks we need consistent hardware. We could run in the same worker the benchmarks for main and the PR of interest, that would double the execution time
• Virtual hardware, like a GitHub CI worker introduces a lot of noise. The time to execute a function will depend as much as how busy is the rest of the physical host than the time our implementation takes

For now, we are giving up on executing the benchmarks for every commit in an open PR, and the focus has been on executing them after merging a PR. When running the benchmarks, we run them in a physical server, not a virtual server. The server we've been using was bought by Wes many years ago and was running 24/7 from his home, and at some point it was moved to Tom's home. This is still how we run the benchmarks now. Some months ago, OVH Cloud donated to pandas credits to use a dedicated server from their cloud for our benchmarks (we also got credits to host the website/docs and for more things we may need). There was some work on setting up the server and improving things, but we didn't complete even the initial work.

There are three main challenges to what otherwise would be a somehow simple project:

• Having stable benchmarks is very hard
• Our benchmarks suite takes a very long time
• ASV is not as good as we would like. The codebase was unmaintained for several years, and while we put a lot of work more recently, the codebase is complex and not easy to deal with, and the UX not always intuitive.

The most common approach for benchmark stability is what it's usually called statistical benchmarking. In its simplest form, the idea is for a single benchmark run, running the function to time something like 100 times and taking the mean. ASV does a slightly smarter version of it, where the first few runs (warm up) are discarded, and the exact number of times a benchmark runs depends on the variance of the first runs. But 100 repetitions is common.

This repetition brings more stability, but obviously makes the second challenge worse, as timing every function 100 times makes benchmarks 100 times slower. We have a CI job where we run our benchmark suite with just one call for each function, and the timing is very reasonable, the job takes 25 minutes in the CI worker. But the results are discarded, since they are very unstable both because the lack of repetition, and the instability of using a virtual machine / CI worker.

For now, what we want is:

• Configure our dedicated server to be as stable as possible. An amazing resource to better understand the problem and possible solutions is Victor Stinner's blog series. Also this page: https://pyperf.readthedocs.io/en/latest/system.html
• Find a way to run benchmarks for each commit to main. The existing server uses the code in https://github.com/asv-runner. There are many options here, but anything running in the server affects its stability. We could implement a web server that receives webhooks from github for each commit, but we can only run the benchmarks once at a time, and I'm not sure if the activity of the web server would be significant or not for the benchmark results
• Find a way to run the benchmarks faster than adding them to the queue. If the benchmarks take 6 hours, we can run 4 times the suite per day, it's likely that we merge more than 4 PRs per day, so the queue would keep growing and over time our last executed benchmark would be very far from the recent commits. If we can run the benchmarks for each commit that would be amazing, it may require reducing the amount of data used to test the slower benchmarks, cap the number of repetitions, or other solutions. If we need to skip commits, we may want to do that in a known and consistent way.
• Make sure that results are meaningful and useful. The results are currently available here: https://asv-runner.github.io/asv-collection/pandas/. ASV is able to run the benchmark suite with different environments. This means that for a specific pandas commit it can run for a matrix of Python version, Cython versions, NumPy versions... I'm not sure if there is a clear decision on how we'd like to benchmarks things regarding the environments, but I personally find it quite hard to understand the results easily. See for example: https://asv-runner.github.io/asv-collection/pandas/#algos.isin.IsIn.time_isin_mismatched_dtype. Even when filtering all but one result, results aren't obvious. There are like 50 results with their commit hash, but not easy to tell the dates of these measurements. Is ASV just keeping the last 50 measurements? Are there commits that didn't run? I guess most of the problem is in ASV itself, but since other projects are using it, I guess we can set it up in a way that it's more useful.

Once we have a reasonable version of the above, some other things can be considered. Some ideas:

• Implement more complex benchmarks (benchmark whole pipelines, not only individual functions). TPCH for example, as Polars do
• Consider ASV alternatives. I did some research on this, and there is no obvious too that we should be using instead of ASV, but ASV will be a blocker for any innovation
• Anything that can be helpful towards the final goal of early detection of performance regressions.

CC: @DeaMariaLeon @lithomas1 @rhshadrach

[rhshadrach]

When running the benchmarks, we run them in a physical server, not a virtual server. The server we've been using was bought by Wes many years ago and was running 24/7 from his home, and at some point it was moved to Tom's home. This is still how we run the benchmarks now.

I now run the ASV machine from my home. It is my personal hardware (dedicated to doing nothing but running benchmarks).

Find a way to run the benchmarks faster than adding them to the queue. If the benchmarks take 6 hours, we can run 4 times the suite per day, it's likely that we merge more than 4 PRs per day, so the queue would keep growing and over time our last executed benchmark would be very far from the recent commits.

The way I inherited it from Tom, we grab the last 5 commits and run benchmarks on those commits if benchmarks have not already been run yet. The benefit of this is that we don't have an ever growing queue. The downside is that the machine may have no new commits to run and does not look further back in history so may unnecessarily sit idle.

I've been slowly working toward automating the process of reliably detecting regressions and making/tracking comments on PRs. Currently I'm building a front end in dash. I don't think this competes with the work as outlined here; I believe both setups are good to have as they each have strengths and weaknesses.

[lithomas1]

I had something to be able to trigger a runner on OVH to be spun up on Github Actions with cirun and run some of the tslibs benchmarks, there was too much variability (>20%) too.

Here is my branch.
https://github.com/lithomas1/pandas/tree/cirun-test

Maybe we should try to set up a meeting with everyone that's interested so we can figure out what we have, and what needs to be done?

[rhshadrach]

@lithomas1 - do you know if that was a problem with the setup, or a problem with the benchmark? E.g. at least at face value this appears to me to be a problem with the benchmark:

https://asv-runner.github.io/asv-collection/pandas/#arithmetic.Ops2.time_frame_dot

[lithomas1]

I think I was just experimenting with a standard cloud VM (b2-7 from OVH) to see how bad the noise was.

I didn't tune the system at all back then and we weren't using the bare metal server, so maybe there's room for improvement.

I was mainly sharing to show a possible way to connect Github with whatever we use to run the benchmarks.

[DeaMariaLeon]

I'm trying to understand the current workflow (for the benchmark suite).

We have a CI job where we run our benchmark suite with just one call for each function, and the timing is very reasonable, the job takes 25 minutes in the CI worker. But the results are discarded,

Are the results discarded after running asv in
code-checks.yml ? (as noted above).

Is this up to date? Benchmark-machine

BTW, thanks @lithomas1 and @rhshadrach for the info. I have been getting familiar with asv and the server, that's why it took me a while to comment.

[rhshadrach]

Are the results discarded after running asv in
code-checks.yml ? (as noted above).

Correct.

Is this up to date? Benchmark-machine

For the most part. We now use instead https://github.com/asv-runner rather than Tom's repo as mentioned there. We also no longer publish from the asv-collection repo to the pandas website. The benchmark machine is running in my home and has no access to the wider internet, so only I can "debug" (read: reboot) it.

It is still written in ansible. I plan to rewrite it using Docker.

[datapythonista]

Thanks @rhshadrach and @lithomas1 for the information, this is helpful.

The way I inherited it from Tom, we grab the last 5 commits and run benchmarks on those commits if benchmarks have not already been run yet. The benefit of this is that we don't have an ever growing queue. The downside is that the machine may have no new commits to run and does not look further back in history so may unnecessarily sit idle.

Is this a daily cron job? If there are 5 commits for that day, do you know how long does it take? Is it possible that if it takes more than 24h (more than 4.8 to run the benchmarks of each commit) the process is still alive when the execution of the next day starts? I guess that would add a decent amount of extra noise.

We're finishing setting up things, I need to see how benchmarks take exactly in the server, but I was thinking on running the last commit from the repo twice or 3 times per day, depending on what's possible.

And then, in parallel and in a separate server, keep running the benchmarks for the same commit again and again, and test changes in the machine, asv params or whatever to see what make things more stable and faster.

I think there are other things that can be done, like:

• Improve the asv interface
• Have a thought and set up a frozen environment (ping all dependencies in asv.conf.json), so benchmarks keep consistent and easier to read (see here for example how the changes to cython or python versions make things quite complicated to follow). Or not sure if people see more value on testing new version of dependencies as soon as they are released. We could possibly have both too, in different servers and asv websites, but I'm not sure if the was asv is displaying things now is the most helpful, I would possibly not using asv environments and just have a single line in the plot for "latest version of everything"
• Detect regressions and send notifications, maybe create issues on github if something is detected?

And I guess other things. Does anyone in the @pandas-dev/pandas-core team has anything they'd really like to see improved in the benchmarks that it's doable? As you know, @DeaMariaLeon will spend the next couple of months working on this, besides the work of others. We can surely have a meeting as suggested above, maybe we can gather some feedback here first, and have the meeting in couple of weeks?

[rhshadrach]

It is not a cron job. Once a set of commits is finished, the machine will pull from main and run the 5 latest commits, only running those that have not been benchmarked already. If there are no new commits to benchmark, it will sleep for 10 minutes and try again.

[datapythonista]

We not got the benchmarks automatically being published here from our benchmarks server: https://pandas.pydata.org/benchmarks/

So far it uses a cron job that every 3 hours executes the last commit (the benchmarks take a bit more than two hours in the server). This is just a first step, we can discuss and implement things in any other way that it's more convenient.

I also set up another benchmarks server, but that keeps running the benchmarks for the same commit (same exact code, but with different hash). This is to see how much noise we've got in the server,. For certain benchmarks seems to be a lot. We'll be setting up the server to be more stable (cpu isolation...). This other server should help see if what we change really has an effect: https://pandas.pydata.org/benchmarks/same-commit/

All is work in progress, but wanted to share those already, in case it's useful.

[rhshadrach]

On the benchmarks call today, the issue of parsing the current ASV json files into something more usable (e.g. a pandas DataFrame) came up. Wanted to share what I use today:

https://github.com/rhshadrach/asv-watcher/blob/10cec8096b5bb1d1ef7ef4b320bf8b2db482c98b/asv_watcher/_core/update_data.py#L19

[jorisvandenbossche]

On that topic, here is an older notebook where I convert the json file to a DataFrame and did some basic analysis (listing the ones that take the most time): https://nbviewer.org/gist/jorisvandenbossche/0af1c0a20ef187197ecdcfdb3545306a (from #44450 (comment))

[datapythonista]

I created a repo with what we've got so far in the OVH servers: https://github.com/pandas-dev/pandas-benchmarks

We're still quite far from having enough stability in the servers to be happy, but I'll keep updating that repo as we make progress.

So far we're going to continue with the next tasks:

• Build a PoC of conbench for pandas (@DeaMariaLeon is already working on it)
• Implement a smarter way to pick which commits to run (instead of leaving the server idle, look for the commits that add more value to have benchmarks for, for example the middle commit in a long gap in our recent history)
• Reduce the noise in the server (CPU frequency scaling, CPU isolation...). Some work has been done already, but results seem far from good

We will consider the other proposed topics after we get some progress in the above tasks.