Cratebank

An unofficial and ongoing census of Rust compilation times

We all want Rust to compile faster. So then, why are throwing out the data that might hold the key to making that happen? Cratebank collects information from developers and organizations about Rust compilation times, processes the data so that it is easy to work with, then publishes datasets that anyone can use for free. People volunteer this information with the hope this data can be used to improve Rust generally and Rust compile times specifically.

How To Get Started

It's as easy as installing two crates and then running the right command.

cargo install cargo-cratebank
cargo install samply          # the profiler that measures compiler phases

cargo cratebank build         # builds, measures, and sends compile time info

On many Linux distributions samply needs permission to read perf events, once per boot:

sudo sysctl kernel.perf_event_paranoid=1

For automatic sending on Mac and Linux, add this to your ~/.zshrc or ~/.bashrc:

cargo() {
    if [[ "$1" == "build" ]]; then
        shift
        command cargo cratebank build "$@"
    else
        command cargo "$@"
    fi
}

For Windows, run notepad $PROFILE and add this: 

function cargo {
    if ($args.Count -gt 0 -and $args[0] -eq "build") {
        $rest = @()
        if ($args.Count -gt 1) {
            $rest = $args[1..($args.Count - 1)]
        }

        & cargo.exe cratebank build @rest
    }
    else {
        & cargo.exe @args
    }
}

Every cargo build then runs under the sampler and contributes an observation. If measurement or the upload fails — an offline laptop, a collector having a bad day — cratebank says so and your build still succeeds; once your code has compiled, nothing here changes the exit code. Windows sampling has not been verified end to end yet, so treat the PowerShell function as experimental.

We also publish a Github Action that you can use as well as part of your continous integration testing.

What gets sent

Nothing wraps rustc, so there is no conflict with sccache or any other RUSTC_WRAPPER. The data is only sent when you invoke cargo cratebank build. To see exactly what would get sent in your repo, run cargo cratebank --dry-run build.

The client also tells you when a newer one is out, and never refuses an old one -- your data is welcome either way.

Reading what you sent

Contributing prints a link to the observation it just uploaded, and that page shows every table in it: what the machine was, how long the build took, and then every row, value by value. Nothing is summarised away.

Read any observation -- every upload waiting to be merged, as a tree you can open rather than a path you have to walk.

Asking the data something

Explore the census runs DuckDB in your browser against the published parquet, and puts the answer in a pivot table you can regroup, filter and plot without writing a second query. Nothing is uploaded and no query leaves the tab -- the files are read over HTTP, a footer and the columns a query touches, which is why it starts in about a second on a table of millions of rows.

The same files are at https://data.cratebank.io/ if you would rather point your own DuckDB at them.

The raw uploads are published as well as the processed tables, so nothing here is the only copy: every object is world-readable under https://data.cratebank.io/sessions/, and the viewer links to the bytes it read.

Using the data

Everything is public parquet on R2 — no account or API key needed. You can use these parquet files however you like! We provide some scripts to make it easy to use with DuckDB.

Install persistent views directly from GitHub, then open DuckDB:

curl -fsSL https://raw.githubusercontent.com/PowderworksCode/cratebank/main/docs/install.sql | duckdb cratebank.duckdb
duckdb cratebank.duckdb

Once the views are installed, queries are ordinary SQL:

SELECT package, phase, sum(samples) AS samples
FROM phases 
WHERE thread = 'serial'
GROUP BY 1, 2
ORDER BY samples DESC;

Public tables, described by a machine-readable schema:

schema/v1/tables.json describes every column, and carries the warnings that matter — chiefly that sampled phases are CPU while the section boundaries in units are wall clock, and the two are not interchangeable.

Licensing

The code is MIT licensed. The public dataset is licensed under CC BY 4.0. Credit “cratebank contributors,” link to this site and the license, and say if you changed the data. By uploading an observation, you confirm that you are contributing it under CC BY 4.0.

Contact

Zack got tired of waiting on Rust to compile, so he made Cratebank as part of his open source work on The Powderworks Agentic Code Consortium. The best way to contact him about this is via the Powderworks Zulip.

Source on GitHub · opt-in · data: CC BY 4.0