DuckDB ·
A DuckDB Release Isn't a Distribution
When DuckDB ships a new version, every community extension has to be rebuilt against it before anyone can install one. I measured how much of the catalog is actually there when a release lands, how long extension updates take to merge, and why the same builds run 2.5x faster on different CI. The answer to all three is distribution work.
DuckDB ships a great engine. What it doesn’t ship is a distribution.
The engine and its community extensions are built and released separately, and every extension binary is compiled against one specific engine version. When the engine moves, the whole catalog — 293 extensions as I write this — has to be recompiled and re-published before anyone on the new version can install any of it. That includes point releases. v1.5.4 to v1.5.5 obsoletes every binary in the catalog.
This is the same split Linux has between a kernel and a distribution, and it produces the same class of problem. Almost nobody runs a raw kernel in production; they run Ubuntu, because somebody did the work of building, testing, and packaging everything around the kernel into a coherent thing that ships as a unit. DuckDB has no Ubuntu.
I spent a while measuring what that actually costs, using Haybarn — Query.Farm’s distribution of the DuckDB community extensions — as the comparison case. Three findings, and they all point the same direction.
Almost nobody runs a raw kernel in production. They run Ubuntu. DuckDB has no Ubuntu.
1. When a release lands, most of the catalog isn’t there yet
The obvious question is: at the moment DuckDB announces a version, how much of the extension catalog can you actually install?
I measured it directly rather than guessing. For every extension in the catalog I asked the official extension repository whether a linux_amd64 build exists for a given engine version, and read the deploy timestamp off it, then compared that against when the engine was released.1Last-Modified reflects the most recent deploy of a file, so an extension re-published later looks later than it first appeared — the “at release” numbers are lower bounds. The denominator is the catalog as it stands today, which dilutes older versions with extensions that didn’t exist yet, and includes roughly one in eight that build for no recent version at all.
5 days old
40 days old
68 days old
linux_amd64 build published for each engine version, measured 27 July 2026 against the official extension repository. Bars are percentages of the whole catalog, so they never reach 100%.The full curve, including the intermediate points:
| Engine version | Released | At release | +3 days | +7 days | Today |
|---|---|---|---|---|---|
| v1.5.5 | 2026-07-22 | 52% | 60% | 60% | 60% |
| v1.5.4 | 2026-06-17 | 15% | 47% | 56% | 84% |
| v1.5.3 | 2026-05-20 | 35% | 46% | 55% | 70% |
Two things jump out of that table, and one of them is good news.
The good news: upstream now builds ahead of the tag. For v1.5.5, duckdb/community-extensions ran its full-catalog sweep against the engine’s pre-release commit — and I checked, the v1.5.5 tag points at exactly the commit they built — finishing roughly two and a half hours before the release was published. That’s why 52% of the catalog was installable the instant the release appeared, against 15% for v1.5.4 five weeks earlier, where the median extension landed about 60 hours after the release.
That is a real improvement and it deserves to be called out, because it’s precisely the “engine and extensions are jointly ready” moment I’d argue a distribution should provide. Building the catalog before the tag is the right move and I’d like to see it become the norm.
The remaining problem is completeness, not start time. Five days after v1.5.5 shipped, 60% of the catalog has a build — 176 of 293 extensions — and it has stopped moving. v1.5.4, now 40 days old, crawled from 15% to 47% to 56% and has reached 84%. The tail is long and it never quite closes.
The gap isn't the first few hours after a release. It's the tail — and the tail is where the extension you actually depend on lives.
So the honest shape of this is not “hours” but days to weeks, with a tail that never fully closes. And that’s the part that bites: it isn’t the first few hours after a release that decide whether you can upgrade, it’s whether the specific extension your application needs is in the 60% or the 40%. Which is unpredictable per extension, which means you can’t tell your own customers when it’s safe.
2. Between releases, merge times are drifting
Extensions don’t only change when DuckDB does. Authors ship fixes and new versions continuously, each as a pull request to duckdb/community-extensions, and an update isn’t usable until it merges, rebuilds, and publishes. So the everyday latency of that pipeline is the other half of the predictability story.
I pulled every PR merged there in the trailing year — 1,593 of them, 1,148 being extension release or version PRs — and measured merged_at − created_at by month.
duckdb/community-extensions, July 2025 through July 2026. The solid column is the median, the pale column behind it the p90, both on the same scale in hours — so the pale portion is the tail. For reference, the 30 h gridline is a day and a quarter; the top of the axis, 120 h, is five days. The outer two months are partial windows and are excluded from the trend read. Hover any column for its exact figures and PR count.Across complete months the median rose from about 5.5 hours to about 20.1 hours, and the p90 widened from 27 to 113 hours. Comparing the first half of the window against the second half — a steadier read than any single pair of endpoint months — the median roughly doubled, 5.7 to 12.3 hours, with the p90 going 27 to 50.
I want to be careful about what that does and doesn’t say. It is emphatically not a knock on the maintainers, who are notably responsive; for most of the year a typical PR merged inside a working day. It’s also end-to-end latency, so it includes author round-trips and CI, not just maintainer wait — an upper bound on the human part. The point is the direction and the spread, both of which track submission volume growing several-fold over the same window. Throughput here is ultimately bounded by maintainer attention, which is finite and batched into working hours and time zones.
3. The same extensions build 2.5x faster, and it’s all caching
Here’s the part that surprised me most when I first measured it. On identical GitHub-hosted runners, through the same build.yml → _extension_distribution.yml matrix, the same extensions build substantially faster on Haybarn’s CI. I compared 14 widely-used extensions across both — 1,325 Haybarn build legs against 1,360 upstream ones — and the gap is largest exactly where you’d predict.
(MinGW)
Community extensions are mostly thin wrappers over heavy C/C++ libraries pulled in through vcpkg. So the dominant cost of a build isn’t compiling the extension — it’s compiling those third-party dependencies. Whether you pay that cost comes down almost entirely to the build cache, and that’s the whole story of the chart above: the ratio is 1.5x on macOS, where there’s little to cache, and 3.5x on Windows-MinGW and the WebAssembly triples, which are the dependency-heavy, miss-prone targets.
What’s actually different about the caches
I want to be precise here, because the obvious version of this claim is wrong and I had it wrong myself in an earlier draft.
duckdb/community-extensions points vcpkg at the shared cache vcpkg-cache.duckdb.org, and extension builds read from it but don’t write to it — save_cache is enabled only for a single canary extension, and only outside pull requests. That’s a sound default for a repo that builds untrusted contributions: a PR can never poison the shared store.
But it would be wrong to say that cache can’t be populated. It can, and it is: a scheduled job pre-compiles a curated per-platform dependency list into it every night.2cache_warming.yml, plus the generated dependency list under .github/config/vcpkg_caching/. It has been in place since mid-2025. Widely-shared dependencies are generally warm. Credit where it’s due.
The real difference is that upstream’s warming is ahead-of-time and partial, rather than driven by what the builds turn out to need. The warmed list is a curated union, and it’s thinnest exactly where builds are most expensive: each WebAssembly triple gets a dozen entries against twenty to twenty-seven for the native targets, and the companion ccache warming job excludes windows_amd64_mingw altogether. Anything outside that list — a dependency only one extension pulls in, a new compiler or triple — gets compiled on the build leg and then thrown away.
Haybarn’s cache is read-write and estate-global: a dependency is compiled once, written back on success, and reused by every later build across the whole catalog. It’s a Cloudflare Worker in front of R2 object storage, reads anonymous, writes authenticated. The asset cache doubles as a mirror with origin fallback, so a flaky upstream tarball download stops failing builds — reliability, not just speed.
A dependency compiled for one extension should be available to every other one. That single sentence is most of the 2.5x.
I’ll flag the limit of this explanation: the partial-warming difference is documented and real, but I haven’t proven it accounts for the entire gap. Attributing it precisely would need per-platform cache hit rates, which I haven’t measured yet.
One extension, step by step
To make the mechanism concrete, here’s a5 — deliberately the floor case, a small Rust extension with almost no native dependencies, so the dependency cache barely helps and what’s left is mostly fixed per-build overhead.
a5 linux_amd64 build leg, seconds per step group, on each CI. The build-environment container dominates: rebuilt inline on the DuckDB Linux leg, pulled pre-built from GHCR on Haybarn. Sampled from one run on each side; the DuckDB Docker step ranged 227–288 s across the two runs GitHub still has step timings for.The single biggest line is the build-environment container. On its Linux legs DuckDB rebuilds the Docker image inline; Haybarn pulls a pre-published one from GHCR. That’s pure fixed overhead, paid on every Linux leg of every extension, and eliminating it also shortens the critical path of a whole sweep rather than just its compute.
Then look at the compile line: 46 s against 72 s, on an extension with essentially no dependencies. Even a5 still links against DuckDB core and its static libraries, which Haybarn reuses from cache instead of recompiling. A no-dependency extension already shows a gap; dependency-heavy ones show far more.
Two smaller details make the remote cache possible at all. Haybarn ships ccache 4.13.6, because older builds lack the bearer-token authentication needed to talk to a remote HTTP cache — without it ccache simply can’t use a shared backend. And DuckDB’s compiler cache is GitHub Actions’ built-in actions/cache: per-repository, branch-scoped, LRU-evicted around 10 GB, which structurally cannot share compiled objects across hundreds of independent extension builds.
What it adds up to across the estate
A per-extension speed-up is interesting. The estate-wide number is the point, because shipping a new engine version means rebuilding everything, on every platform.
I measured one full Haybarn sweep — 210 extensions, the v1.5.4-rc1 rebuild — and modeled the DuckDB-equivalent cost by scaling each platform’s measured Haybarn compute by that platform’s observed ratio. That’s a projection of the chart above, not a second measurement, and I’d rather label it as such: DuckDB doesn’t do single-shot full-catalog rebuilds the same way, so there’s no like-for-like sweep to measure.
| Platform | Legs | Haybarn (h) | DuckDB-equivalent (h) | Ratio |
|---|---|---|---|---|
| Windows x64 (MinGW) | 128 | 19.8 | 69.4 | 3.5× |
| Windows x64 | 160 | 17.7 | 42.4 | 2.4× |
| Linux x64 | 202 | 17.2 | 36.1 | 2.1× |
| Linux ARM64 | 201 | 15.6 | 29.6 | 1.9× |
| macOS x64 | 189 | 10.7 | 16.0 | 1.5× |
| macOS ARM64 | 192 | 10.2 | 17.3 | 1.7× |
| Wasm (EH) | 122 | 4.3 | 15.2 | 3.5× |
| Wasm (MVP) | 121 | 4.3 | 15.0 | 3.5× |
| Wasm (threads) | 122 | 4.2 | 14.0 | 3.3× |
| Total | 1,437 | 104 | 255 | 2.5× |
It’s tempting to say “CI compute isn’t free,” but for public repositories GitHub Actions genuinely is free in dollar terms. The real constraint is different and more binding: Actions concurrency is a fixed, shared pool per organization. Every job a full-catalog rebuild queues competes with everything else the org needs to run. A rebuild is one large indivisible task — the release isn’t done until the last extension lands — so halving its compute means the org clears it in roughly half the runner-time and frees concurrency for everything else sooner.
Against that, the cache costs almost nothing. It’s 75.68 GB in an R2 bucket, about $1.14/month of storage at list price, and R2 charges nothing for egress, so the cache is read on every build with no bandwidth bill. During the v1.5.4 sweep it served 1,345,990 reads against 166,160 writes — a ratio of 8.1 to 1, peaking at 273,680 reads an hour. Dependencies get compiled and written once, then served back many times.
A few dollars a month of object storage displaces about 150 runner-hours per release. That trade is not close.
Dependency caching may be the single highest-leverage use of object storage in this whole pipeline. Every extension build otherwise recompiles DuckDB core and a stack of shared C/C++ libraries; storing those artifacts once trades a couple of dollars for hundreds of runner-hours. The per-repository Actions cache can’t capture that value precisely because it isn’t shared across extensions.
The through-line
Three findings, one conclusion.
Engine releases outpace extension availability, and the gap lives in a tail measured in days to weeks. The PR pipeline feeding extension updates is bounded by finite maintainer attention and is getting less predictable as volume grows. And rebuilding the whole estate is expensive without estate-global caching, which makes “just rebuild everything promptly” costly enough to discourage doing it often.
A distribution addresses all three at the point of integration instead of pushing them onto every user individually. It rebuilds and tests the whole catalog against each engine release, cheaply enough to do consistently, and presents a coherent “engine and extensions are jointly ready” moment. That’s the role Haybarn fills, and it’s installable today:
npx haybarn@rc
# or
uvx haybarn-cliNone of this is a criticism of DuckDB Labs. The engine, the public vcpkg cache, the CI tooling, and now the pre-release catalog builds are what make a distribution possible at all — Haybarn is built directly on them, and the v1.5.5 improvement is genuinely the right direction. A focused core team advancing a database engine has its hands full, and the long tail of packaging and devops is a different, open-ended job that the engine roadmap shouldn’t have to absorb. That’s exactly the job a distribution exists to do, the same way Ubuntu does it for Linux.
Where the techniques here are useful upstream — the read-write cache design, the pre-built build images, the ccache upgrade — I’d be glad to contribute them back. Questions and collaboration welcome at [email protected].