rusty.today
35 Articles Plus 41 short updates, originally posted to LinkedIn, kept here for the record.

Complete index — 2014–2026

Everything, in the order it happened.

Writing about data engineering, infrastructure, and the tools I build.

DuckDB isn't losing its contributors. It's renewing them.

Tracking 562 DuckDB core contributors and extension maintainers across five years of GitHub: individuals move on fast, but they were rarely all-in on DuckDB, newcomers outpace departures, and the extension registry is now the main way in. AI is taking a growing share of everyone's time. DuckDB

Embracing nanoarrow along with Arrow C++

I'm interested in nanoarrow for smaller binaries, faster builds, and lower memory use, especially in WebAssembly. Here's what I've been adding and fixing to use it in my DuckDB work. Data Engineering

From Time Travel to Lateral Temporal Tables in DuckDB

A DuckDB prototype for scalar subqueries in the AT clause reveals a larger idea: composable time travel and lateral joins across table history. DuckDB

DuckDB Upgrade Speed, Measured from 110M Extension Loads

Nine months of telemetry from Query.Farm's DuckDB extensions — 110 million loads — show each release landing slower than the one before it. Fourteen days in, v1.5.0 reached 33.7% of the population I can see; v1.5.5 reached 12.1%. The decline survives excluding CI traffic, excluding the v1.4 LTS line, and restricting to patch releases only. DuckDB

Who Actually Runs DuckDB, Measured from 110M Extension Loads

The companion to my post on DuckDB upgrade speed. Once you strip out the CI traffic that is 70% of my telemetry, the extension catalog isn't the monopoly the load counts imply or the flat field the raw config-days imply — and a third of real installations turn out to be on a desktop. DuckDB

Paging Through a Parquet File in DuckDB: file_row_number or OFFSET?

You have a large Parquet file and an API that has to return its contents, but responses have a size ceiling, so the caller pages through it. LIMIT/OFFSET is the obvious way to write that and it is the wrong one. I measured file_row_number against it: 2.53x faster on a file with 163 row groups. Speed is the boring half of the answer. OFFSET will also hand back the wrong rows without telling you. DuckDB

A DuckDB Release Isn't a Distribution

When DuckDB ships a new version, every community extension has to be rebuilt against it before anyone can install one. I measured how much of the catalog is actually there when a release lands, how long extension updates take to merge, and why the same builds run 2.5x faster on different CI. The answer to all three is distribution work. DuckDB

Compiling Isn't Running: Functionally Testing DuckDB-WASM Extensions

A DuckDB extension that compiles for WebAssembly has only proven that it compiles. Whether it loads, and whether it actually runs, are separate questions. I built a Node harness to ask them across 124 community extensions. Here's what it found and the fixes that came out of it. DuckDB

Quack and VGI: Two Approaches to Bringing RPC to DuckDB

Quack — the new client/server protocol extension for DuckDB — was announced today. It's been built since November 2025, almost exactly as long as I've been building VGI. Here's what's interesting, what's surprising, and where the two projects converge and diverge. DuckDB

DuckDB ↔ Arrow Compatibility: A Status Page

A working status page for the Arrow-related issues and PRs I've filed against duckdb/duckdb — the UNION saga, schema/data disagreements in nested appenders, type fidelity through round-trips, and the bigger question of how defensive the C Data Interface should be. Data Engineering

GeoSilo vs GeoArrow + Byte Stream Split: Two Approaches to Geometry Compression

A detailed comparison of GeoSilo's delta-integer encoding against GeoArrow's columnar float64 layout with Parquet BYTE_STREAM_SPLIT, covering compression ratios, precision tradeoffs, and when each approach wins. Data Engineering

VGI Injector: A Tiny HTTPS Download-and-Execute Binary in Zig

I needed a self-contained binary that downloads a program over HTTPS and exec's it — small enough to run in a FROM scratch container with nothing else. Go and Rust couldn't get small enough. Zig could. Programming

TIME Data Type Compatibility Across Databases

A survey of the TIME data type across 14 databases, comparing supported ranges, maximum values, and whether the special value 24:00:00 is accepted. Data Engineering

Telemetry for DuckDB Extensions Without the Pain

I open-sourced the telemetry client I use across Query.Farm's DuckDB extensions. It's two files, one function call, and it only collects platform and version info. DuckDB

Releasing vgi-rpc: An RPC Framework Built on Apache Arrow

I built an RPC framework for Python that uses Apache Arrow IPC as the wire format and Python Protocol classes as the interface definition. No .proto files, no codegen — just type annotations. Programming

Acronym-Aware Case Conversions in the DuckDB Inflector Extension

The Inflector extension for DuckDB now supports configurable acronyms, so case conversions preserve terms like HTML, API, and URL as fully uppercase — configured through a native DuckDB setting. DuckDB

Streaming Data into DuckDB with Arrow and Python Generators

How to use Python generators and Apache Arrow's RecordBatchReader to stream large datasets into DuckDB without loading everything into memory at once. DuckDB

Cron Expressions For DuckDB

Introducing a DuckDB extension that interprets cron expressions to generate scheduled timestamps directly in SQL, built with the Rust croner crate. DuckDB

Enhancing DuckDB with Unix Pipe Integration: Introducing the shellfs Extension

Introducing shellfs, a DuckDB extension that enables seamless integration with Unix pipes for both input and output, allowing command-line programs to be used directly within DuckDB queries. DuckDB

Turn The Tape Recorder On And Keep It Running

How to build a kitchen-sink observability event stream using AWS EventBridge, Firehose, and Apache Iceberg to capture and query every event in your distributed system. Data Architecture

The Bare Minimum of Metadata For Any Data

A practical guide to the essential metadata columns every data platform should store alongside its data, enabling traceability back to source files and loader versions using Apache Parquet's efficient encoding. Data Engineering

DuckDB is Strategically Important

Why DuckDB is becoming the go-to tool for data tasks in 2024, along with key areas where improvements are still needed including Iceberg support, Parquet encodings, and remote data throughput. Data Engineering

Images, Emails and Content Addressable Storage

An investigation about how to store a large number of referenced URLs while considering WebP and AVIF recompression for images. Web

Cost-effective S3 for Subscription Data Distribution

Solving the problem where you have a lot of data you want to allow customers to obtain (provided they purchased access), but you don't want to pay for their internet transfer fees can be tricky. This architecture makes it easy. Cloud Architecture

Anomaly Detection for Cities and Airports

Cities and airports stream their radio traffic to the internet, you can listen to police or firetrucks being dispatched or airplanes being given headings and altitudes. By analyzing the duration of transmissions over time with machine learning, we make be able to achieve automated detection of emergencies or abnormal events. Prediction

Web Chat Survey

Looking into various web chat front end implementations Web

Thoughts on Purgeable Memory and Caching

A summary of thoughts about purgeable memory allocations and distributed caching. Computer Science

Predicting New York State's Electricity

A short adventure in prediction for electrical generation and demand in New York State. Prediction and Forecasting

Multi-dimensional Indexing

My personal research notes into multi-dimensional indexing strategies for data, involving Z-order curves and Hilbert curves. Computer Science

Bridger Foothills Fire

A fire started on the Bridger Range last Friday. These mountains are iconic if you've ever lived in Bozeman. They aren't going to look the same for a long time. Montana News

Which Image Should You Show?

Preferred Pictures is a software-as-a-service that algorithmically determines which image from many that will result in the most sales and conversions in a way that is arguably that is better than A/B tests. Ideas

Rusty's Guide to Tulum, Mexico

My short guide to adventures and happiness in Tulum, Mexico Travel

Finding Available Short Domain Names

Find short domain names that are able to be registered Internet

Largest Integer Consisting of Unique Triple Primes

Goal: Find the largest integer where all triples are unique and prime. Programming

ffmpeg Scene Change Detector Examination

ffmpeg is a very versatile piece of software for working with video. At its core there are video filters, and one of these filters is called select. The select filter allows a user to only write certain frames that pass thresholds to the output, in other words you can pull out interesting pieces of video using a select filter. A common use of the select filter is to produce thumbnails for each scene in a video. Programming
41 Short updates Originally posted to LinkedIn, collected here for the record.

Short updates

I built a thing for runners and walkers who are tired of checking five weather apps before...

I'm teaching a hands-on DuckDB Extension Development Workshop in Amsterdam on January 30th

Announcing the ADBC Scanner Extension for DuckDB

The DuckDB extension workshop in Amsterdam this January is already completely full—it filled up...

Big Update to the DuckDB crypto Extension!

Deep Dive into DuckDB Extensions — Let's Workshop in Person!

Excited to share something very cool with the DuckDB + Arrow Flight community.

I've been deep in Kafka + DuckDB development the past few weeks, but here are some late-night...

Excited to announce the Inflector Extension for DuckDB by Query.Farm!

Help support Query.Farm!

Introducing the JSONata Extension for DuckDB by Query.Farm!

#Kafka Reading/Writing in #DuckDB is possible

New DuckDB Extension Launch!

The DuckDB Developer Meeting 2026 has been announced!

Excited to announce the JSON Schema extension for DuckDB!

Announcing the Tera DuckDB Extension!

DuckDB Lab's roadmap for the next year (2025/2026) has been updated -https://duckdb.org/roadmap.html

Heading to Paris today for PyData Paris and to speak at the Apache Arrow Summit next Thursday,...

Fun Friday afternoon hack!

It’s always great to see the work we’re doing at Query.Farm show up in the wild.

I had the chance to sit down with Michael Watson on the Hedgineer Podcast for an hour-long...

Any time I wind up using scikit.optimize.differential_evolution (https://lnkd.in/e8mAHtRZ) is a...

Query.Farm Announces the integration of QuackScience's DuckDB Extensions

Lets look into using DuckDB with Arrow and Python generators to handle data that is streaming or...

Announcing Textplot - bringing beautiful ASCII/Unicode data visualization directly to your SQL...

NEW: Stochastic Extension for DuckDB!

Fuzzy string matching in DuckDB just got easier!

New DuckDB Extension: bitfilters - Probabilistic Data Structures for Lightning-Fast Queries

Excited to announce the release of the hashfuncs extension for DuckDB!

New DuckDB Marisa Extension by Query.Farm

Just published a new benchmark report for Airport, our DuckDB extension that uses Apache Arrow...

Thank You for the Incredible Interest in Airport

The Airport extension for DuckDB is now live!

Kafka + DuckDB, now seamlessly connected.

Are you getting "extension not found" errors in DuckDB?

New DuckDB Extension: Radio

Thoughts on DuckDB's DuckLake so far:

Here’s a short clip from my presentation at DuckCon 6 in Amsterdam: Airport for DuckDB: Letting...

Today is DuckDB's DuckCon 6 in Amsterdam.

DuckCon 6 in Amsterdam is just a few weeks away!

I'm excited to announce the datasketches extension for DuckDB, which integrates the capabilities...

Showing 1–10 of 41