May 2026

A quieter day of work, with activity focused on a private repository. Rusty made a single commit to maintain ongoing progress on internal projects, keeping momentum on work that isn't yet ready for public view.

May 30 was a substantial multi-repo day, with 42 commits across 8 public repositories plus 18 commits in private work. The focus spanned distributed query infrastructure, WebAssembly releases, geospatial charting, and community extension maintenance.

vgi-java — Core Infrastructure Refactoring

Rusty continued hardening the core VGI Java layer with several targeted fixes and refactors:

• Secret token handling: Fixed HTTP-layer round-tripping of resolved secrets through ScalarStreamState, ensuring credentials remain intact across distributed boundaries
• Dead code removal: Eliminated legacy TIO finalize machinery and migrated exception_process/exception_finalize to buffering semantics, dropping obsolete fixtures
• Distributed aggregation: Made sum_all_columns_simple_distributed a buffering function for consistency
• Feature sync: Synced with upstream VGI Python to incorporate late-materialization, value pruning, and view column comments

vgi-rpc-go — Performance Optimization

A notable performance improvement landed in the Go RPC layer: record-batch payloads now write directly into the shared memory (SHM) region, eliminating intermediate buffering and reducing copy overhead.

haybarn-wasm — Release Cycle and Bug Fixes

The WebAssembly engine moved through a release cadence with multiple version bumps (rc10 → rc11). A critical fix addressed URL-encoding of HTTP header values—now properly preserving quoted ETags that were being incorrectly escaped.

cupola — Geospatial and AI Charting Features

Substantial work landed across geometry visualization and AI-powered charting:

• Geometry rendering: Fixed polygon winding to ensure maps render correctly; geometry charts now output as GeoJSON Features with worldwide spatial-distance prompting
• WKT shell support: Added geometry plotting and WKT rendering capabilities
• AI robustness: Eliminated empty tool input loops that triggered parse errors; added a loop-breaker mechanism and chart image pruning
• Schema introspection: Enhanced describe_table to work with views, advertise loaded extensions, and render Decimal columns with proper precision/scale notation (DECIMAL(p,s))
• Function details: Introduced rich FunctionDetail objects exposing signature, parameter, and return metadata
• Infrastructure: Updated haybarn-wasm to rc11, rebuilt against vgi-typescript, and removed legacy kepler.gl map integration

vgi-rpc-java — State Management

Fixed a subtle issue in TransferPair where copyRoot and shadowed state fields needed qualification to prevent collision.

Open-Source Stewardship

Rusty opened PR #1991 on duckdb/community-extensions to update the airport extension to the latest commits, and bumped the extension version accordingly. This maintains currency with upstream DuckDB community tooling.

Related Repos

Minor maintenance also touched vgi-go (late materialization + view column comments) and haybarn-community-extensions (stress-test harness and launcher/pool updates).

Private repos saw 18 additional commits, keeping unreleased infrastructure in step with the public development pace.

Rusty focused on performance optimization across both Java and Go implementations of the VGI scalar dispatch system.

In vgi-java, he tackled two key efficiency gains. The first commit reuses a VectorSchemaRoot across batch processing iterations, eliminating allocation overhead in the hot path. The second introduces broader hot-path optimizations for the scalar dispatcher, likely reducing branching and call overhead in the core execution loop.

Work in vgi-go centered on the multiply operation with three targeted optimizations. He implemented a direct-buffer fast path to bypass intermediate builder abstractions, followed by a pure-Go pooled allocator for the Int64Builder buffer to reduce garbage collection pressure. The final commit removes the Int64Builder entirely in favor of writing directly into pooled buffers, cutting allocations even further in the multiplication hot path.

Additionally, Rusty contributed to private repository work with 4 commits on a separate project.

haybarn-extension-ci-tools received a fix to normalize engine tags during per-extension registry publishing, ensuring consistent tagging behavior in the CI pipeline.

Over in vgi-java, performance improvements were made to the example worker by hardening and speeding up core scalar, table, and aggregate fixtures—work aimed at making test infrastructure more robust and faster.

cupola saw a prompt fix related to chart rendering, clarifying that render_chart operations should be treated as user-initiated rather than agent-initiated.

The bulk of the day's effort landed in vgi-rpc-python, which received several quality and maintenance updates. Ruff linting rules were progressively enabled, including TC006 (quote typing.cast() type expressions), RET, C4, PIE, ISC, and FURB110 rules—steadily raising code quality standards. The CI configuration was upgraded from astral-sh/setup-uv v6 to v8.1.0 to support Node 24. Dependency management was tightened with a uv.lock sync to account for pywin32's Windows marker, and type checking was enhanced by enabling warn_unreachable and removing dead str/bytes coercion branches.

Private repositories also received 5 commits across 3 projects, maintaining momentum across the broader codebase.

A productive day spanning multiple infrastructure and UI layers across the Query Farm ecosystem.

vgi-java and vgi-rpc-java received Java 21 support. The bytecode target was bumped to ensure the example worker runs on JDK 21+, and vgi-rpc-java gained multi-release JAR support along with a __transport_options__ capability handshake responder. A new GitHub Actions workflow was added to run tests across JDK 21 and 25 in 25 parallel lanes.

vgi-rpc-go and vgi-rpc-python both received the __transport_options__ transport capability feature as part of a coordinated handshake enhancement. The Python implementation also included docstring and import order fixes to satisfy ruff linting.

cupola saw extensive work on chart rendering and AI integration. The Vega-Lite charting system was overhauled to support multi-source data via extraData, with fixes for faceted/concat/repeat specs, scroll flicker, and right-clipping. Chart rendering now defers until the agent evaluates, integrates tabbed maximize dialogs, and surfaces render failures and warnings to the AI. The AskAIChat feature gained a Vega-Lite chart tool that sends rendered PNG feedback to the agent for visual context. Fixes also addressed JSON-safe row reading, BigInt support, download menus, and chart feedback instruction organization.

Haybarn infrastructure received attention across multiple repos. haybarn-community-extensions fixed deployment to pass descriptor fields via environment variables rather than shell interpolation, disabled the source-archive job, and cleaned up version normalization. haybarn-status added fixed-width run-table columns, per-extension insights charts, and build-time visualization, while excluding cancelled legs from aggregate metrics. haybarn-github-actions introduced status feed retention pruning, tags, reliability metrics, and per-extension community compute breakdowns. A PR on haybarn-extension-ci-tools (PR #2) addressed version tag mapping, stripping the haybarn-v1.5.3* prefix back to plain semver v1.5.3 in override expressions and dropping support for the deprecated 1.5.2 release.

Private repo activity included 2 commits across separate projects.

Rusty had a prolific day across the Query.Farm ecosystem, with 42 commits touching five active repositories plus significant private repo work.

vgi-java received a performance optimization targeting the example worker: byte-level concatenation in concat_values eliminates unnecessary String transcoding overhead.

haybarn-wasm saw infrastructure and feature improvements. The CI configuration bumped the engine submodule to pick up a haybarn.h installation fix, while a new feature now honors POST cancellation flags at the request boundary — enabling cleaner shutdown semantics for long-running operations.

cupola dominated the activity with a sustained refactoring and feature push across 24 commits. Work included:

• A multi-stage refactoring initiative: stage 4 extracted shell initialization and AI tool executor logic; stage 3 centralized URL parameter handling and Arrow decoding; stage 2 improved bridge lifecycle hygiene; and stage 1 applied surgical security and correctness fixes
• Extensive branding updates swapping the Cupola logo to a weathervane PNG, enlarging it to 32×32px, improving header layout, and refining the welcome/connecting/error screen presentation with rounded corners, shadows, and soft rings
• Safari compatibility fixes pinning the header logo dimensions via HTML attributes (working around Tailwind preflight height:auto override) and ensuring proper centering with self-center and shrink-0
• Shell UX improvements showing "Warming up Haybarn" during WASM compilation and pthread spin-up, and keeping the boot overlay visible until Haybarn is ready
• Mosaic integration cleanup: removed the integration entirely (validator, lint, schema tools, docs, examples, editor tab, agent tools), reversing earlier preaggregation and semantic lint work
• Version bump to 0.4.54

vgi-rpc-java received performance enhancements: a pooled non-zeroing allocator for large buffers (enabled by default) and opt-in zero-copy inbound resolve via the VGI_RPC_SHM_ZEROCOPY flag reduce memory allocation and copying overhead.

haybarn-community-extensions was kept current with engine pin bumps to haybarn-v1.5.3-rc3 and VGI dependency updates incorporating opt-in per-pass client timing for data round-trip measurement.

Across the board, the work balances performance optimization, infrastructure stability, user experience refinement, and architectural cleanup.

A particularly productive day across the VGI ecosystem and Cupola, with substantial work on infrastructure, storage backends, and user-facing features.

VGI Core & Transport

Significant progress on vgi-java centered on FFM (Foreign Function & Memory) and storage architecture. Rusty upgraded the build to JDK 25 toolchain for shared-memory transport support and documented the new shm transport layer. More ambitiously, he implemented a three-tier FunctionStorage system supporting memory, SQLite, and Cloudflare Durable Objects backends with per-attach sharding—a major architectural shift for distributed function persistence.

The vgi-rpc-java repo received a complete bidirectional zero-copy POSIX shared-memory side-channel, enabling high-performance inter-process communication. Over in vgi-go, the SQLite backend was rewritten onto the unified core with per-attach sharding to match the Java architecture, and a new make test-shm target was added to validate the integration suite over the shm transport. He also fixed attach-opaque-data unwrap logic and dict filter behavior. The vgi-rpc-python binding received a critical fix for pointer batch construction using pa.nulls(0) to properly support union and extension types.

Haybarn WASM & Extensions

haybarn-wasm bumped to version 1.5.3-rc9 with several fixes and documentation. Rusty resolved a stale HEAPU8 view issue in Emscripten XHR helpers that was causing RangeError under Cross-Origin-Isolated (COI) conditions, silenced spurious warnings on engine-glue postMessages in the async dispatcher, and captured debugging techniques for eh-variant extensions plus release mechanics in the docs.

The haybarn-community-extensions repo saw VGI dependency bumps to fix vgi_table_scan survival through LogicalGet deep-copy in WindowSelfJoin and struct_extract filter serialization. Dispatch and list logic now skip fully-disabled extensions, and descriptors were resynced from upstream duckdb/community-extensions.

Cupola: Feature & Polish Sprint

cupola dominated the day with a flurry of features and fixes spanning multiple releases (v0.4.38, v0.4.32, and ongoing work).

Chart & Visualization work included:

• New Mosaic tab editor with AI render-time error loop and fixed download MIME types
• Fullscreen overlay with SVG/PNG export for Mosaic charts
• Refresh, watch, and Cupola-specific doc guidance to prefer SQL over inline data
• Per-render data-name uniquification to prevent multiple charts from sharing Coordinator cache
• Theme widgets with earth/harvest palette, viewport fill in fullscreen, and per-plot vconcat/hconcat scaling
• Async error capture with 54 bundled Mosaic example specs registered as AI tools
• Quiet-period settle algorithm for reliable inline-array data handling and unhandled-rejection capture

AI chat integration expanded with generate_chart and read_chart_docs tools, allowing the agent to render Mosaic vgplot specs inline within conversation.

Chat UI enhancements delivered:

• Inline Cancel button on running queries with tool indicators
• Elapsed-time counter displaying query duration (worker-level progress unavailable, so timer substitutes)
• Actual abort semantics so cancel truly halts agent queries and interrupts during Thinking phase

Infrastructure fixes included dropping data-name mangling in favor of Coordinator.clear() to enable cross-source SQL references, emitting client source maps via Vite's 6 environments API, bumping haybarn-wasm to 1.5.3-rc9, and correcting the shell boot message from "Downloading DuckDB" to "Downloading Haybarn." A new animated boot screen with phase, progress, and elapsed timer improves the initial load experience.

Private Work

21 commits across 8 private repositories, details undisclosed.

Rusty tackled a significant investigation into WebAssembly exception handling across multiple projects today, alongside branding refinements and RPC infrastructure improvements.

WebAssembly & Extension Build Tools

The day's most substantive work involved diagnosing and addressing a critical issue with Rust C-API extensions on WebAssembly. After discovering that community extensions like evalexpr_rhai and lindel crash on the eh (non-COI) wasm variant with an opaque TypeError: c is not a function, he opened issue #9 on haybarn-wasm and submitted PR #1 to haybarn-extension-ci-tools proposing a fix: building wasm_eh Rust extensions with native wasm exceptions support. The root cause appears tied to exception handling differences between the COI and non-COI variants, with the PR description indicating the investigation is still underway but a clear path forward has emerged.

RPC & Serialization

On the Go side, vgi-rpc-go received updates to CORS headers handling, with commits adding explicit Allow-Headers echo and introducing Cross-Origin-Resource-Policy headers for better cross-origin resource control.

In vgi-rpc-java, he addressed a resource management issue by ensuring ArrowStreamWriter dictionary copies are properly released on close, and forwarded DictionaryProvider support along the HTTP stream path—improvements that strengthen dictionary handling across RPC boundaries.

Branding & UI Polish

cupola saw a comprehensive visual refresh with the adoption of the Query.Farm earth/soil/harvest color palette and the Outfit display font. He introduced a new Cupola icon (keeping the tractor with Query.Farm) and refined the wordmark with animated connecting screen elements, including a harvest spinner halo centered around the VGI logo. Supporting changes included breadcrumb and header refinements, quieter typography, and a fix to load spatial extensions from core rather than community.

Workspace Maintenance

haybarn-wasm received a release bump to 1.5.3-rc8, advancing the workspace version as development progresses.

Private repository activity accounted for 17 commits across 8 repos, indicating parallel work on internal infrastructure.

May 23 was a high-velocity day across multiple repositories, with work spanning preview optimization, RPC improvements, CI/CD enhancements, and a new CLI tool release.

cupola received two targeted fixes addressing preview performance and reliability. The first commit resolved a timing issue where the preview was not properly awaiting bridge.query before proceeding, while also cleaning up timezone sync timer logic. The second commit introduced a significant optimization: replacing COUNT(*) queries with a LIMIT N+1 approach that leverages rowid, primary key resolution, and ORDER BY ladder logic. This change should substantially improve preview responsiveness when dealing with large result sets.

vgi-rpc-typescript was updated to improve CORS handling by reflecting preflight Access-Control-Request-Headers back in the response headers. This ensures that browser preflight requests receive proper header information, enabling more flexible cross-origin requests.

On the haybarn-community-extensions side, Rusty added a CI/CD optimization that introduces a no-rebuild npm republish path. This allows already-built extensions to be republished without requiring a full rebuild, streamlining the release workflow.

vgi-nl-trains-cli saw the most activity, marking its initial release cycle. The tool received several improvements including a spinner loading indicator and title truncation for long pane titles (v0.1.2), then was renamed and scoped as @query-farm/vgi-nl-trains-cli for npm publishing. A critical fix addressed npx and symlink entry point detection by properly resolving the realpath (v0.1.1), and the package was published to npm under the query-farm-vgi-nl-trains-cli name.

In addition to public work, Rusty made 14 commits across 4 private repositories.

Rusty had a heavy day spanning seven repositories, with 50 commits focused on improving fixture ergonomics, extension loading, worker stability, and cross-platform compatibility.

vgi-java core improvements dominate the day's work. He introduced ScalarFn, a Pythonic compute() base pattern for scalar functions, along with comprehensive documentation in CLAUDE.md. The ParameterExtractor and BatchUtil.emit utilities were added to streamline fixture setup, and fixtures across the example worker were swept to use the new patterns. A new FunctionSpec API collapses the FunctionDescriptor trio into a cleaner spec() interface, aligned with vgi-python. Additional refinements include OpaqueDataSealer accepting external keys for multi-replica HTTP deployments, Worker.runUnixSocket properly piping --idle-timeout through to transport, and the aggregate finalize method now taking FieldVector directly rather than VectorSchemaRoot. License headers were also dropped for cleaner diffs.

haybarn-wasm steady iteration pushed through release candidates (1.5.3-rc4 through rc7) with targeted fixes. TIMESTAMP WITH TIME ZONE output was fixed to honor session timezone in Arrow IPC export, and Arrow itself was modernized from version 52 to 58. A new SAB-based query cancellation API was added for better async control, and default extensions now auto-load silently at startup rather than requiring manual setup.

cupola shell and token flow work addressed several browser integration issues. Extension pre-loading was hardened: ducklake, icu, json, iceberg, and spatial are now explicitly loaded before vgi, with a 5-second timeout on timezone sync to prevent shell deadlock. Auth token handling was fixed so both SPA-flow (getAuthTokenForService) and cookie-flow tokens reach ATTACH correctly. Browser worker compatibility was improved by avoiding Blob wrapping (which caused Safari null-origin sourcemap errors) and ensuring haybarn worker .map files ship with absolute URLs for COI pthread sub-worker resolution. The DuckDB engine was ported to @haybarn/[email protected].

vgi-rpc-typescript client fixes eliminated cross-implementation mixing in the browser—introspect and buildRequestIpc now use impl-arrowjs exclusively, dropping flechette. Socket-aware IpcStreamWriter was introduced with proper awaiting of all write calls.

vgi-rpc-java transport and type work refined exception logging (dropping type-name prefix) and ensured UnixSocketTransport.serveForever honors --idle-timeout. A subtle fix to castRoot prevents mangling of list<dict> types by recursing equivalence checks into nested structures.

haybarn-community-extensions bumped vgi to gate async table-init dispatch behind VGI_ASYNC_INIT_ENABLED (off by default on WASM).

vgi-go developer experience received OAuth PKCE + HTTP prefix/CORS env config in RunHttp, typed scalars, named loggers, and panic recovery. A new CatalogTable.InlineBind method skips per-scan bind RPC, improving query performance.

Private repo activity across three repositories accounted for an additional 7 commits.

A substantial day across the VGI ecosystem and related infrastructure, with 99 commits spanning six active repositories and significant private work.

vgi-rpc-go saw focused improvements to the RPC metadata and transport layers. Key changes include a fix for projected-batch metadata writes with guard logic on column-count rewrapping, a new OutputCollector.EmitWithMetadata method for per-batch annotations, improved error handling that errors instead of panicking when Arrow structs are mapped to non-struct fields, and the addition of Server.RunUnix for AF_UNIX launcher transport support.

vgi-go received extensive work on core subsystems. Rusty ported the table-buffering subsystem (sink/combine/source mechanics) along with buffer_input, added batch_index and partition-value emit helpers, expanded fixture coverage for ordered/batch_index/large_state/crash/hang/slow scenarios, and implemented echo pushdown with witness and buffering variants. Additional changes include transaction ID allocation with legacy migration support, vgi_percentile validation, partition_columns and batch_index fixtures with declared-fields contract validation, and a new AF_UNIX launcher transport via Worker.RunUnix with worker flags. The protocol was also resynced to the upgraded vgi-python version with scan-branches ported.

vgi-rpc-python released version 0.18.3, introducing Windows named-pipe RPC transport (serve_named_pipe) in Phase 3c of Windows support. Version 0.18.2 patched Dependabot security alerts and updated Node CI actions to version 24.

haybarn-wasm underwent maintenance and feature work: the engine submodule was pinned to haybarn-v1.5.3-rc2, a VGI interactive OAuth bridge was added to the shell layer with popup and shared-buffer relay mechanics, the workspace was bumped to 1.5.3-rc3, an intermittent LOAD deadlock on threaded (COI) builds was fixed, and the packaging layer was updated to match the engine submodule.

haybarn-community-extensions saw substantial integration work for the VGI extension. After adding the vgi extension, multiple descriptor and dependency bumps refined build compatibility: fixes for C++17 linking, arrow-wasm vcpkg overlays, vcpkg-gitlink handling, and cross-platform considerations (initially excluding Windows, then reverting that exclusion). The deployment pipeline was simplified to a single-job design with parallel signing/compression and AWS S3 sync. Browser decompression detection, httpfs 1.5.3 pinning, MinGW compatibility, and HTTP payload/compression logging were all integrated through incremental bumps.

On the community side, Rusty opened PR #1943 and PR #1942 to duckdb/community-extensions for upstream updates to airport and inflector, while also patching an inflector version bump in the DuckDB extensions repository.

Private repositories received 54 commits across 7 projects, indicating parallel development and integration work.

A productive day spanning the VGI ecosystem and Haybarn infrastructure, with particular focus on RPC protocol enhancements, WASM integration, and community tooling.

VGI Language Bindings & Protocols

The vgi-java implementation received a substantial update, porting recent features from the Python and C++ codebases including multi-branch scanning, table buffering, and release management. Over in vgi-rpc-rust, Rusty implemented opt-in sticky sessions for both unary and streaming call types, alongside adding protocol_version to the describe RPC call. The TypeScript bindings (vgi-rpc-typescript) saw multiple review-driven refinements: hardening the flechette patch tooling, fixing resource teardown in the reaper, improving DecimalBigNum detection, ensuring sticky key parity, and strengthening the incremental-encoder facade. In vgi-rpc-python, version 0.18.1 was released with CLI support for forwarding protocol_version on every call. Type safety improvements landed in vgi-go, where the remaining string-coded enums (Phase, order-by, write-op) were converted to proper typed definitions.

Haybarn WASM & Status Infrastructure

The haybarn-wasm project received intensive attention across caching, threading, and release management. Several commits address WASM artifact delivery: serving the COI (Cross-Origin Isolate) WASM from Haybarn's R2 object storage instead of jsDelivr, fixing the R2 path resolution, implementing version-keying to defeat immutable caches, and disabling response caching for error conditions. On the engine side, updates targeted v1.5.3 (and earlier v1.5.2-rc iterations), with single-threaded extension LOAD enforcement on threaded builds and hardened initialization logic. The shell UI received rebranding updates and now serves extensions over HTTPS.

A new project, haybarn-github-actions, was initiated with an GitHub org webhook collector and StatusFeed component. The haybarn-status page underwent a comprehensive redesign, now themed to match the query.farm Astro site with multi-version support, per-platform build grids, sound notifications, and auto-refresh. The backend was refactored to read all community extension runs from DigitalOcean without pagination caps, and PR #1 was opened to integrate these changes alongside authentic OS logos and the new design system.

Community Extensions & Operations

In haybarn-community-extensions, significant DevOps work landed: a new Makefile for dispatching, cancelling, and resubmitting per-extension builds; artifact lifecycle management (expiring pip wheels and npm leaves after 2 days); and uncompressed .duckdb_extension files now shipped in community npm leaves. The CI configuration was updated to track the extension-ci-tools Haybarn branch tip instead of pinning SHAs, and defaults were bumped to target engine v1.5.3. Documentation corrections fixed stale references to pinned ci-tools commits.

Open-Source Stewardship

Rusty replied to PR #137 on the duckdb-quack project, providing feedback on the CancelRequest remote command implementation.

Private Work

Additional activity across 3 private repositories with 21 commits total.

Rusty had a productive day across the Query-farm and Haybarn ecosystem, with 32 commits focused on infrastructure upgrades, protocol conformance, and WebAssembly enhancements.

vgi-rpc-go received attention for protocol version management. Commits bumped the external vgi-rpc dependency to >=0.18.0 and enforced protocol_version in CI, while deferring SetProtocolVersion("1.0.0") calls until PyPI packages catch up to upstream changes.

haybarn-wasm saw the most substantial work—a major Emscripten toolchain upgrade from 3.1.71 to 5.0.7, along with Arrow submodule updates to 21.0.0. The feature branch feature/oauth-api was merged into haybarn, bringing OAuth surface APIs for extensions. Notable additions include duckdb_web_get_query_progress bindings, OAuth popup support with open_auth_url and get_auth_error, Web Crypto and SHA-256 platform stubs, and a rewritten HTTP transport using EM_JS with unified wire format. Infrastructure improvements included bundling DuckDB workers same-origin while keeping WASM on CDN (fixing shell initialization hangs), emitting proper index.html routing, and deploying the Haybarn shell to Cloudflare Pages. Pthread worker code was rewritten for the emsdk 5.x model, and Chrome test reliability was improved enough to remove continue-on-error guards.

vgi-rpc-typescript received conformance and CI work. Python test defaults were refined, the build and HTTP-conformance jobs were driven green, and vgi-rpc[external] >=0.18.0 is now required in CI.

haybarn-community-extensions underwent CI restructuring. The build_all workflow now defaults to R2-only deployment without requiring duckdb_tag, registry publishing is opt-in, and the emsdk 5.0.7 upgrade was pinned across the build pipeline.

Rusty also engaged in community stewardship, replying to issue #20068 on duckdb regarding batch index collisions in PhysicalBatchInsert, and reviewing PR #11 on pcap which added packet parsing tests.

A high-velocity day across the VGI-RPC protocol ecosystem and Haybarn WASM infrastructure, with significant work on community extensions and CI/CD modernization.

VGI-RPC Protocol Stack

The vgi-rpc-go implementation received protocol enforcement updates, syncing to the 0.17.x release line with gzip codec support, error_kind categorization, and sticky session handling. Rusty pushed enforcement of protocol versions to the dispatch boundary, ensuring backward compatibility at request ingress rather than deeper in the call stack.

Corresponding work in vgi-rpc-typescript threaded protocolVersion through the HTTP dispatch layer and drove remaining arrow/flechette conformance failures to zero using facade type constructors and per-record-batch custom metadata via local patches. The TypeScript implementation also synchronized sticky sessions, gzip, error_kind, and AEAD primitives from the Python reference.

vgi-rpc-java picked up the HTTP transport improvements, porting sticky sessions, soft-cap producers, and zstd uploads from Python, while switching StateSerializer to CBOR encoding for more compact stream-state tokens.

Haybarn WASM Modernization

The haybarn-wasm repository underwent sweeping infrastructure refactors. He consolidated the CI/CD from 6 separate jobs into a single matrix job running in Docker, integrated ccache and sccache via Cloudflare R2, and switched to a shared Haybarn toolchain image for faster builds. Publishing now uses HAYBARN_NPM_TOKEN (org secret) instead of OIDC Trusted Publisher.

Patch management was restructured to keep upstream DuckDB verbatim and layer minimal Haybarn pre-adapter changes. Key fixes included:

• Accepting .duckdb_extension.wasm suffix validation in extension load paths
• Passing DUCKDB_WASM_VERSION through CMake ExternalProject to preprocessor flags
• Correcting extension_load.cpp call signatures and config-scope handling for loadable WASM extensions
• Sourcing emsdk_env.sh before npm bootstrap and pinning emsdk install paths

Documentation was substantially expanded with local smoke-build recipes (CLAUDE.md), updated READMEs, and migration guidance from DuckDB to Haybarn.

Community Extensions Ecosystem

In haybarn-community-extensions, Rusty deployed significant tooling and triage improvements. He added triage_failures.py for ongoing per-extension build-vs-upstream comparison and triaged 13 haybarn_regression cases. The publish pipeline was opened to all community extensions (dropped allowlist) and now surfaces extension descriptions and source-repo links in generated npm READMEs, forwarding descriptor licenses to package.json.

Cache-Control headers were set on direct R2 binary uploads, and the upstream sync now preserves excluded_platforms, opt_in_platforms, and haybarn_skip_tests flags. Twenty-six extensions that don't yet build against DuckDB 1.5.2 were disabled, with npm + PyPI publishing bumped to use updated ci-tools.

The .github organization profile was updated with pre-release banners and current install commands.

Haybarn Status & Monitoring

haybarn-status now caches per-run jobs in KV, dropped the cron from hourly to every 15 minutes to clear GitHub rate limits, and stopped filtering ~150 extensions from the CI matrix—all extensions now run through status checks.

haybarn-community-extensions — CI/CD and Deployment Refactoring

Rusty spent the bulk of the day restructuring the community extensions CI/CD pipeline and deployment infrastructure. Key changes include:

• Unified R2 bucket migration: Retired the community-only bucket and consolidated publishing to a single haybarn-extensions.query.farm/community namespace with per-commit immutable layout for PyPI and npm artifacts.
• Workflow consolidation: Integrated a consolidated publish workflow from ci-tools, eliminating redundant scripts. The registry publish step is now its own re-dispatchable workflow with proper GitHub Actions permissions (actions:read at the deploy job).
• Extension allowlist expansion: Widened the publish gate from a single extension (waddle) to 11 extensions, and swapped chsql for rapidfuzz in the allowlist.
• CI robustness: Added npm EEXIST tolerance on republish, split r2-smoke into a standalone diagnostic workflow for R2 PutObject auth debugging, and pinned awscli checksum env vars for R2 compatibility.
• Dependency updates: Bumped ci-tools multiple times throughout the day, bringing in sccache for Rust builds, shallow engine clones, Windows PowerShell fixes, and git-clone variable prefix improvements.
• Upstream comparison tooling: Enhanced compare_with_upstream.py to separate Haybarn-only failures from broken-upstream ones.
• Extension compatibility: Disabled 6 extensions pinned to pre-1.5.2 duckdb-rs crates to align with current toolchain requirements.

vgi-rpc-python — Sticky Sessions and Error Handling

Work on the RPC framework focused on session management and improved error semantics. Opened two PRs:

PR #9 introduces opt-in HTTP sticky sessions (first of a planned 3-PR series). Sessions are framework-managed, token-based (riding in the VGI-Session header—no cookies), and controlled client-side via with conn.with_session_token(): context. The default is off, and the non-sticky wire path remains byte-identical to prior versions.

PR #8 adds a typed MethodNotImplementedError (subclass of AttributeError) so clients can pattern-match on a stable error_kind identifier instead of substring-searching error messages. The error_kind metadata is hoisted from exception extras to a top-level wire key.

Additional commits:

• Implemented graceful sticky session drain with access-log instrumentation for session_id and session_action.
• Added sticky session echo headers and Fly.io quickstart helpers.
• Silenced mypy on pa.nulls() abstract-DataType overload.
• Fixed broken sticky-sessions-spec documentation link and added it to the nav.
• Merged the typed error marker feature.

Private Repos

9 commits across 3 private repositories.

Rusty logged a substantial day across three primary repositories, with significant activity in CI/build infrastructure and a new status monitoring system.

The bulk of work in haybarn-community-extensions focused on iterative CI tooling refinement. Multiple commits addressed ccache diagnostics and WebAssembly build caching, culminating in a configuration that achieved 100% remote cache hits by removing redundant local diagnostics. Along the way, he tackled several infrastructure issues: correcting R2 secret names that were silently causing no-op deployments, updating Node.js action versions, and switching the cache strategy to R2-only (dropping GitHub's L1 cache layer). The work also included fixing GHCR login logic and properly formatting multi-line GitHub Actions outputs via heredoc syntax.

Work on haybarn-status represented the scaffolding and initial implementation of a new status monitoring worker. This involved building out the data collection layer to parse extension names and attribute builds by commit, creating a per-extension build matrix view, and optimizing the UI with org logos, compact matrix layouts, and parallel job fetching. Performance improvements included table layout optimization and cron pre-warming. Configuration wiring connected the STATUS_KV key-value store and custom domain routing for haybarn-status.query.farm.

Over in vgi-rpc-python, Rusty enhanced the RPC protocol layer by surfacing protocol implementation details on CallContext and implementing HTTP content encoding negotiation to support zstd and gzip compression on the wire.

Beyond these visible repositories, 17 commits across two private codebases indicate continued work on related infrastructure or integration projects.

Rusty focused on infrastructure and CI improvements across the haybarn-community-extensions repository. He made 15 commits addressing a critical authentication issue in the build pipeline — specifically fixing a vcpkg token mapping problem that was causing 401 errors during cache write operations. This kind of CI debugging work is essential for maintaining reliable builds, especially in a community extension ecosystem where build failures can block contributors.

Beyond the visible activity, he also worked across two private repositories, making 14 commits there. The day was split between public infrastructure work and private development efforts.

A comprehensive refactoring and security hardening day across the VGI ecosystem, with cross-repository alignment on schema changes and cryptographic primitives.

In vgi-java, Rusty replaced string-based tag representations with proper ComparisonOperator and PushdownFilterType enums, improving type safety and pushdown filter handling.

The Rust RPC layer (vgi-rpc-rust) received significant attention. A production-hardening pass addressed security concerns across authentication, wire protocol, and HTTP handling, including external fetch operations. Complementing this work, he extracted a generic AEAD seal/open primitive into the vgi_rpc::crypto module for reusable authenticated encryption across the codebase.

In vgi-java RPC transport, the RpcConnection now properly drains end-of-stream markers, ensuring connection reuse survives multiple sequential calls without protocol violations.

The Go implementation (vgi-go) underwent significant schema synchronization. Work included syncing FunctionInfo schema definitions, adding new enum types and AEAD-seal opaque data fields, and renaming transaction identifiers (attach_id/transaction_id → *_opaque_data) for consistency. Additional changes added catalog tags, time/duration typed arguments, HTTP signing keys, and support for dynamic schemas.

Python RPC (vgi-rpc-python) gained a TestConnectionReuse conformance test covering many calls over a single connection—validating the transport fixes. The utils module now handles extension and nested-extension types when generating empty batches. Like the Rust layer, Python also received a generic AEAD seal/open primitive and an ambient current_auth() utility for crypto operations.

Organization infrastructure (vgi-rpc-haybarn/.github) was enhanced with a unified build-status matrix on the organization profile and a new org profile README featuring a hero banner and icon branding.

A new demo-repository was initialized under vgi-rpc-haybarn for demonstration purposes.

Rusty also provided community support, replying to issue #49744 on apache/arrow regarding inconsistent skips in the integration format datagen.

Significant private repository work—39 commits across 6 private repos—rounds out the day.

A productive day focused on cryptographic hardening across the RPC layer and refinements to the core VGI schema and type system.

In vgi-java, Rusty enhanced the FunctionInfo metadata to include supports_batch_index and partition_kind fields, addressing function capability signaling. He also fixed a double-projection issue in TableInitParams that was unnecessarily re-projecting output schemas, and added convenience helpers to the fixture-author utilities in vgi-core for better test authoring ergonomics.

Across the RPC implementations—vgi-rpc-rust, vgi-rpc-go, vgi-rpc-typescript, and vgi-rpc-java—Rusty systematically sealed stream-state tokens with XChaCha20-Poly1305 AEAD encryption. This security upgrade hardens the HTTP transport layer against token tampering. In the Java RPC layer specifically, he also refactored OutputCollector to use a per-emit DictionaryProvider for producer streams, improving dictionary lifecycle management.

The vgi-go repository received a regenerated FunctionInfo definition to reflect the new schema fields added in the Java implementation.

In vgi-rpc-python, Rusty released two versions (v0.16.0 and v0.16.1) with significant test infrastructure improvements and operational fixes. He implemented the same AEAD encryption for stream-state tokens, refined the launcher to drive accept() on a timeout so Linux idle-shutdown actually fires, and hardened the test suite with faulthandler instrumentation at 4-second intervals, stderr capture, thread dump diagnostics on launcher failures, and POSIX-gated socket/OS calls for cross-platform compatibility. The porting guide documentation was also refreshed to reflect the post-AEAD conformance status.

Along with these public contributions, Rusty also made 20 commits across 4 private repositories.

May 12, 2026

Work on vgi-java advanced significantly with test coverage improving from 117/128 to 121/128 passing tests. Key accomplishments include implementing tensor functions (nest_tensor and unnest_tensor variants), fixing the combine semantics in AggregateRunner to preserve source states for segment-tree reuse, and wiring per-column statistics through both catalog and table-function paths. Additionally, he documented why one remaining failure in versioned_tables_impl.test:231 isn't Java-fixable, and threaded per-schema comments through to catalog_schemas in the Worker class.

Infrastructure improvements across both vgi-java and vgi-rpc-java included enabling maven-publish on all Java subprojects, aligning vgirpc coordinates on the farm.query group while maintaining legacy aliases, and adding an HttpServer.Config overload to Worker.runHttp(). A 15-second stop timeout for graceful shutdown was configured in HttpServer, with additional logging of shutdown progress in the worker.

On the duckdb-quack front, Rusty opened four issues addressing protocol and data integrity concerns: capability discovery for request/response size negotiation (issue #117), lack of byte-level caps on responses (issue #116), client catalog name rewriting in view creation (issue #115), and phantom data returned by INSERT RETURNING queries (issue #114). These capture important edge cases in the Quack server implementation.

Private repository work continued with 4 commits across 2 repos.

Daily Activity Summary – May 11, 2026

Rusty logged 86 commits across 7 public repositories and pushed 18 commits to 5 private repos—a day focused on refactoring, test coverage gains, and production readiness across the VGI RPC ecosystem.

vgi-java – Major Refactoring & Test Coverage Push

The bulk of the work landed in vgi-java, where Rusty undertook a sweeping refactoring of the service layer and argument handling. Early commits extracted foundational abstractions: CatalogRegistry to consolidate attach-version state, IpcUnpacker for 1-row IPC request decoding, EnumDict to dedupe dict-encoded field declarations, and VectorProjector for column-by-name projection. He also introduced OverloadResolver and PassthroughTIOFunction as shared bases for fixture implementations.

Argument handling saw significant cleanup. He reworked ArgSpec to reject mixed positional + default patterns, migrated 52 call sites to use named() and varargs() builders, and refactored constant column routing for Struct, List, Map, and FixedSizeList types through a unified argField path. Dictionary-encoded field support was centralized.

The refactoring netted −1,030 LOC and −352 additional LOC on deferred cleanup, while test coverage climbed from 89/129 to 111/129 passing (86% pass rate). Near the end, he documented the test workflow and current state in CLAUDE.md, diagnosing a nested_type_combinations segfault and noting progress from 112 to 114 passing after dict-encoded fixes.

vgi-rpc-rust – 0.1.0 Release Hardening

Prepping for the 0.1.0 release, Rusty added Apache-2.0 licensing with NOTICE attribution, set up a trusted-publishing release workflow, and pinned versions on internal path dependencies. He tightened the API surface and polished the README. Critical fixes included:

• Vendored per-batch custom_metadata and removed the arrow-rs fork dependency
• Blocked IPC schema-message OOM and integrated real Sentry SDK
• Added external extra to CI install for aiohttp (external_fetch tests)
• Synced Python updates: TransportKind, Sentry/OpenTelemetry/Bearer/JWT/external-fetch upgrades, and a fuzz harness
• Fixed linter gates: unused_mut, clippy::err_expect, and items-after-test-module

vgi-rpc-go & vgi-rpc-java – Wire Protocol & Synchronization

In vgi-rpc-go, he synced Python fixes and closed production-readiness gaps, including a Sentry module.

For vgi-rpc-java, the wire layer underwent a major rewrite. IpcStreamWriter now emits dictionary batches between schema and record batch, delegates the WriteChannel for alignment correctness, and Marshalling.castRoot skips casting when schemas differ only in dictionary memory format. The core Ipc{Stream,Writer,Reader} classes were refactored as subclasses of stock Arrow{Writer,Reader}.

vgi-go – Storage Backend Abstraction

A new FunctionStorage interface enables pluggable shared-storage backends. Rusty implemented SQLite storage with per-user default paths and matched Python pragmas, plus a full conformance test suite. He added a Cloudflare Durable Object FunctionStorage client (vgi/storage/cfdo), wired the Worker to the shared backend, and introduced env-driven storage selection via VGI_WORKER_SHARED_STORAGE. The driver was upgraded from the community fork to the official duckdb/duckdb-go.

Community & Housekeeping

Rusty merged cleanup commits in haybarn-httpfs and duckdb-httpfs (removing duplicate crypto.cpp from HTTPFS_SOURCES), ensuring the build remains clean across the ecosystem.

Rusty tackled a variety of integration and compatibility challenges across the DuckDB ecosystem today.

Work on vgi-rpc-typescript focused on improving the Arrow/Flechette implementation with a workerd fallback mechanism, nullable/metadata preservation, and Map coercion enhancements. This commit refines the RPC layer's type handling and runtime robustness.

In ducklake, Rusty opened PR #1139 to standardize SQL cast syntax across the codebase. The change replaces PostgreSQL-flavored :: cast operators with ANSI-standard CAST(...) syntax. This addresses a critical compatibility issue: when inlined-INSERT and filter-pushdown SQL batches are sent to SQLite-backed metadata backends, the :: operator causes parser failures. This fix ensures broader database backend support.

On the duckdb-httpfs extension front, PR #324 introduces per-request cancellation hooks to HTTPFSCurlClient. The implementation adds an optional should_cancel function field to HTTPFSParams, wired into libcurl's CURLOPT_XFERINFOFUNCTION progress callback. This enables extensions to abort in-flight HTTP transfers gracefully, improving resource management and responsiveness.

Rusty also filed issue #22519 on duckdb itself, documenting a progress bar timing issue where updates lag significantly during CTAS and COPY-FROM-SELECT operations until queries near completion. This likely ties into the cancellation work—visibility and control over long-running transfers are essential for a polished user experience.

Beyond public repositories, Rusty made 3 commits across 2 private projects, continuing parallel development efforts.

Rusty focused on low-level transport and worker lifecycle improvements across the vgi-rpc-typescript and vgi-rpc-python codebases, with parallel development on private infrastructure.

On vgi-rpc-typescript, he tackled several interconnected systems. Work on http/dispatch introduced producer-stream externalization accounting and worker-visible budgets—refinements to how the system tracks resource allocation across distributed producer streams. A major addition was the AF_UNIX worker launcher, which implements cross-process locking (serveUnix and related mechanisms) to coordinate worker lifecycle events, replacing or supplementing earlier launcher strategies. He also synchronized type definitions and security fixes with the Python implementation, ensuring TransportKind and other critical structures remain consistent across language boundaries.

Parallel effort on vgi-rpc-python mirrored the TypeScript work: the AF_UNIX worker launcher with cross-process flock coordination landed here as well, maintaining feature parity between the two implementations.

Beyond the public repos, Rusty made 17 commits across 3 private repositories, indicating sustained work on private infrastructure or integration layers.

Four new issues were opened across the public repos, likely capturing follow-up work or edge cases discovered during this sprint.

Rusty spent the day working across two main repositories, with significant activity in private repos as well.

In vgi-rpc-typescript, he focused on improving the HTTP dispatch layer and build tooling. Two key commits touched the request handling pipeline: one that packs resolved input schemas into state tokens during initialization, and another addressing a Web Crypto migration alongside workerd-compatible bundling. These changes suggest work to improve compatibility with Cloudflare Workers while refining how request metadata flows through the system.

Over in ducklake, Rusty opened PR #1126 introducing a VgiMetadataManager for handling pluggable metadata backends. The change routes DuckLake metadata operations to support Durable Objects as an initial implementation, with an architectural question posed to maintainer @pdet about how to handle additional metadata manager implementations. This represents foundational work on making the metadata layer extensible beyond the current Durable Objects approach.

Beyond these public repositories, he made 19 commits across 6 private repositories, indicating parallel work on internal projects alongside the open-source initiatives.

A productive day across multiple DuckDB-related projects, with focus on compatibility improvements and upstream synchronization.

Over in jsonata, Rusty released version 0.1.2 with hardened exception handling and automatic JSON loading. The same day, he opened PR #1879 on community-extensions to sync the extension with the latest commits from the upstream library, ensuring users have access to the latest improvements.

Logging received attention in vgi-rpc-python, where access log output was refined by gating request_data and stream-state tokens to DEBUG level. This reduces noise in production logs while preserving detail for troubleshooting.

Database portability was improved in ducklake through PR #1124, which replaces PostgreSQL-specific ::boolean cast syntax with standard ANSI CAST(... AS boolean) in the metadata manager's stats update query. This change broadens compatibility across different catalog backends that don't support the :: shorthand.

He also surfaced an issue in duckdb (issue #22471) regarding the TableFunction::dynamic_to_string callback being skipped when upstream operators return FINISHED status—such as when a LIMIT clause terminates early. This edge case warrants investigation in the query execution path.

Beyond these public contributions, significant activity occurred across private repositories, with 32 commits distributed across 5 projects.

DuckDB Testing Infrastructure

Rusty focused on expanding the testing capabilities of duckdb/duckdb, opening PR #22461 to introduce per-test-file parallelism to the unittest binary. The change adds three new command-line flags: --jobs N for concurrent execution, --shard-index K and --shard-count N for distributed test sharding. When parallelism is enabled, the binary operates as a coordinator that enumerates matching tests via a filter, then spawns child processes to run test files concurrently. This enhancement allows the test suite to scale across multiple cores and machines, reducing overall test execution time.

Across the day, he pushed 33 commits to the repository, reflecting substantial work on the testing infrastructure redesign. Beyond the visible open-source contribution, additional activity on private repositories suggests parallel work on related or complementary systems.

Today focused on community engagement with the Apache Arrow project. Rusty replied to issue #49744 on apache/arrow, which addresses inconsistent skips in datagen functionality. The issue involves integration and format concerns, suggesting work around data generation tooling and how skip behavior is handled inconsistently across different scenarios. By providing feedback on this integration issue, he contributed to the ongoing effort to improve Arrow's data generation and testing infrastructure.

DuckDB Community Extensions: Rusty completed a maintenance cycle across multiple community extensions, bumping versions for airport, datasketches, bitfilters, geosilo, textplot, and lindel. This work was followed by six corresponding PRs on duckdb/community-extensions that update each extension to their latest commits — PR #1862 through PR #1866. These updates keep the extension ecosystem current with upstream development.

DuckDB Core: The focus shifted to performance and correctness issues in the query engine. He opened PR #22445 to fix a correctness regression in Arrow extension propagation for nested appenders. The issue (opened as #22444) involved BOOLEAN children of UNION types becoming corrupted under arrow_lossless_conversion = true; the fix addresses the same root cause across all container appenders (UNION, STRUCT, LIST, MAP, ARRAY) and targets the v1.5-variegata branch.

Parallel to the correctness work, Rusty tackled aggregate function performance with two optimization PRs. PR #22438 makes segment-tree fanout configurable per-aggregate, allowing tuning when combine and update operations use RPCs instead of in-process calls. PR #22437 defers segment-tree combine flushes across same-level parents in window functions, addressing the issue where the aggregate combine buffer is underutilized.

Apache Arrow: He provided community support by replying to issue #36388 regarding pyarrow.repeat returning invalid arrays when a chunked array is required.

Beyond the public repositories, Rusty also made 6 commits across 2 private repositories.