Embracing nanoarrow along with Arrow C++

I’ve been adding Apache nanoarrow to my DuckDB work alongside Arrow C++. I want to reduce binary size and build times, and take advantage of opportunities to use less memory. Memory use is particularly important in WebAssembly: an extra copy of an IPC buffer takes space that the database could otherwise use to process data.

For code that only needs the C Data Interface and IPC read/write, nanoarrow gives me a smaller library to build and ship. The build time matters while I’m developing the extensions; the binary size and memory use matter when someone runs them, especially in the browser. The reader’s allocations and buffer lifetimes are part of that decision too.

To use it in my IPC code, I need the message metadata and dictionary handling that my RPC protocol relies on. I’ve been adding those features, investigating the reader’s performance, and fixing the compatibility problems that show up when other Arrow implementations consume the output. It continues the DuckDB and Arrow compatibility work I wrote about earlier this year, with more of the work now happening in nanoarrow.

Message metadata

The first thing I needed was Message.custom_metadata. My RPC protocol uses it for message types, cache instructions, provenance, and references to payloads stored elsewhere. nanoarrow supported schema and field metadata, but couldn’t yet read and write the metadata attached to an individual IPC message. I opened an issue in August and then added that support.

The initial patch worked at the encoder and decoder level. A follow-up reduced the API surface before release: callers can use nanoarrow’s existing packed metadata representation, or visit entries without copying them. Both changes have merged.

Using it through ArrowIpcWriter needed another patch because the writer keeps its encoder private. Without a setter on the writer, I’d have had to drop down to the encoder and reproduce the framing and dictionary handling myself. The writer metadata setter has now merged too.

That patch also has to account for dictionaries. The writer can emit a DictionaryBatch before the record batch, so immediately passing metadata to the encoder could attach RPC state to the wrong message. I made the writer hold it until the record batch is encoded, then clear it. The test writes a dictionary in between to check that behavior.

Dictionary support

DuckDB ENUM columns arrive as dictionary arrays, and my extension writes batches individually to a worker process. nanoarrow’s stream writer already knew how to write a batch’s dictionaries, but that operation wasn’t available to callers writing one batch at a time. My batch writer, which has merged, exposes that step, so the caller doesn’t have to work out the dictionary IDs and emit each dictionary itself.

The broader dictionary work grew into several patches. I originally had array appending, delta decoding, and writer changes together. Review on that PR led me to split them up, since each was useful on its own.

The appender became ArrowArrayAppendStorageFromArrayView(). It copies existing Arrow storage into an array being built, including nested values. A lot of the review was about getting slices, nulls, and run-end encoding right. That also produced a separate 16-bit chunk resolver, giving 16-bit run ends the same binary-search helper already available for the wider types. Both patches have merged.

With the appender available, I could use it to decode dictionary deltas. If a stream has already sent a dictionary and later sends additional values, the decoder now combines them instead of failing. It builds a new dictionary for each delta so batches the caller is still holding keep their existing references. That copies the dictionary, but keeps ownership straightforward. The patch is merged; value types the appender doesn’t support still return an error.

The dictionary writer changes have also merged. These avoid re-emitting a dictionary when consecutive batches share the same dictionary buffers, and fix nested dictionary and file-writing behavior. The comparison uses buffer identities, so the writer has to keep the previous batch alive until it has compared the next one. Otherwise an allocator could reuse an address for different data and make the dictionary appear unchanged.

The batch-at-a-time writer needs a different approach because its input is borrowed. In #979, I keep a copy of each dictionary and compare the next batch’s values with it. That lets the writer skip an unchanged dictionary even when the producer builds a fresh array for each batch. It also means paying for a saved copy, which matters when evaluating the memory cost.

For dictionaries that grow, skipping unchanged values isn’t enough. #980 checks whether the new dictionary extends the previous one and writes just the appended values as a delta. Both PRs are still drafts, building on the writer work above.

I tested them with 1,000 batches of 2,048 rows and a string dictionary starting at 10,000 values. Skipping an unchanged dictionary reduced the serialized output from 258.5 MB to 8.6 MB, and writing time from 126 ms to 23 ms. When the dictionary grew by 16 values per batch, delta writing reduced the output from 458.3 MB to 9.2 MB, but took 346 ms compared with 205 ms for writing complete dictionaries. These are native Release measurements on Linux aarch64, best of three runs. They measure writing time and output size; keeping and comparing dictionaries adds work and memory of its own.

Reader performance and memory

For the WebAssembly build, Arrow JS produces IPC streams in the same browser page and hands them to DuckDB. I care about both how quickly the reader handles those streams and how much memory it needs while doing so. After switching readers, I measured an insert slowdown and found that full validation accounted for a substantial part of it.

ArrowIpcArrayStreamReader always used full validation, including walking string and list offsets and dictionary indices. The lower-level decoder already let callers choose a validation level. I exposed that option on the stream reader, which has now merged. Passing no options still gives full validation. Choosing DEFAULT skips some checks; the test includes an invalid dictionary index that FULL rejects and DEFAULT accepts.

I also found that reading from an input buffer copied each message body into another allocation, even though the stream already owned the bytes. My in-place reader patch lets the reader reference those bodies directly when using shared buffers.

Avoiding that allocation is particularly useful in WASM, where I’d like to leave as much memory as possible available for the data and query execution. There is an ownership tradeoff: a decoded array can keep the entire input buffer alive. The memory behavior therefore depends on how long the arrays live as well as how much the reader copies.

I measured the two changes separately and together with a 23 MB in-memory stream containing one million rows of int32, double, and string values in 16 batches. These are native Release builds on an Apple M3, taking the median of 50 runs:

These timings cover reading the resident IPC stream, before the additional UTF-8 and view validation work described below. Browser execution, transport, and inserting the arrays into DuckDB have additional costs. Removing the copy and reducing validation leaves very little work in this particular reader benchmark.

The reader PR is still under review, including discussion of a more general input-stream interface for accessing existing buffers. Configurable validation has merged; the in-place results describe the proposed implementation.

Full validation

I’ve also been checking what FULL actually validates. It was documented as the level for untrusted input, but didn’t check whether string values contained valid UTF-8. My open #981 adds that check for string and large_string, without introducing a dependency. It checks value boundaries too: a UTF-8 character split between two strings is invalid even if concatenating their bytes would produce valid text. Null values are skipped, and sliced arrays only validate the values in the slice.

Binary and string views needed bounds checks before I could safely validate their contents. A view could refer to a variadic buffer that didn’t exist, or to bytes outside that buffer, and still pass FULL. My draft #982 checks sizes, buffer indices, and byte ranges, and adds UTF-8 validation for string views. It depends on #981. Adding those checks also exposed an internal array view that wasn’t being given the variadic buffers when finishing a newly built array; that is fixed in the same patch.

These changes make FULL reject inputs it previously accepted. Lower validation levels are unchanged, and the extra checks have a cost, which I’ve measured separately in the PRs.

Fixes from testing across implementations

Alongside those features, I’ve been running cross-language conformance checks. The .NET reader caught a problem in the IPC writer’s null counts. The C Data Interface allows null_count = -1 when the count hasn’t been computed, but IPC needs the actual count. nanoarrow was writing the -1 unchanged. It could also write zero for a null-type array, where every entry is null.

Arrow C++ tolerated both: it could recompute an unknown count, and its null-array implementation corrected the other one. Apache.Arrow 23.0.0 for .NET rejected them. I patched the encoder to compute unknown counts and use the array length for null-type arrays. The test checks [1, null, 3] and a three-element null array: the encoded counts are now 1 and 3, instead of -1 and 0. That fix is merged.

Empty arrays have needed attention on both sides of the IPC boundary. I first hit this with a MAP column containing one NULL, which produced an empty child array during a worker round-trip. After nanoarrow decoded it, Arrow C++‘s C Data importer read offsets[0] from a buffer with no offset bytes. Depending on the surrounding memory, validation would fail with a negative offset.

The merged fix supplies a zero offset of the appropriate width when finalizing an empty array. It applies at DEFAULT validation or higher; lower levels skip that finalization step.

I’ve since patched the encoder too. IPC permits a zero-length offsets buffer for an empty array, but some consumers still expect one offset. An empty string array written by nanoarrow could be read by Arrow Go and then panic when the caller accessed ValueOffsets(). The new patch writes a single zero offset, as Arrow C++ does. It has merged, and complements the earlier fix for arrays decoded by nanoarrow.

I also fixed metadata truncation at NUL bytes. The encoder used a FlatBuffers helper that stopped at the first NUL despite having an explicit length. A key such as "field\0key" came back as "field", and a value beginning with NUL came back empty. Switching to the length-exact helper fixed schema, field, and message metadata. That patch has merged.

Tests and remaining work

While testing these changes, I kept hitting the same integration-test failure on unrelated branches. Two tests shared a temporary filename, so parallel runs could overwrite or remove each other’s input. My patch gives each test its own file. In the targeted reproducer, failures went from 199 of 200 runs to zero of 200. I also reported a comparison gap where metadata could disappear from nested child fields without the integration tests noticing.

The larger outstanding piece is metadata on dictionary value types. The C Data Interface can carry it separately from the parent field, but preserving it in IPC needs a proposed Arrow format change. I’ve implemented the nanoarrow side, including round-trip tests, and the PR remains a draft pending format approval. This matters for dictionary values carrying extension types.

The writer APIs for the batch-at-a-time path my extension needs have now merged: write the dictionaries, attach the RPC metadata to the record batch, and write the batch. With that path covered, I can use nanoarrow for this part of the extension and work toward the smaller binaries, shorter builds, and lower WASM memory use that brought me to it. The dictionary-skipping and delta drafts, the in-place reader, and the stricter validation are what’s left in review.

PRs and issues

Status as of October 25, 2026. Merged refers to the upstream repository; it does not necessarily mean the change is in a released version.