A Big Step Forward for S3 Compatibility in Auto Drive
The first cut of Auto Drive’s S3 endpoint gave developers a familiar way to push and pull bytes against the DSN. As more people pointed S3 tooling at Auto Drive, the gaps became clear: buckets weren’t really buckets, listings didn’t fold into virtual directories, ETags returned CIDs rather than MD5 hashes, and delete requests came back with a generic 404.
This release closes those gaps and exposes the underlying logic as reusable building blocks for anyone building their own S3 layer on top of the DSN.
What’s New
Four backend changes shipped under the S3 Compatibility milestone:
- First-class bucket support with a backwards-compatible schema migration and a
ListBucketsendpoint ListObjectsV2withprefix,delimiter,max-keys, andcontinuation-token(the request shape used by virtually every S3 client)- MD5 ETags on
PutObject,HeadObject, and multipart uploads, in the standard AWS format - Explicit 403 Forbidden on
DeleteObject, with a message that explains why rather than leaving callers to interpret a generic 404
Alongside the backend work, the reusable parts of this logic now live in @autonomys/file-server, a package in the TypeScript Auto SDK, so other S3-compatible layers built on Autonomys storage can pick them up directly.
Connecting an S3 Client
Nothing about how you authenticate has changed. Auto Drive’s S3 endpoint is reachable using your existing Auto Drive API key, presented as the AWS access key id when you configure an S3 client. The endpoint URL, region setting, and path-style addressing are all covered in the S3 layer docs, which we’ll be refreshing alongside this release with examples of the new bucket and listing behaviour.
If you haven’t already, you can grab an API key from ai3.storage. The free tier is enough to test everything in this post.
Buckets, For Real This Time
Previously, every S3 key lived in a single flat namespace. A PUT /my-archive/file.txt stored the literal key my-archive/file.txt. That worked, but it meant ListBuckets had nothing meaningful to return and any client that expected per-bucket isolation was effectively pretending.
The new behaviour treats the first path segment of every S3 request as the bucket name. So my-archive/file.txt is now bucket my-archive, key file.txt. Anything uploaded under the old scheme (a single-segment key like test.txt, with no slash to split on) is migrated into a default bucket so nothing existing breaks.
| Request | Resolved bucket | Resolved key |
|---|---|---|
GET /test.txt |
default |
test.txt |
GET /my-archive/file.txt |
my-archive |
file.txt |
GET /my-archive/sub/file.txt |
my-archive |
sub/file.txt |
The URL structure clients send is unchanged, so this is invisible to any S3 client already pointed at Auto Drive. The object_mappings table gains a bucket column and a composite (bucket, key) primary key, and ListBuckets returns the set of distinct buckets in standard <ListAllMyBucketsResult> XML. Existing clients keep working without changes.
ListObjectsV2 With Real Directory Folding
The biggest functional gap was object listing. Without ListObjectsV2, S3 tools that want to browse, sync, or mount a bucket have nothing to walk. The new implementation supports the four request parameters that matter in practice:
prefixto scope the listing to keys starting with a given stringdelimiterto fold everything after the delimiter into virtual subdirectories (CommonPrefixes)max-keysto cap the page size (default 1000)continuation-tokenfor stable pagination across pages
A request like GET /my-archive/?list-type=2&prefix=2026/&delimiter=/ will return the objects directly under 2026/ plus a CommonPrefixes entry for each virtual subfolder beneath it. That’s the request shape standard S3 SDKs and CLI tooling use when walking a “directory tree”, and it now works.
MD5 ETags for Real Integrity Checks
S3 clients treat the ETag header as an MD5 hash of the object’s bytes. Auto Drive previously returned the object’s CID, which is meaningful inside Auto Drive but confuses any client that wants to verify integrity by recomputing the MD5 locally and comparing.
The new behaviour computes MD5 at upload time, stores it alongside the CID, and returns it as the ETag header on PutObject and HeadObject responses in the standard quoted-hex format:
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Multipart uploads return the AWS-style composite ETag, which is the MD5 of the concatenated part MD5s, suffixed with the part count:
ETag: "9e8a5f1f4e3a0c8b7d1a9c2e6f4b1a7d-12"
ListObjectsV2 listings also surface the same MD5 ETag for each object, matching what HeadObject returns, so clients that verify checksums from a listing no longer have to fall back to a HeadObject per row. Multipart assembly was also hardened along the way: parts that arrive out of order now produce a correctly assembled object instead of a corrupted one.
This is the format that integrity-checking tooling and standard S3 SDKs expect, and it makes Auto Drive a drop-in destination for workflows that previously required custom checksum-bypass flags.
To be explicit: this is a breaking change to the value returned in the ETag header, and we coordinated with integration partners ahead of the release so nobody was surprised. The CID hasn’t gone away. It is now returned in the standard S3 user-metadata header x-amz-meta-cid so any client that was relying on the previous behaviour can still read it directly.
Cleaner Error Semantics
Three error-handling fixes shipped alongside the headline features, all of which make Auto Drive behave more like a well-behaved S3 endpoint when something goes wrong.
403 Forbidden on DeleteObject. Auto Drive storage is permanent by design. Previously, delete requests hit the Express router’s default handler and returned a generic 404 or 405. The new handler returns 403 with a clear message: Auto Drive storage is immutable - objects cannot be deleted from the Autonomys DSN. This is the standard pattern for immutable S3-compatible stores and turns a confusing error into a self-explanatory one.
404 Not Found on missing keys. GetObject and HeadObject against an absent key were returning 500 because the internal ObjectNotFoundError was being mapped through the generic error handler. The fix makes the error class extend the HTTP error base type so the existing handler emits a proper 404.
501 Not Implemented on unsupported operations, and a safety fix. The dispatch logic resolved S3 operations from query parameters alone, without considering the HTTP method. That meant safe requests like GET /key?uploadId (ListParts) were being misrouted to CompleteMultipartUpload and silently finalizing live uploads. A handful of other unsupported operations (CopyObject, AbortMultipartUpload, ListMultipartUploads, POST-object) were also being misrouted to mutating handlers. The fix resolves operations method-first, then disambiguates by query param, and returns a clean 501 with a standard S3 <Error><Code>NotImplemented</Code> body and no side effects.
Reusable Building Blocks in Auto SDK
While building the listing and ETag work, it became clear that several pieces of logic had no reason to live exclusively in the Auto Drive backend. The folding algorithm, the continuation-token construction, the multipart ETag computation: none of these are Auto Drive-specific. Any S3-compatible layer built on Autonomys storage will need exactly the same logic.
Rather than have downstream implementations reinvent it (and the edge cases around it), the pure pieces have been extracted into a new s3/ module in the Auto SDK’s @autonomys/file-server package. The exports cover both halves:
buildListResult,computeListObjectsDbLimit, andfinalizeListObjectshandleListObjectsV2folding, pagination, and the full-batch truncation overridemd5Hex,formatETag, andmultipartETagcover the ETag side, including AWS-style composite multipart ETags
The Auto Drive backend now consumes these exports from the latest @autonomys/file-server in the Auto SDK release, so the helpers and the production code stay in lock-step.
What’s Next
Some of these design decisions were driven by a specific integration we’ll have more to share about shortly. The enhancements stand on their own, but they also make a class of off-the-shelf workflow viable that wasn’t before. Keep an eye on the forum.
Get Started
The new endpoints are live on the Auto Drive S3 surface. If you’ve been writing against the S3 API already, no client changes are needed to pick up bucket routing or MD5 ETags. Just send requests as usual. To exercise the new listing and bucket endpoints, point any standard S3 SDK or CLI at your Auto Drive instance and try:
ListBucketsCommandto enumerate your bucketsListObjectsV2CommandwithPrefixandDelimiterto walk a bucketHeadObjectCommandto inspect an object’s MD5 ETag
For a refresher on endpoint setup and authentication, head to the S3 layer docs. If you’re building an S3-compatible layer of your own on top of Autonomys storage, the new s3/ helpers are available in the latest Auto SDK release.
As always, come find us on Discord if you have questions or if you’re building something we should know about.