How Auto Drive Became an Official rclone Provider

[Jim-Autonomys]

From 22 ignored tests to 3: how Auto Drive became an official rclone provider

rclone now supports Auto Drive as a named storage provider. rclone’s lead maintainer merged it on 16 October (rclone#10093), and it’s included in rclone v1.76.0, released on 6 November. Update rclone, pick “Autonomys Auto Drive” from the setup menu, enter your API key, and copy, sync or back up files to permanent storage on the Autonomys Network.

In June, when we announced the first round of S3 compatibility work, we said some of its design decisions were driven by an integration we’d share more about. This is the story of that integration. The rclone change itself is 110 lines. Nearly all the work happened on Auto Drive, finding the places where “S3-compatible” wasn’t yet quite true.

Why rclone, and why its tests were the bar

rclone is the most widely used command-line tool for syncing files to cloud storage. Think of it as rsync for the cloud: one tool and one set of commands for dozens of storage services.

Many of those services speak S3, the API Amazon created for its Simple Storage Service, which has become the common language of object storage. In S3, files are objects, stored in buckets under names called keys. Auto Drive already had an S3-compatible API, so rclone could reach it through a generic “Other” setup and some hand-tuned settings. A named provider bakes those settings in.

Getting listed meant meeting a high bar. rclone ships an integration test suite that drives a real storage service through hundreds of operations: uploads of awkward sizes, listings, metadata round-trips, copies, deletes and cleanup. It’s a demanding, independent check of how closely an S3 API behaves like the real thing. Our goal was to pass it, skipping (in rclone’s terms, ignoring) only tests we could justify.

Where we started

A first round of work in May had built the basics: real buckets, standard listings, MD5 checksums and clean errors. Even so, our first run of rclone’s suite on 18 June failed in five distinct ways:

• Deletes. Auto Drive refused them, so every test that tidies up after itself failed.
• Copies. Server-side copy, where the storage service duplicates an object without the client uploading it again, wasn’t implemented.
• Metadata. Only the content type was stored. Modification times and other headers came back empty.
• Listings. Keys containing control characters made the listing call fail with a server error.
• Large uploads. Files that rclone uploaded in parts were recorded with the wrong checksum.

Some of these were plain bugs. Others were about what permanent storage means in a protocol designed around deletion.

The first fixes: clients disagree on the details

Large files go to S3 as a multipart upload: sent in parts, then get stitched together by a final XML message listing each part’s ETag. An ETag is a fingerprint of the content, usually its MD5 hash. The AWS SDK for Go, which rclone uses, writes the quote marks in that message as ", and other SDKs don’t. Auto Drive’s hand-written parser missed this, so every multipart upload from rclone was recorded with the ETag of an empty file. A real XML parser fixed it, with tests covering both styles (#759, #761).

Listings had a similar edge. S3 allows almost any character in a key, including invisible control characters, but some of those can’t legally appear in XML, the format S3 uses for listings. S3’s answer is an opt-in, encoding-type=url, which asks the server to percent-encode keys. Auto Drive now honours it, and returns a clean 400 InvalidArgument error naming that parameter when a client hasn’t asked (#758, #760). One twist: rclone only falls back to encoding when it can’t parse a response, and a clean error parses fine. So the Autonomys provider always asks for encoding.

Two more changes set things up for later. S3’s Object Lock lets a service declare that objects can’t be changed or deleted, and backup and compliance tools check for it. Auto Drive is permanent by design, so it now says so in S3’s own terms: Object Lock on, in its strictest COMPLIANCE mode, retained until 31 December 9999, the latest date clients can safely parse (#757, #764). And the S3 API moved to its own hostname, s3.auto-drive.autonomys.xyz (#762, #763).

In the next run, the modification-time failures, server errors and corrupted multipart ETags were all gone.

Making permanent storage fit S3

The biggest problem was conceptual. rclone’s tests, and real users, expect to be able to delete things. On Auto Drive, nothing written to the network is ever erased.

The answer was to separate what you see from what is stored (#778). Deleting an object now hides it and returns success, instead of refusing. The data stays on the network, as promised. The same change made server-side copy a re-mapping: a second name for the same stored content, with nothing duplicated. It also started storing modification times and added a way to abort an unfinished multipart upload.

The rest followed one principle: describe what Auto Drive actually does, honestly, in S3’s terms.

• Buckets are implicit (#779). The first upload creates a bucket, so there are no endpoints for creating or deleting one, and rclone is told not to try.
• Each user has their own namespace (#782). Two people can both have photos/cat.jpg.
• Metadata comes back exactly as stored (#786, #788). Express, the web framework behind the API, had been quietly adding ; charset=utf-8 to text content types. Cache-Control, Content-Language, Content-Disposition, Content-Encoding and custom metadata now round-trip too.
• Versioning is reported honestly (#781, #789). Every version is kept anyway, so the API reports versioning is enabled and uses each version’s content ID (CID) as its version ID.

We chose not to support one feature. rclone link creates presigned URLs, temporary links signed with your credentials, and these don’t fit Auto Drive’s API-key authentication. The public gateway (gateway.autonomys.xyz/file/<CID>) is already a better way to share a file. Our tracking issue, #787, records the remaining work and the reasoning behind each ignored test.

Testing storage that never forgets

Permanent storage changes how you test. Every test upload is permanent, and because the tests write random data, nothing deduplicates. A full run of the rclone suite writes about 360 MiB, almost all of it from the tests that exercise chunked uploads.

rclone runs many providers through its suite every day. For Auto Drive, that would add about 130 GiB a year of data that can never be removed, so we decided not to enrol. When rclone’s maintainer merged the provider, he independently reached the same view. Instead, we used rclone’s -short mode, which skips its heaviest tests and brings a run down to about 11 MiB.

There was a subtler cost too. When a test fails before cleaning up, its leftovers appear in every later listing in the run. One real failure shows up as many apparent ones, and that shaped the whole ignore list.

Two subtle bugs

In August, two bugs took real detective work.

In the first, reading an object straight after a multipart upload sometimes failed part-way through with an unhelpful stream reset. Matching multiple backend services’ logs showed why. A background job moves a new upload’s data from one database table to another, and a read that landed mid-move couldn’t find a piece in either. The fix reads both tables in one consistent query, and if data isn’t available yet, returns a retryable 503 before sending any bytes (#815, #819).

The second looked simpler. An object stored with Content-Encoding: gzip came back decompressed: 55 bytes instead of 70, with no length header and a “weak” ETag, one marked as only approximately matching the content (#814). Emil’s analysis showed the application was returning exactly the stored bytes. Something between Auto Drive and the client was changing them.

Who is changing the bytes?

Two layers sat between Auto Drive’s API and the internet: nginx, the web server that routes requests, and Cloudflare, a CDN (content delivery network) that speeds up and protects websites.

nginx was guilty for some objects. Its gzip filter compressed text responses on the fly and weakened their ETags, so we turned it off for the S3 hostname (#822). To find the rest, we sent the same request three ways: straight to the application, through nginx, and in from the public internet. Only the last hop changed the bytes. Cloudflare’s documented opt-out, Cache-Control: no-transform, stopped it compressing responses but not decompressing them.

Everything we found the CDN doing is normal and helpful for a website. For an object store, where the bytes and headers you get back must match what was stored exactly, each one is a bug:

• Decompressing stored gzip objects for clients that hadn’t asked, which also made range reads (fetching part of a file) come back empty.
• Compressing responses on the fly, which dropped the length header and weakened the ETag, breaking MD5 checks.
• Rewriting email addresses and http:// links inside stored HTML files.
• Stripping the length and ETag from metadata-only (HEAD) requests for HTML files, so rclone concluded its own uploads were corrupt and deleted them.
• Capping uploads at 100 MB.

A Cloudflare rule that turned off HTML rewriting fixed the body changes but not the stripped headers (infra#584). That settled it: take the S3 hostname, and only that hostname, off the CDN. The website and gateway stay behind it.

We made the move in small steps, each confirmed before the next. First, the S3 hostname got its own rate and connection limits in nginx, which return 503 rather than 429 when throttling, because the AWS SDK retries a 503 but not a 429 (#850). Next came a dedicated address that serves only the S3 API (infra#585, #858), then the DNS switch and removing the proxy (infra#586, infra#587).

Afterwards, HEAD requests returned the right length and an ETag equal to the MD5. HTML uploads survived, rclone check compared files by MD5 again, range reads of gzip objects returned the exact stored bytes, and rclone’s metadata test passed.

Big files

Two size limits surfaced late. rclone sends files under 200 MiB in a single request, but Auto Drive accepted at most 100 MiB per request, so uploads in between failed, and only after the whole file had been sent. S3 servers normally stream uploads rather than buffering them as Auto Drive does, which is why no other rclone provider had needed a size setting. Rather than add one to rclone, we raised the server limit to 200 MiB to match rclone’s default (#878).

Reads over 100 MiB failed too, because S3 reads went through the anonymous download path, which is capped at 100 MiB. They now go through the owner’s account (#885). During code review, we changed how these reads are charged. The first version counted every read against the owner’s download allowance, a lifetime free-tier budget that would soon run out. Now reads up to 100 MiB stay free, and larger reads are charged.

We checked it live: a 150 MiB file went up in a single request and came back intact.

From 22 ignored tests to 3

By early October the suite passed with 22 tests ignored. Four changes brought that down to three.

• Taking S3 off the CDN let rclone’s metadata test pass.
• The read-after-write fix let the chunked-upload writer test pass, four runs out of four. Seven listing tests that had only failed on its leftovers passed with it.
• Switching off S3 purge in the provider. rclone’s purge command removes a directory and everything in it, and its S3 version deletes every version of every object, which permanent storage always refuses. The provider switches that feature off, the same way rclone already switches off server-side copy for some providers. rclone then falls back to its generic purge, which hides the current objects. That cleared both purge tests, a test that cleans up with purge, and five tests that had failed on their leftovers.
• A one-line nginx fix (#872). A pattern in nginx’s internal rewrite of S3 paths used ., which doesn’t match a newline. Keys containing a newline were misrouted, and a trailing newline was silently dropped, so the request reached a different key.

The three that remain are things Auto Drive deliberately doesn’t do. Two directory-removal tests expect “delete this bucket” to report “not empty” or “doesn’t exist”, which can’t happen without bucket endpoints. The third only runs without -short: it puts a NUL byte in a key, and web servers reject that in a URL before the application sees it. Minio and Google Cloud Storage ignore the same test.

Upstream

The rclone change is one commit: a provider definition, a six-line code block that switches off purge and bucket checks, and documentation with a real rclone config session, following rclone’s guide for new providers. The provider uses path-style requests and copies in a single request, because copying by parts isn’t supported. It reports Object Lock as something clients can’t configure, and it never checks for or creates buckets, since a create-bucket request would otherwise be stored as an empty object named after the bucket.

Nick Craig-Wood, rclone’s lead maintainer, approved and merged it the day after it was opened, unchanged and with every check green.

What we learned

• Permanent storage changes how you test. Every run costs real storage, every failure leaves debris, and a cheap test profile is essential.
• S3 needs byte-exact delivery. Web servers and CDNs optimise responses in ways that suit websites and break an object store’s guarantees. Comparing the same request at each layer settled weeks of uncertainty in an afternoon.
• Different clients produce different bytes. The multipart bug only appeared with the Go SDK’s escaping, so testing with one client isn’t enough.
• Fix things in the right place. Server problems were fixed on the server. That kept the rclone change small and built on patterns rclone already uses, which is why it merged quickly.

Using it today

You’ll need rclone v1.76.0 or later. If you installed rclone from rclone.org, rclone selfupdate will get you there; package managers may take a little longer to catch up. Run rclone config, choose s3, then Autonomys, and enter your Auto Drive API key as the access key. The secret key can be any value. You’ll end up with:

[autodrive]
type = s3
provider = Autonomys
access_key_id = YOUR_API_KEY
secret_access_key = ANY_VALUE
endpoint = s3.auto-drive.autonomys.xyz

On older versions of rclone, use provider = Other with endpoint = <https://s3.auto-drive.autonomys.xyz> and no_check_bucket = true. Either way, use the s3.auto-drive.autonomys.xyz hostname. The older public.auto-drive.autonomys.xyz/s3 path still goes through the CDN, quirks and all. You can get an API key at ai3.storage, and the full setup walkthrough is in rclone’s documentation.

A few things are still open. We’ve prototyped a general rclone change that would let purge work on COMPLIANCE-locked buckets for any provider, and may propose it upstream. We may also refine how reads over 100 MiB are charged, for example charging only for the bytes actually sent.

Thanks and references

@Emilios diagnosed the shrinking gzip file and built soft delete, metadata fidelity, honest versioning, the stored-bytes and XML error fixes (#817) and the read-after-write fix. Nick Craig-Wood reviewed and merged the provider. Thank you both.

rclone: rclone#10093, merged 16 October 2026, released in v1.76.0 on 6 November 2026.

Auto Drive issues: #757, #758, #759, #762, #781, #786, #787 (tracker), #814, #815.

Pull requests, in order:

Pull request	What it did
auto-drive#760	Control-character keys in listings
auto-drive#761	Multipart XML parser
auto-drive#763	Dedicated S3 hostname
auto-drive#764	Object Lock contract
auto-drive#778	Soft delete, copy, abort, modification times
auto-drive#779	No bucket endpoints (decision)
auto-drive#782	Per-user namespace
auto-drive#788	Metadata fidelity
auto-drive#789	Honest versioning
auto-drive#817	Stored bytes and XML errors
auto-drive#819	Read-after-write
auto-drive#822	nginx gzip off for S3
auto-drive#850	S3 throttling
infra#584	CDN rule for HTML rewriting
infra#585	Dedicated address
auto-drive#858	S3-only address
infra#586	DNS switch
infra#587	Proxy off
auto-drive#872	Newlines in keys
auto-drive#878	200 MiB request limit
auto-drive#885	Large-object reads