Choose a method
| Method | Best for | Needs | Resumes |
|---|---|---|---|
| aria2 manifest | Whole collections, thousands to millions of files | Researcher account or higher | Yes, per file |
| curl | A handful of files, scripting | None for Open tier | Yes with -C - |
| rclone | Syncing and re-syncing a directory | Researcher account or higher | Yes, by checksum |
| Transfer endpoint | Institution-to-institution copies above 100 TB | Associate or Full membership | Yes, managed |
Build a command
Select a collection, a tool and a number of connections. The generated command uses your tier's limits as the upper bound; asking for more than your tier allows results in HTTP 429 responses with a Retry-After header.
Tokens and signed manifests
Create a token in the account console under Tokens. A token carries your tier and expires after 90 days unless you renew it. Export it as NM_TOKEN and never put it in a URL: manifests contain signed file addresses valid for seven days, so they are safe to hand to aria2 but should not be shared.
A manifest entry has three parts: the mirror list, the output path and the checksum. The mirror list is ordered by the region nearest to the client at the time the manifest was generated.
https://se2.enotcloud.com/files/clm-1120/surface/t2m/1999.zarr.tar?sig=7Hq2xWk1 https://se2.enotcloud.com/mirror/nm-eu-central/clm-1120/surface/t2m/1999.zarr.tar?sig=Pd91cLmA
out=surface/t2m/1999.zarr.tar
checksum=sha-256=9f2c1a6e4b7d0058e3a1c9d2f64b8e07a5d3c1b9e82f7a604d5c3b1e9a8f7c26
Choosing connection counts
More connections do not always help. Throughput is limited by your tier and by disk speed, and every connection to a single region shares one fair-share queue. These values have worked well for the transfer sizes we see:
- Many small files (imagery tiles, FASTQ):
-j 16files in parallel with-x 4connections each. - Few large files (CRAM, Zarr archives):
-j 4 -x 16 -s 16. - Spinning disks: keep total parallel writers below 12; use
--file-allocation=noneto avoid a long pre-allocation pause. - Shared links: set
--max-overall-download-limitto leave room for colleagues.
Verifying and repairing
Each directory contains a SHA256SUMS file. Run sha256sum -c SHA256SUMS --quiet and re-fetch any file that is reported. With aria2, --check-integrity=true does this during the transfer and repairs only the damaged chunks.
Transfer endpoints
Members can request a Globus-style endpoint, which moves data directly between our storage and yours without passing through a client machine. Endpoints retry automatically for up to 72 hours, encrypt in transit with TLS 1.3, and send a completion message with a checksum report. Setup takes about two working days; write to transfers@se2.enotcloud.com with your institution's endpoint name.
Limits and fair use
- HTTP 429 means you exceeded connections or throughput for your tier; wait for
Retry-Afterseconds. - HTTP 403 with
expired_signaturemeans the manifest is older than its lifetime; request a new one. - Do not mirror the archive to redistribute it commercially; the licence on each collection says whether that is allowed.
- For repeated full-archive copies, ask for a data node agreement instead.