Skip to content
All regions operational
NorthmereData Commons
Home/Documentation/Bulk download

Bulk download

Moving tens of terabytes is mostly about not losing progress. This guide covers the four supported methods and how to pick connection counts that your tier will accept.

Updated 14 October 2026Tested with aria2 1.37, rclone 1.68 and curl 8.x

Choose a method

MethodBest forNeedsResumes
aria2 manifestWhole collections, thousands to millions of filesResearcher account or higherYes, per file
curlA handful of files, scriptingNone for Open tierYes with -C -
rcloneSyncing and re-syncing a directoryResearcher account or higherYes, by checksum
Transfer endpointInstitution-to-institution copies above 100 TBAssociate or Full membershipYes, managed

Build a command

Select a collection, a tool and a number of connections. The generated command uses your tier's limits as the upper bound; asking for more than your tier allows results in HTTP 429 responses with a Retry-After header.

Tokens and signed manifests

Create a token in the account console under Tokens. A token carries your tier and expires after 90 days unless you renew it. Export it as NM_TOKEN and never put it in a URL: manifests contain signed file addresses valid for seven days, so they are safe to hand to aria2 but should not be shared.

A manifest entry has three parts: the mirror list, the output path and the checksum. The mirror list is ordered by the region nearest to the client at the time the manifest was generated.

https://se2.enotcloud.com/files/clm-1120/surface/t2m/1999.zarr.tar?sig=7Hq2xWk1  https://se2.enotcloud.com/mirror/nm-eu-central/clm-1120/surface/t2m/1999.zarr.tar?sig=Pd91cLmA
  out=surface/t2m/1999.zarr.tar
  checksum=sha-256=9f2c1a6e4b7d0058e3a1c9d2f64b8e07a5d3c1b9e82f7a604d5c3b1e9a8f7c26

Choosing connection counts

More connections do not always help. Throughput is limited by your tier and by disk speed, and every connection to a single region shares one fair-share queue. These values have worked well for the transfer sizes we see:

  • Many small files (imagery tiles, FASTQ): -j 16 files in parallel with -x 4 connections each.
  • Few large files (CRAM, Zarr archives): -j 4 -x 16 -s 16.
  • Spinning disks: keep total parallel writers below 12; use --file-allocation=none to avoid a long pre-allocation pause.
  • Shared links: set --max-overall-download-limit to leave room for colleagues.

Verifying and repairing

Each directory contains a SHA256SUMS file. Run sha256sum -c SHA256SUMS --quiet and re-fetch any file that is reported. With aria2, --check-integrity=true does this during the transfer and repairs only the damaged chunks.

Transfer endpoints

Members can request a Globus-style endpoint, which moves data directly between our storage and yours without passing through a client machine. Endpoints retry automatically for up to 72 hours, encrypt in transit with TLS 1.3, and send a completion message with a checksum report. Setup takes about two working days; write to transfers@se2.enotcloud.com with your institution's endpoint name.

Limits and fair use

  • HTTP 429 means you exceeded connections or throughput for your tier; wait for Retry-After seconds.
  • HTTP 403 with expired_signature means the manifest is older than its lifetime; request a new one.
  • Do not mirror the archive to redistribute it commercially; the licence on each collection says whether that is allowed.
  • For repeated full-archive copies, ask for a data node agreement instead.