NIRD S3

What is object storage?

Object storage is specifically designed to manage large volumes of unstructured data. It stores data as discrete units, known as objects, each accompanied by metadata and a unique identifier. Each object is accessible via a dedicated key-pair, ensuring that all data blocks within a file are kept together as a single entity, complete with its relevant metadata and unique ID.

In the context of object storage, traditional file storage directories (POSIX) are analogous to ‘buckets.’

About NIRD S3

On NIRD, object storage is available as a service component within the NIRD Data Lake, supporting the efficient consumption and ingestion of unstructured data. This service also facilitates data streaming, data sharing with external collaborators, and more.

The NIRD Data Lake provides unified access to data, allowing the same data can be accessed simultaneously through multiple protocols, such as POSIX, NFS, and S3.

The object storage on NIRD Data Lake facilitates seamless collaboration without the necessity for data staging between locations. It also allows integration with data processing frameworks and AI/ML platforms, providing fast, and easy access to large datasets for training models and running simulations.

NIRD S3 comes with enhanced load balancing and high I/O throughput, with a measured throughput of 27GB/s.

NIRD S3 supports 10TB single object sizes and up to 100 million objects per bucket.

Working with NIRD S3

This guide covers what you can do with the S3 management interface: viewing your accounts and buckets, retrieving credentials, sharing bucket access with project colleagues, and using S3 from the command line. It does not cover all possible use cases and functionalities.

Creating and managing S3 accounts and buckets is handled by Sigma2 support — see the sections below for how to request one.

Getting an S3 account

S3 accounts are created by Sigma2 support on request.

  1. Email contact@sigma2.no and include your project number and the account name you’d like to use.

  2. Support creates the account for you, with the project number appended to your requested name automatically. Account names follow this convention — for example, requesting alice for project NS1234K gives you the account alice-ns1234k.

  3. Once created, find it in the administration system (MAS) under the S3 dropdown menu → Accounts, with credentials populated.

Retrieving your credentials

Open your account from the account list under the S3 dropdown menu → Accounts (or directly at https://www.metacenter.no/mas/s3/accounts/).

Both Access key and Secret key are masked (•••••••) by default:

  • Click the Copy button next to each field to copy the value to your clipboard without it appearing on screen.

  • Treat the secret key like a password — do not paste it into emails or shared documents.

If both keys show as — (no value), your credentials have been revoked.

Getting a bucket

Buckets are created by Sigma2 support on request, once you have an S3 account.

  1. Email contact@sigma2.no and include your account name and the bucket name you’d like to use.

  2. Bucket names follow a naming convention too: your account name is prepended to your requested name automatically — for example, requesting rawdata under account alice-ns1234k gives you the bucket alice-ns1234k-rawdata. Bucket names must also be globally unique — support will let you know if your preferred name is taken.

  3. The bucket appears on your account detail page once created.

Sharing bucket access with project members

Any project member who has their own S3 account under the same project number can be granted access to your bucket.

  1. Open the bucket detail page from your account.

  2. The Access control panel lists all S3 accounts in your project, with each account’s current access level.

  3. Click the button for the level you want to grant:

    • Read — read-only access

    • Read/Write — full read and write access

    • Revoke — remove all access for that account

  4. The row updates immediately — no page reload required. Changes are synced to the S3 backend automatically every 10 minutes.

Access levels

Level

Permitted S3 actions

Read only

s3:ListBucket, s3:GetObject

Read/Write

s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject

Who appears in the list

The Access control panel lists every S3 account registered under the same project as the bucket, except the account that owns the bucket (the owner always has full access). Note that access is granted to S3 accounts, not to people directly — a project member without an S3 account will not appear in the list. If a colleague is missing, they need to request their own S3 account first (see Getting an S3 account above).

S3 client configuration

Several CLI clients are available like AWS CLI, s3cmd, MinIO Client, rclone, etc. As graphical user interface, Cyberduck may be used. Please consult each tool’s documentation for detailed help.

Below are two examples: the AWS CLI and rclone. The S3 endpoint is https://s3.nird.sigma2.no.

AWS CLI

Once you have your access key, secret key, and a bucket, you can use the standard aws CLI to work with your files.

Installing the AWS CLI

  • Linux (Ubuntu/Debian): sudo apt install awscli, or follow the official installation instructions.

  • macOS: brew install awscli, or download the official installer from the same page.

  • Windows: download and run the MSI installer from the same page, then use the commands below in PowerShell or Command Prompt.

Setting up a profile

Run aws configure --profile <profile> and enter your access key and secret key when prompted. This works the same on Linux, macOS, and Windows. Then either:

  1. Add the endpoint to each command: append --endpoint-url https://s3.nird.sigma2.no to every command below, or

  2. Add it to your config file once: add endpoint_url = https://s3.nird.sigma2.no under your profile in your AWS config file, so you don’t need to repeat it. The config file is at ~/.aws/config on Linux and macOS, and %UserProfile%\.aws\config on Windows:

[profile <profile>]
endpoint_url = https://s3.nird.sigma2.no
Command cheat sheet

The commands below assume you added the endpoint to your config file as described above. If you didn’t, append --endpoint-url https://s3.nird.sigma2.no to each command.

list buckets: aws s3 ls --profile <profile>

list files: aws s3 ls s3://<your-bucket-name> --profile <profile>

list files recursively (include subfolders): aws s3 ls s3://<your-bucket-name> --recursive --profile <profile>

copy file (bucket to local): aws s3 cp s3://<your-bucket-name>/<file-name.ext> /path/to/local/destination/ --profile <profile>

copy file (local to bucket): aws s3 cp /path/to/local/file.ext s3://<your-bucket-name>/ --profile <profile>

move file (local to bucket): aws s3 mv /path/to/local/file.ext s3://<your-bucket-name>/ --profile <profile>

move file (bucket to local): aws s3 mv s3://<your-bucket-name>/<file-name.ext> /path/to/local/destination/ --profile <profile>

delete a file: aws s3 rm s3://<your-bucket-name>/<file-name.ext> --profile <profile>

delete an entire folder inside a bucket: aws s3 rm s3://<your-bucket-name>/<folder-name>/ --recursive --profile <profile>

sync local folder to bucket (only copies new/updated files): aws s3 sync /path/to/local/folder/ s3://<your-bucket-name>/ --profile <profile>

sync bucket to local folder: aws s3 sync s3://<your-bucket-name>/ /path/to/local/folder/ --profile <profile>

rclone

Examples of setting up rclone and using rclone command line interface. The rclone application can do a range of file transfers (cp, scp, rsync type etc), but only S3 is shown here.

Make a configuration for rclone. by running rclone config Or editing the config file directly, it’s located at:

$HOME/.config/rclone/rclone.conf

Editing the config file is simpler than running the config option as only a few lines are needed to set up rclone with S3.

An example config could look like this, fill in the credentials from your account page in the administration system (see Retrieving your credentials above).

[S3]
type = s3
provider = Other
access_key_id =  xxxxxxxxxxx
secret_access_key = yyyyyyyyyyyyy
endpoint = https://s3.nird.sigma2.no
acl = private

After the config file is updated the rclone is ready for use against the NIRD datalake.

rclone ls  S3:<user>-ns<project>k-<user>/
1099511627776 tmp.dta
102400000000 tmp.m1
263095463936 tmp.medium
1073741824 tmp.small

Once we have retrieved the names we can copy the file we want to use.

rclone copy  S3:<user>-ns<project>k-<user>/tmp.medium  .

To monitor progress and transfer speed use the option -P.

rclone copy  S3:user-nsXXXXk-user/tmp.medium . -P
Transferred:      245.027 GiB / 245.027 GiB, 100%, 19.587 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:      4m56.9s

Copy other direction:

rclone copy tmp.m1  S3:user-nsXXXXk-user/ -P  --ignore-checksum

As the receiver has to read the whole file to calculate the checksum it takes a considerable amount of time, it’s turned off to provide the transfer speed only.

It’s possible to run multiple transfers simultaneously with many files (wildcards don’t work, hence the include option) :

rclone copy  S3:user-nsXXXXk-user/ . --include "tmp.m?"\ 
--transfers=10 -P  --ignore-checksum
Transferred:      953.674 GiB / 953.674 GiB, 100%, 5.225 GiB/s, ETA 0s
Checks:                 0 / 0, -, Listed 19
Transferred:           10 / 10, 100%
Elapsed time:      2m22.0s

The combined transfer bandwidth can be quite good. On the other hand doing checksum at both ends takes time and impedes the transfer, causing it to take considerably longer time.

When transferring much data as the below example shows quite high transfer speed can be obtained as measured from the User Access Nodes (uan or login nodes). The example transfers a set of 1000 smaller files at 512 MB each with a varying number of transfers (threads) (rclone uses sftp as the transfer layer).

Transfers

Transfer speed [GB/s]

10

7.376

20

10.26

30

11.15

40

10.20

50

11.33 !

60

10.56

At 10 GBytes/s it’s possible to transfer 36 TB/h which is actually quite good.

For more details of what rclone can do see: Documentation

For more information and examples, visit the Data transfer page.