NIRD Data Peak vs NIRD Data Lake

NIRD consists of two distinct storage systems, namely NIRD Data Peak (known internally as TS) and NIRD Data Lake (codenamed DL).

NIRD Data Peak has several tiers spanned by single filesystem and designed for performance and used mainly for active project data.

NIRD Data Lake has a flat structure, designed mainly for less active data, sharing data across multiple projects, and interfacing with external storages.

Both are based on IBM Elastic Storage System.

Architecture comparison

	NIRD Data Peak	NIRD Data Lake
Tiers	Performance and capacity tiers Automatic, transparent tiering Dedicated pools for metadata	Flat architecture (no tiers)
Designed for	- high-performance storage for any type of active research data - high-performance storage for I/O intensive computing such as HPC and AI workloads - storage of valuable research data requiring a secondary copy - storage for structured data of any volume larger than 1 TiB	- long-term storage of non-persistent data - storage for any type of inactive/cold data - storage for structured or unstructured data of any volume larger than 1 TiB - sharing datasets and libraries for collaboration across projects and institutions in the sector - interfacing with sensors or third party storage system - object storage
Data integrity secured by	- erasure coding - snapshots - backup[1]	- erasure coding - snapshots

Functionality comparison

	NIRD Data Peak	NIRD Data Lake
Protocols	POSIX, GPFS and NFS	POSIX, GPFS and S3[2]
APIs	GPFS, Discover REST API[3]	GPFS, S3, Discover REST API[3]
Possibilities for	- file access logs -data insight: metadata harvesting[3]	- file access logs - data insight: metadata harvesting[3] - encrypted projects
Access controls	- ACLs - extended attributes	- ACLs - extended attributes - RBAC via S3[2]

Filesystems

NIRD Data Peak

Project storage /nird/datapeak
User’s home /nird/home
Scratch storage /nird/scratch[4]
Archive /archive[5]

NIRD DL

Project storage /nird/datalake
Backup /backup[5]
Archive /archive[5]

–