Course · RCC ClusterDocs

Class 2: reproducible scientific workflows

Learning objectives

You will create a project that separates raw data, workflow definitions, software declarations, logs, benchmarks and generated results. You will also compare a project-authored Snakemake workflow with a versioned community pipeline run through Nextflow and nf-core.

Recommended layout

project/
├── config/
├── workflow/
├── scripts/
├── envs/
├── data/raw/        # treated as read-only
├── results/         # generated
├── logs/
└── benchmarks/

Git should contain workflow logic, text configuration and documentation. It should not contain credentials, private keys, patient identifiers, raw research data or large generated outputs.

VS Code with Remote - SSH is the suggested project interface for most users. Open this repository directory rather than the project-storage root, review Git changes before committing, and exclude data/, results/, environments, and workflow caches from search and file watching.

Good cluster pattern

Use a workflow engine to describe dependencies and submit scientific work through Slurm. Do not keep a large workflow running as ordinary processes on an SSH gateway. Class 6 teaches the ready-now managed Snakemake path; Class 7 documents the not-yet-released managed Nextflow path.

The first safe question is always a non-mutating dry run. For Snakemake:

snakemake --dry-run --printshellcmds

Then continue with Class 6: Snakemake on RCC. If a reviewed project uses Nextflow or nf-core, read Class 7 for the managed controller, Slurm, shared-work, scratch, Apptainer, and -resume boundary. Managed Nextflow support is ready now.

Software environments inside jobs

Keep the environment declaration in Git and create it inside an allocated worker. In a non-interactive Slurm script, load the shell hook before activation:

eval "$(conda shell.bash hook)"
conda activate analysis
srun python analysis.py

Do not run conda init in every job. Confirm environment and package caches use the approved node-local paths rather than metadata-sensitive shared storage.

Reference companion: Conda, Snakemake, Nextflow, nf-core, and Apptainer covers batch activation, Snakemake sessions, Nextflow and nf-core, explicit container binds, cache placement, GPU exposure, and reproducibility records.

Security moment

A reproducible workflow is also a security control: changes can be reviewed, inputs and outputs are explicit, and unexpected code is easier to identify. Pin software versions, review contributed scripts, and never run downloaded code merely because it is in a shared project directory.

Self-learning exercise

Build a three-rule workflow that creates a small synthetic input, transforms it, and writes a checksum. Run it twice and verify that the second run performs no unnecessary work.

Knowledge check

Completion gate

• The workflow dry run succeeds.
• The first run creates the expected checksum.
• The second run reports that no work is required.
• git status contains no credentials, private data, raw datasets or generated result directories.