Storage and File Structure

Holmes Lab Storage

Holmes lab storage and user group is located in Amarel at /projects/f_ah1491_1

To be added to the user group to acces this storage, email help#oarc.rutgers.edu with your NetID and CC Avram, avram.holmes@rutgers.edu. If no response, email pgarias#oarc.rutgers.edu

NOTE:

To access the Holmes Lab files from Jupyter Notebook in Amarel, you must create symlinks to the project and scratch directories.

The formula for creating a symlink is: ln -s /path/to/original /path/to/symlink. For example:

ln -s /projects/f_ah1491_1 ~/projects_f_ah1491_

This will create a folder in your home directory called projects_f_ah1491_1 that links to the Holmes Lab project directory. You can call the link any name. You can do the same for the scratch directory:

ln -s /scratch/f_ah1491_1 ~/scratch_f_ah1491_

Again, you can name it ‘scratch_f_ah1491_’ or any name you want.

Read the Holmes Lab File Structure and Norms page

Holmes Lab Storage

• /projects/f_ah1491_1 > 100TB of storage, backed up – 5 million files
• /scratch/f_ah1491_1 > 100TB of storage, NOT backed up, no filecount limit For more information see the Holmes Lab File Structure and Norms page.

Personal Storage

Each NetID Gets personal storage of:

• /home/NetID > 100GB of storage, not fastest, backed up
• /scratch/NetID > 1TB of storage, fast, up to 2TB before purging, not backed up

Storage Info from OARC

• Storage file sets and how to use them
• /home
• /scratch
• /projects
• /mnt/scratch
• Home and Projects Snapshots
• Checking storage utilization
• Snapshots of /home and /projects data

Storage Best practices

• Move files to your /scratch directory and run your job from that location.
• Don’t leave files sitting unused (unaccessed) for a long time because you may lose them to the 90 day purge process.
• Frequently check the utilization or quota for your /home and /scratch directories to ensure that they don’t become unusable due to over-filling with files.

• The general approach for using /scratch is to copy your job’s files (input files, libraries, etc.) to /scratch, run your job and write output files to /scratch, then move the files you need to save to your /home or /projects directory.

• Helpful Code Snippets: You can stage needed files in /scratch from within a job (either within your job script for a batch job or on the command line during an interactive job)

    mkdir /scratch/$USER/$SLURM_JOB_NAME-$SLURM_JOB_ID
      
    scp my-dir-of-input-files.tar.gz /scratch/$USER/$SLURM_JOB_NAME-$SLURM_JOB_ID
      
    tar -zxf /scratch/$USER/$SLURM_JOB_NAME-$SLURM_JOB_ID/my-dir-of-input-files.tar.gz