Using storage

As mentioned earlier in the architectural description, palmetto takes advantage of a file storage system that:

• is multi-level in terms of technology and media,
• is multi-speed including fast storage solutions,
• provides both temporary and long-term solutions,
• is implemented as directories for ease of use,
• can support secure/classified data as needed.

We will describe the file storage system in terms of the directories.

Locating the directories

First we can take a look at the file system and get some sense of the directories with the df command. For example

[myuserid@user001 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              36G   13G   22G  37% /
/dev/sda2              36G  177M   34G   1% /scratch
/dev/sda1              46M   22M   22M  51% /boot
tmpfs                 5.9G     0  5.9G   0% /dev/shm
sammds001:/export/sf-home
                      1.7T 1000G  674G  60% /home
sammds002:/export/qfs-small
                      8.8T  5.7T  3.1T  66% /projsmall
sammds002:/export/qfs-large
                       15T   12G   15T   1% /projlarge
master:/opt/xcat       64G   56G  4.5G  93% /opt/xcat
master:/usr/local      64G   56G  4.5G  93% /usr/local
master:/opt/matlab     64G   56G  4.5G  93% /opt/matlab-new
pbs001:/opt/gold/etc   64G   31G   31G  51% /opt/gold/etc
pbs001:/opt/gold/lib   64G   31G   31G  51% /opt/gold/lib
mds001@tcp0:/hpc       58T  2.9T   52T   6% /lustre

We see the hardware device name in the first column, the size of the device in the next column, how many blocks are used and available in the third and fourth columns, the percentage of the device that is used in the fifth column, and the name of the directory (as we know it) in the last column. (See man df for more information.)

Five of these directories are of interest to us and will be described here.

Long term storage: /home

The /home directory is the usual root directory where you store your permanent files. A Sun Microsystem StorageTek ST2540 disk array and a StorageTek SL8500 robotic tape system combine to provide a storage system of over 120 TBytes. Frequently used files are kept on disk while others are kept on the robotic tape system. File services are provided to you transparently via Sun's SAM-QFS Storage Management Software. For more information on SAM-QFS, see Sun StorageTek Storage Archive Manager and Sun StorageTek QFS Software.

/home and /proj

StorageTek SL8500 robotic tape system

Your /home directory is shown using the ls command:

[myuserid@user001 ~]$ ls -l
total 1116
-rw-------  1 myuserid cuuser      0 Apr 10 17:39 Array-decomp.e40835
-rw-------  1 myuserid cuuser   3026 Apr 10 17:39 Array-decomp.o40835
drwxr-xr-x  2 myuserid cuuser   4096 Jul 17 18:32 bin
drwxr-xr-x  2 myuserid cuuser   4096 Jun 27 18:18 grotest
drwxr-xr-x  3 myuserid cuuser   4096 May  7 14:18 images
-rw-r--r--  1 myuserid cuuser 202811 Mar 24 22:35 mifort
-rw-------  1 myuserid cuuser    212 Mar 12 15:59 Pade.e35049
-rw-------  1 myuserid cuuser 826352 Mar 12 15:59 Pade.o35049
drwxr-xr-x 12 myuserid cuuser   4096 Apr 25 10:03 restest
-rw-------  1 myuserid cuuser      0 Mar 24 20:14 workshare.e36842
-rw-------  1 myuserid cuuser   3889 Mar 24 20:14 workshare.o36842

Long term storage: /projsmall and /projlarge

The /proj directories, similar to /home, are available to the palmetto owners. These directories are intended for large files and for owner group files. Only palmetto owners can create directories and files on /projsmall and /projlarge.

[myuserid@user001 ~]$ ls -l /projsmall
drwxrwx--- 15 root    bioengr   4096 Jan  8 15:24 bioengr
drwxrwx---  6 root    camsil    4096 Jan 15 13:35 camsil
drwxrwx---  5 root    catalyst  4096 Jan 15 10:33 catalyst
drwxrwx---  2 root    crwcitc   4096 Jan  9  2008 crwcitc
drwxrwx---  2 root    cugi      4096 Oct 29 11:45 cugi
drwxr-xr-x  2 root    dean      4096 Oct 29 12:48 dean
drwxrwx--- 13 root    dominy    4096 Aug 28 10:58 dominy
drwxrwx---  2 root    feltus    4096 Nov 10 11:34 feltus
drwxrwxr-x  5 root    root      4096 Jan  2 10:37 hpc
drwxrwx---  6 root    ival      4096 Sep 23 13:39 ival
drwxr-xr-x 28 joehsen root      4096 Jan 13 06:12 joehsen
drwx------  2 root    root     16384 Sep 19 13:26 lost+found
drwxr-xr-x  3 root    root      4096 Oct  6 11:12 metadumps
drwxrwx---  8 root    osg       4096 Jan 14 14:39 osg
drwxrwx---  5 root    stuart    4096 May  9  2008 stuart
drwxr-xr-x  2 root    root      4096 Oct  1 20:39 test_mikec
drwxrwx--- 11 root    tinman    4096 Jan  8 10:38 tinman
drwxrwx---  8 root    tpl       4096 Dec 22 22:22 tpl
drwxrwx---  9 root    wilson    4096 Nov 12 17:31 wilson
[myuserid@user001 ~]$ cd /projsmall
[myuserid@user001 proj]$ mkdir myuserid
mkdir: cannot create directory `myuserid': Permission denied

Note: Condo owners decide how much of their purchased space they want on the large filesystem and how much on the small filesystem. We will adjust the quotas accordingly. Condo owners can get their quotas adjusted at any time in order to match their current needs based on the work they are doing.

Fast, large storage: /lustre

The /lustre directory is fast, large, shared storage. Access to the data stored in /lustre is very fast. The /lustre directory is very, very large. The /lustre directory is shared by all palmetto nodes. Therefore you should place your large files here for the best performance of your programs.

Any user of palmetto can use the /lustre directory.

[myserid@user001 ~]$ ls -l /lustre
total 5668
drwxrwxrwx  2 fpierce cuuser     4096 May 30 14:58 20PPE-2
drwxr-xr-x  2 fpierce cuuser     4096 Jul  1 15:51 4xpolym
drwxr-xr-x  3 fpierce cuuser     4096 Jun  2 14:01 Accelrys
drwxr-xr-x  3 bradles cuuser     4096 Jun  3 13:47 bradles
-rw-r--r--  1 root    root    2334204 Jun 26 11:13 du.out
-rw-r--r--  1 root    root    2334204 Jun 26 11:14 du.out.sorted
drwxr-xr-x 14 fpierce cuuser     4096 Jul  7 00:58 fpierce
-rwxr-xr-x  1 smaskey cuuser    33141 Jun 23 16:48 insight2lammps.pl
drwxr-xr-x  2 joehsen cuuser     4096 Jun 20 09:50 joehsen
drwx------  3 nvellor bioengr    4096 Jun 29 12:11 nadeem
-rw-r--r--  1 smaskey cuuser   342997 Jun 23 16:48 pcff.frc
drwxrwxrwx  4 fpierce cuuser     4096 Jul  3 09:39 PolyEthexPPE
-rw-r--r--  1 smaskey cuuser    99327 Jun 23 16:48 PolyEthexPPE.car
-rw-r--r--  1 smaskey cuuser   477544 Jun 23 16:48 PolyEthexPPE.data
-rw-r--r--  1 smaskey cuuser   103524 Jun 23 16:48 PolyEthexPPE.mdf
drwxrwxrwx  3 smaskey cuuser     4096 Jul  3 00:42 Polylactide
drwxr-xr-x  3 fpierce cuuser     4096 Jul  2 16:01 PPE
-rw-r--r--  1 root    root        309 Jun 24 16:07 README
drwxr-xr-x  2 fpierce cuuser     4096 Jun 18 14:51 SPCE-F16
drwxrwxrwx  3 fpierce cuuser     4096 May 30 11:27 Toluene
drwxr-xr-x  2 walt    cuuser     4096 May 30 13:38 walt
drwxr-xr-x 13 fpierce cuuser     4096 Jul 17 17:36 wetting
drwx------  3 wolf      47414    4096 Jul  4 02:05 wolf
[myserid@user001 ~]$ cd /lustre
[myserid@user001 lustre]$ mkdir myserid
[myserid@user001 lustre]$ mkdir myserid/pade
[myserid@user001 lustre]$ cp /home/myserid/restest/pade/pade /lustre/myserid/pade/

Please be aware that the /lustre directory is for temporary use only, during your runs. CCIT administrators will clear older, unused files in order to make way for current needs.

Lustre storage units

Also a product of Sun Microsystems, Luster is a scalable, production level file system. It is capable of aggregating petabytes of storage into a single file system. (Lustre is open source, multi-vendor, multi-platform, and POSIX compliant. For more information, see the Lustre File System - Overview at the Sun Microsystems website.)

Temporary/local storage: /scratch

Each node has its own /scratch directory which is a temporary storage areas. You can move data to and from /scratch to improve job performance. The space is limited though (each node contains about 35 GB of disk), so /scratch is generally useful for smaller files. Currently each /scratch on each node is independent and therefore you need to put some thought into getting data to and from the appropriate nodes.

As with the /lustre directory, /scratch is for temporary use only, during your runs. CCIT administrators will clear older, unused files in order to make way for current needs.

File storage quotas

Currently each person has a permanent, long term file storage (/home) quota of 50GB. You can check your quota with the checkquota command.

[myuserid@user001 ~]$ checkquota
 
DISK QUOTA (in KB) for HOME DIRECTORY /home/myuserid :
 The default home directory disk quota is 50GB.
 The default home directory file quota is 1 million files.
 
                                 Online Limits                Total Limits
        Type    ID    In Use     Soft     Hard    In Use     Soft     Hard
/export/sf-home
Files   user 128060       197 100000000 100001000       197 100000000 100001000
Blocks  user 128060     11092 52428800 53477376     11092 52428800 53477376
Grace period                    1w                          1w
 

Here we see that 11,092 blocks (around 11KB of 50GB) are used in 197 files. The online limit refers to disk while the total limit refers to disk and tape combined. When your usage reaches the soft limit, you will get warnings. When your usage reaches the hard limit, you will no longer be able to create files. You will have a one week (1w) grace period to move or delete files before your usage is locked out.

Secure storage

Some data, due to grant and sponsor requirements, must have restricted access. This could be true of certain private medical data, classified security data, sensitive financial information, and so forth. Please contact us if you have secure storage needs. We will set up the appropriate file environment for you.