Changes between Version 9 and Version 10 of PublicPages/CosmoRepo

Timestamp:

Aug 22, 2014 10:31:08 AM (10 years ago)

Author:

Benjamin Alan Weaver

Comment:

--

PublicPages/CosmoRepo

@@ -2 +2 @@
 
 [[PageOutline]]
+
+== Introduction ==
 
 The NERSC Cosmo Data Repository co-locates cosmological survey data to facilitate the analysis of multiple datasets.
@@ -9 +11 @@
 [http://portal.nersc.gov/project/cosmo/ HTTP interface to those files under www]
 
-== Staging area vs. final data ==
+== Top-level structure of the Repository ==
 
-`/project/projectdirs/cosmo/staging/` is intended for temporary staging work and preparation of data releases,
-and is writable by any member of the cosmo group.
+Conceptually, there are five areas of the repository:
 
-After these data are ready, they can be moved into a final `cosmo/data/{survey}/{release}/` directory.
-That area is restricted to the cosmo project user to ensure some level of consistency for the
-data written there.  This process should be viewed like SDSS data releases: the data are processed and
-organized elsewhere, and then when finalized (and only then) they are moved into the final data release
-directories, controlled by the cosmo user.  Once written and released, these data are frozen -- never
-updated nor removed.
+ 1. Staging area for raw data
+ 1. Work area for data processing
+ 1. Final data release area
+ 1. Area for web access
+ 1. Area for software
+
+First, data is copied from elsewhere to the staging area in preparation for processing.  Processed data is written to the work area.  After the processed data are finalized, they can be moved into a final `cosmo/data/{survey}/{release}/` directory. That area is restricted to the cosmo project user to ensure some level of consistency for the data written there.  This process should be viewed like SDSS data releases: the data are processed and organized elsewhere, and then when finalized (and only then) they are moved into the final data release directories, controlled by the cosmo user.  Once written and released, these data are frozen -- never updated nor removed.
+
+Here is the specific directory layout:
+
+ /project/projectdirs/cosmo/staging/ :: This is the staging area.  Each survey has a directory under staging/.
+ /project/projectdirs/cosmo/work/ :: This is the work area.  Each survey has a directory under work/.
+ /project/projectdirs/cosmo/data/ ::  This directory is intended for final, static data releases.  
+ /project/projectdirs/cosmo/www/ ::  This directory provides HTTP access to the repository.
+ /project/projectdirs/cosmo/software/ ::  This is for software needed by the cosmo group.
+
+== Cosmo User Account ==
 
 Currently Ben, Dustin, and Stephen have access to the cosmo user account via:
@@ -33 +45 @@
 }}}
 
+== Data Sets ==
 
-== 2MASS data ==
+=== 2MASS data ===
 
-== DECam Data ==
+=== DECam Data ===
 
 [attachment:decam-exposures.fits.gz​ FITS file with coordinates of all DECam images through 16 June 2014]
 
-== SDSS Data ==
+=== SDSS Data ===
 
  $BOSS_PHOTOOBJ=/project/projectdirs/cosmo/data/sdss/pre13/eboss/photoObj.v5b/ ::
   SDSS photometric catalogs
-
  $PHOTO_SWEEP=/project/projectdirs/cosmo/data/sdss/pre13/eboss/sweeps/eboss.v5b/ ::
    SDSS trimmed photometric catalogs (datasweep files) used for SDSS-IV/eBOSS target selection
-
  $PHOTO_RESOLVE=/project/projectdirs/cosmo/data/sdss/pre13/eboss/resolve/2013-07-29 ::
   SDSS photometric resolve information defining SDSS footprints, sky areas, and primary identifications of catalog objects
 
-== WISE data ==
+=== WISE data ===
 
  $WISE_IMAGE_DIR=/global/project/projectdirs/cosmo/data/merge/merge_p1bm_frm ::
   WISE Level 1b pixel-level images
-
  /global/project/projectdirs/cosmo/data/unwise ::
   Dustin Lang's unWISE image stacks at full WISE resolution