NERSC Cosmo Public Data Repository

Introduction

The NERSC Cosmo Data Repository co-locates cosmological survey data to facilitate the analysis of multiple datasets. Those data sets in the "www" directory are visible through web interfaces. The other locations are development and staging areas, visible for all users with a NERSC account.

​HTTP interface to those files under www

Top-level structure of the Repository

Conceptually, there are five areas of the repository:

1. Staging area for raw data
2. Work area for data processing
3. Final data release area
4. Area for web access
5. Area for software

First, data is copied from elsewhere to the staging area in preparation for processing. Processed data is written to the work area. After the processed data are finalized, they can be moved into a final cosmo/data/{survey}/{release}/ directory. That area is restricted to the cosmo project user to ensure some level of consistency for the data written there. This process should be viewed like SDSS data releases: the data are processed and organized elsewhere, and then when finalized (and only then) they are moved into the final data release directories, controlled by the cosmo user. Once written and released, these data are frozen -- never updated nor removed.

Here is the specific directory layout, rooted at /global/project/projectdirs/cosmo:

data/

This directory is intended for final, static data releases.

etc/

Files needed to support http operations, etc.

metadata/

Data related to monitoring file size, permissions, etc. of data in this directory tree.

software/

This is for software needed by the cosmo group.

staging/

This is the staging area. Each survey has a directory under staging/.

webapp/

Web applications, like the ​DECaLS image viewer have files here.

work/

This is the work area. Each survey has a directory under work/.

www/

This directory provides HTTP access to the repository.

Cosmo User Account

Currently Ben, Dustin, and Stephen have access to the cosmo user account via:

ssh carvergrid.nersc.gov
export GLOBUS_LOCATION=/usr/common/osg/gsissh/globus
source /usr/common/osg/gsissh/globus/etc/globus-user-env.sh
myproxy-logon -s nerscca.nersc.gov
# Enter NERSC NIM password
gsissh cosmo@localhost
# Now you are the cosmo user

Data Sets

2MASS data

DECam Data

Public DECam data not necessarily affiliated with the DECaLS project is obtained by first querying the NOAO portal with something like this:

SELECT md5sum, reference, dtpropid, surveyid, release_date, start_date,
    date_obs, dtpi, ra, dec, telescope, instrument, filter, exposure, 
    obstype, obsmode, proctype, prodtype, seeing, depth, dtacqnam,
    reference AS archive_file, filesize, noao_id
FROM voi.siap
WHERE dtacqnam LIKE '%001_____.fits.fz' 
AND proctype = 'Raw'
AND telescope = 'ct4m'
AND instrument = 'decam'
AND release_date < '2015-04-01'
LIMIT 100000;

The results of these queries are downloaded in VO Table format, then concatenated and converted into a FITS table. The resulting catalogs are available in directories of the form:

/global/project/projectdirs/cosmo/staging/decam-public/DECam_00XXXXXX-YYYY-MM-DD/

where YYYY-MM-DD refers to the value of the release date in the query above.

FITS file with coordinates of all DECam images through 16 June 2014​

SDSS Data

$BOSS_PHOTOOBJ=/project/projectdirs/cosmo/data/sdss/pre13/eboss/photoObj.v5b/

SDSS photometric catalogs

$PHOTO_SWEEP=/project/projectdirs/cosmo/data/sdss/pre13/eboss/sweeps/eboss.v5b/

SDSS trimmed photometric catalogs (datasweep files) used for SDSS-IV/eBOSS target selection

$PHOTO_RESOLVE=/project/projectdirs/cosmo/data/sdss/pre13/eboss/resolve/2013-07-29

SDSS photometric resolve information defining SDSS footprints, sky areas, and primary identifications of catalog objects

WISE data

$WISE_IMAGE_DIR=/global/project/projectdirs/cosmo/data/merge/merge_p1bm_frm

WISE Level 1b pixel-level images

/global/project/projectdirs/cosmo/data/unwise

Dustin Lang's unWISE image stacks at full WISE resolution ​Paper "unWISE: unblurred coadds of the WISE imaging"

---

Pages linking to PublicPages/CosmoRepo:

• DecamLegacy/DataAccess
• DecamLegacy/ObservingStrategy
• DecamLegacy/Reductions
• ImagingWG
• PublicPages