OBSERVING INSTRUCTIONS FOR MzLS

These are instructions for using the MzLS planning scripts for the nightly program. To see the Mosaic observing manual, go here: ​http://www.noao.edu/kpno/mosaic/manual.

Log on to the mzls account on the mayall-idl.kpno.noao.edu machine:

Username : mzls

Password : Ask Arjun, David, or previous observers

Two environment variables define the locations of data and code:

$MOS3_DATA= -- This is where our raw data files are written
$MOS3_OBS=$HOME/mosaic3 -- This contains the observing product with code and versioned log files.

To set these, type, e.g., "export MOS3_DATA=<path>"

Other directories in the home directory:

ps1/ -- Doug Finkbeiner's Pan-STARRS1 catalogs used by the IDL DECSTAT routine for computing astrometric offsets and photometry

Documentation for the IDL scripts can be printed from the IDL prompt with the DOC_LIBRARY command, for example:

      IDL> doc_library,'mosstat'

Date convention

All dates in log files are set to the local date of the beginning of the night. For example, any data taken during the night of August 17/18, 2014 will be written as 2014-08-17. This is consistent with how the NOAO Science Archive timestamps and saves the raw data files. (This convention has changed from what we did at the start of the survey, but Greg has made everything consistent.)

Official Nightly Strategy

Update 4/26/2015 Do the following to update the current NMSL scripts at any time.

• login to mayall-idl as mzls
• cd products/mosaic3
• python2.7 py/mosaic3strategy.py -t obstatus/mosaic-tiles_obstatus.fits -d 2015-12-13 -p 1 -pass 1 --moon-sep-limit 50 --start-date 2015-12-14 --start-time 02:00:00 -seez 1.15 -sbz 18.45 --dir <dir_name>

Rename the script, so we don't overwrite it later:

mv mosaic_2015-12-13_plan.json mosaic_2015-12-13_plan1.json mv strategy_2015-12-13.txt strategy_2015-12-13_plan1.txt

The mosaicstrategy script outputs a JSON file (e.g., mosaic_2015_12-13_plan.json in the above example). In order to produce the execution script for NOCS, run Behzad Abareshi's script:

python ./py/jnox.py --in mosaic_2015-12-13_plan1.json --out mzls_2015-12-13_plan1.sh

Then copy this output (e.g., mzls_2015-12-13_plan1.sh) to the exec directory of the observer account on the mosaic3 computer:

scp mzls_2015-12-13_plan1.sh observer@mosaic3:/home/observer/exec/

Start uptiles running in an IDL window on mayall-idl; this will always use the most up-to-date version of the mosaic-tiles_obstatus.fits file. Once this finishes running (usually a couple of minutes), in a nocs terminal window on mosaic3, cd to exec and execute the script:

~/exec> ./mzls_2015-12-13_plan1.sh

Instructions and details for mosaicstrategy.py

written by A. Patej

mosaicstrategy.py is a Python script that can generate a nightly observing plan. It requires several Python modules:

• numpy
• pylab (Matplotlib)
• ephem (Pyephem)
• pyfits

The code is adopted from the DecamLegacy project. See those wiki pages for more information.

Selection of Pass 1, 2, or 3

It is very important to adhere to the strategy of when to observe pass 1, 2, or 3 tiles, in particular that pass 1 tiles are always in photometric good-seeing conditions. The overall strategy is outlined at MayallZbandLegacy/ObservingStrategy].

The strategy is as follows:

• Pass 1: Photometric and seeing < 1.3 arc sec
• Pass 2: Photometric or seeing < 1.3 arc sec, or all Pass 1 tiles completed
• Pass 3: Unphotometric and seeing > 1.3 arc sec, or all Pass 1 and Pass 2 tiles completed

These need not be observed in any particular order, e.g. pass 3 observations can certainly occur before pass 1.

If conditions change, then you should definitely change to a different pass. For example, if clouds begin to roll in, you should stop any pass 1 observations right away!

Running the script

At a minimum, the mosaicstrategy.py script can be run with these inputs:

-t *name of file with tile centers*
-d *UT date at start of night in format yyyy-mm-dd*
-p *portion of night (1.0, 0.5 (first half), -0.5 (second half))*
-pass *desired pass - input 1, 2, or 3*
--dir *name of the dir where the output scripts are located*

For instance, to generate a pass 3 plan for the entire night of April 26, 2015, run:

python ./nightlystrategy.py -t moasic-tiles_obstatus.fits -d 2015-04-26 -p 1.0 -pass 3 --dir obsScripts

Additional optional arguments:

1. Observing conditions:
   -sb(f) *sky brightness in filter f; replace (f) = g, r, or z* in mag/sqarcsec
   -see(f) *seeing in filter f; replace (f) = g, r, or z* in arcsec
   -transparency *value of the transparency* (0-1.0)
   

2. Observing constraints:
   --airmass-limit *maximum allowed airmass, default is 2.3*
   --moon-sep-limit *minimum allowed moon separation, default is 50.0 deg*
   --moon-alt-limit *moon altitude in deg required for switching between z and g,r*
   --min-dec *Declination minimum limit for choosing tiles
   --max-dec *Declination maximum limit for choosing tiles
   

3. Custom Scheduling:
   It's possible to create a plan for a specific block of time. To do so, use the following options:
   --start-date *Date of start of observations (UTC) in format yyyy-mm-dd; overrides the default start of night, which is calculated using --date and --portion. Requires --start-time to also be set*
   --start-time *Time of start of observations (UTC) in format hh:mm:ss; overrides the default start of night, which is calculated using --date and --portion. Requires --start-date to also be set*
   --end-date *Date of end of observations (UTC) in format yyyy-mm-dd; overrides the default end of night, which is calculated using --date and --portion. Requires --end-time to also be set*
   --end-time *Time of end of observations (UTC) in format hh:mm:ss; overrides the default end of night, which is calculated using --date and --portion. Requires --end-date to also be set*
   

More complicated example using these options:
Assuming 18.2 as z-band sky brightness, 1.2" seeing in z and 1.0" seeing in r, photometric conditions (pass 1) and allowing a moon separation of 40 deg and including DESI tiles:

python ./mosaicstrategy.py -t mosaic-tiles_obstatus.fits -d 2015-03-24 -p 1.0 -pass 1 -sbz 18.2 -seer 1.0 -seez 1.2 --moon-sep-limit 40.0 --expand

Example using custom scheduling:
To generate a plan for just two hours on the night that starts on 2015-04-06 between 05:00 and 07:00 UTC:

python ./mosaicstrategy.py -t mosaic-tiles_obstatus.fits -d 2015-04-06 -p 1.0 -pass 1 --start-date 2015-04-07 --start-time 05:00:00 --end-date 2015-04-07 --end-time 07:00:00 --min-dec=34 --max-dec=50 --moon-sep-limit 40.0

Outputs

This script will many output data files:

1. a single text file called "strategy_mm-dd.txt", with information about all the pointings for the night (exp time, filter, approximately when they should be run, airmass, etc). There is a corresponding plot of what has been observed previously (by pass) and what you will be observing given the plan for the night: saved as a pdf, 'plot_plan_yyy-mm-dd.pdf' (gray = z, red = r, green = g). Planets and moon are plotted as well.

2. many executable script files of the form "Mosaic_Survey<tile id>.sh" for the night corresponding to the plan. There is one script file for each pointing in the plan. A single meta-script is produced that will call those ".sh" files: moasic_<ut date>_plan.sh.

3. A FITS file called "decam-tiles_obstatus_update_yyyy-mm-dd.fits" that is a temporary updated mosaic obstatus file that assumes all planned observations were completed. This should not be used during the run; instead use the updated tiles file obtained from running the idl program, uptiles (see section on "Creating the Updated Observing Status File" below).

Updating the "observed tiles" file

The FITS file listing which tiles have been completed should be updated throughout the night. This file is $MOS3_OBS/obstatus/decam-tiles_obstatus.fits. The following IDL command will monitor exposures as they are taken throughout the night, automatically updating this file:

   IDL> muptiles

At the end of the night, you should check the updated tile file into the svn repository:

   cd $MOS3_OBS/obstatus
   svn commit obstatus/mosaic-tiles_obstatus.fits --username <YOUR-NAME>

Use of Steve Kent's tools for monitoring observing

Not yet installed at Mayall.

During observing, please check frames as they read out. To use kenttools remotely, you must be logged in as DECamObserver rather than as declsp. To load kenttools log onto observer2 and type "observer" at the unix shell prompt. This will load up an environment in which you can run the kenttools commands.

prompt>

The following commands from kenttools are very useful:

center - find the RA and DEC offsets
seeingall - measures the seeing
psc - provides a heap of useful data
bigload - display the last image
load - display a single CCD from the last image
inv - show an inventory of all the fits files taken during the night

More details about kenttools can be found here: ​https://cdcvs.fnal.gov/redmine/projects/desops/wiki/Introduction_to_kentools

In addition to kenttools, there is an even more useful version of inv in godb called "qcInv". To use this instead, log onto ?? as ?? " and type "godb" and then "qcInv" at the pgtcl> prompt.

Checking the Sky Brightness, Seeing and Transparency

The observing strategy files take as input these environmental conditions for setting the exposure times to achieve the required MzLS survey depth.

From an IDL prompt, use the MOSSTAT routine to analyze the latest image on disk:

Not yet installed at Mayall

   IDL> mosstat

There are keyword options that allow you to choose different exposure numbers or CCDs within that exposure. For example, to analyze chip 'S4' of the exposure number 123456, type:

   IDL> mosstat, 123456, ext='S4'

The full documentation can be seen with:

   IDL> doc_library,'mosstat'

If the telescope pointing has gone off by more than about 30 arcsec (which happens!), then this routine will likely fail to match stars. In that case, the kenttools routine center described above should be used to find the telescope offset. Stop observing and ask the telescope operator to apply a pointing offset.

One can also check the observing progress using the almanac command:

IDL> almanac, 10001, /noprint

will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window). This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether one should create and upload a new JSON observing script. The /noprint just prevents the decstat output for every frame; if you want to see that, then don't use the /noprint keyword.

Creating an almanac file with the observations for the night

At the end of the night, an almanac file should be generated for the night and checked into svn.

cd $??_OBS/observing/logs
idl
   almanac, <START-EXPOSURE>, <END-EXPOSURE>
   exit
svn add Almanac*
svn commit Almanac*

For example, for the night of March 26/27, 2015, this is done with:

cd $??_OBS/observing/logs
idl
   almanac, 424978, 425143
   exit
svn add Almanac_2015-03-26.*
svn commit Almanac_2015-03-26.*

There are actually two versions of this file, one that is an ASCII file (with .txt extension) and one that is a FITS file (with .fits extension).

Writing the human logs

Please write about what happened during your night: weather conditions, which pass you observed, and telescope problems. Follow the example on the pages at DecamLegacy/ObservingLogs.

Catastrophically bad frames (such as saturated frames, or where the telescope moved) are recorded and svn-checked-in to observing/logs/bad_expid.txt.

In case of technical difficulties

We have in the past encountered technical difficulties that have restricted where we can observe to a specific range in RA/Dec. For instance, in April 2015, we had a night during which the windscreen was stuck, requiring us to stick to dec < -5 deg tiles.

If this happens during your run, you have several options:

1. First, try running mosaicstrategy.py with an appropriate --airmass-limit and/or --moon-sep-limit to try to keep to the required range of tiles.
   
2. Shorten your observing period (using --start-date,--start-time and --end-date,--end-time) such that it's scheduling fewer tiles.
3. Increase the assumed seeing such that it schedules longer exposures.

For reference, the tiles we’d be observing with MOSAIC3 in z-band are:

IN_DESI = 1 DEC > 30 88 < RA < 301 PASS <= 3

That’s 48617 tiles.

Example run-through for a given night

Log into the mzls account on the mayall-idl machine.

user@anyshell>ssh ??@??

Update everything

Update the code, log files and most importantly the tile file:

cd ~/mosaic3
svn update

Create nightly plan

Create the nightly plan from the available tiles. This will read the tile file to see which ones are already completed.

cd ~/mosaic3
python py/mosaicstrategy.py -t obstatus/mosaic-tiles_obstatus.fits -d *UT date at ''start'' of night* -p *portion of night* -pass *pass number*
python ./py/jnox.py --in *JSON input file --out *Output shell script
scp *output shell script to the /home/observer/exec area on mosaic3

go to /home/observer/exec, and execute the mzls*.sh script.

Start the automated script for updating the tile file

From a terminal on the mayall-idl compute, in the mzls account, start this running:

idl
   muptiles

From another terminal in the ?? account, start this running:

idl
   mosstat_continuous

This will run mosstat on each frame as it shows up and display the results on the screen.

Start a window for running kenttools

Kenttools can only be run as the "??", and not from our "??" account.

Open a new window logged in as ?? on ??:

observer> observer
prompt> inv

Create the Almanac and Almanac plot files

At the end of the night, create the Almanac files and check them into svn.

- cd observing/logs
- idl
- almanac,<first exp num>
- plotalmanac,'Almanac_date.fits',ps='plot_Almanac_date'
- exit
- svn add *Almanac_date*
- svn commit

Create a log file

- cd logs/
- observer (kent tootls on observer1)
- inv
- cut and paste the output from the first exposure number to the bottom 
  into a file log_date.txt (on observer2)
- use this file to determine the number of g, r, and z images for the 
  summary table on the wiki: grep " g " log_date.txt | wc

Generating a Coverage Map

The code pro/coverage_plot.pro will generate coverage maps for a given date range using the info in mosaic-tiles_obstatus.fits. To run it, login as ??@?? and start IDL:

IDL>.com coverage_plot

To produce the plot for a single night use, for example:

IDL> coverage_plot, datemin='2015-04-07', datemax='2015-04-07'

To produce a plot for a range of nights use:

IDL> coverage_plot, datemin='date-start', datemax='date-end'

To produce a plot for all nights to date use:

IDL> coverage_plot

All plots will be put in $??_OBS/obstatus/Plots. The default is to output a PDF but there are flags to output other formats.

The code pro/coverage_plot_alldates.pro will produce a set of cumulative coverage plots (one per day) over a date range. Within IDL when logged in as ??@?? you enter:

IDL> .com coverage_plot_alldates
IDL> coverage_plot_alldates, datemin='date-start', datemax='date-end'

All plots, consecutively numbered, will be put in $??_OBS/obstatus/Plots/PDFs and can be combined into a movie, e.g with the free graphicconverter ​http://www.lemkesoft.de/en/image-editing-slideshow-browser-batch-conversion-metadata-and-more-on-your-mac/ program on Macs.

Ben Weaver would prefer that we do not upload any plots to SVN but rather just the code. So feel free to mail plots around but there is no need to check them in.

---

Pages linking to PublicPages/MayallZbandLegacy/NotesforObservers:

• MayallZbandLegacy
• MayallZbandLegacy/NotesforObservers
• MayallZbandLegacy/ObservingSchedule
• MayallZbandLegacy/Run08
• MayallZbandLegacy/Run13
• PublicPages/MayallZbandLegacy
• PublicPages/MayallZbandLegacy/NotesforObservers/Copilot