Changes between Version 1 and Version 2 of PublicPages/MayallZbandLegacy/NotesforObservers


Ignore:
Timestamp:
Apr 26, 2015 1:00:55 PM (9 years ago)
Author:
Robert Blum
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PublicPages/MayallZbandLegacy/NotesforObservers

    v1 v2  
    1010
    1111Two environment variables define the locations of data and code:
    12    '''$??_DATA=/home4/images/fits/2014B-0404''' -- This is where our raw data files are written [[BR]]
    13    '''$??_OBS=$HOME/observing''' -- This contains the observing product with code and versioned log files. [[BR]]
    14 To set these, type "setenv ??_DATA /home4/images/fits/2014B-0404"
     12   '''$??_DATA=''' -- This is where our raw data files are written [[BR]]
     13   '''$??_OBS=$HOME/mosaic3''' -- This contains the observing product with code and versioned log files. [[BR]]
     14To set these, type "setenv ??_DATA <path>"
    1515
    1616Other directories in the home directory:
     
    3535everything consistent.)
    3636
    37 
    38 = Visualizing the available tiles (not required) =
    39 
    40 The IDL routine WHICHTILES_TONIGHT can be used to visualize the available tiles for the night:
    41 {{{
    42 IDL> whichtiles_tonight
    43 SYNTAX: whichtiles_tonight,utdate,goodtiles,band,moonlimit=moonlimit,amlimit=amlimit
    44    Inputs:
    45        utdate - string - the UT date for the ''midpoint'' of the night (e.g., "2014-08-12")
    46        band - string - the observation band preferred (e.g., "g", "z", or "r") - default is "z"
    47    Keywords:
    48        moonlimit = minimum acceptable angle from the moon (default=20 deg)
    49        amlimit = maximum acceptable airmass (default=2.2)
    50        ps - string - name of an output postscript file; if not provided, defaults to a screen plot
    51    Output:
    52         goodtiles = boolean 1 or 0 for each tile in decam-tiles.fits
    53 
    54 To execute the program for a given night:
    55 IDL> whichtiles_tonight,'2014-08-15',good,'z',ps='plan_zband_aug15ut'
    56 IDL> whichtiles_tonight,'2014-08-15',good,'r',moonlimit=50,ps='plan_rband_aug15ut'
    57 IDL> whichtiles_tonight,'2014-08-15',good,'g',moonlimit=100,amlimit=1.8,ps='plan_gband_aug15ut'
    58 }}}
    59 
    60 These will produce encapsulated postscript files that are useful for visualizing the
    61 possible fields for tonight.
    62 
    6337= Official Nightly Strategy =
    64 '''Update 4/9/2015'''
    65 Python is now available to the decals user. Do the following to update the current NMSL scripts at any time.
     38'''Update 4/26/2015'''
     39Do the following to update the current NMSL scripts at any time.
    6640
    6741        - login to the home directory of ??
    68         - cd observing
    69         - python2.7 py/nightlystrategy.py -t obstatus/mosaic-tiles_obstatus.fits -d 2015-04-08 -p 1 -pass 1 --moon-sep-limit 40 --start-date 2015-04-09 --start-time 02:00:00 -seez 1.15 -sbz 18.45 --dir /home/observer/exec/<dir_name>
    70 
    71 Assuming that uptiles is running in an IDL window on ??, this will always use the most up-to-date version of the mosaic-tiles_obstatus.fits file. Once this finishes running (usually a couple of minutes), cd to <dir_name>:
    72 
    73         > ./mosaic_<UTdate>_plan.sh
    74 
    75 = Instructions and details for nightlystrategy.py =
     42        - cd mosaic3
     43        - python2.7 py/mosaic3strategy.py -t obstatus/mosaic-tiles_obstatus.fits -d 2015-04-08 -p 1 -pass 1 --moon-sep-limit 40 --start-date 2015-04-09 --start-time 02:00:00 -seez 1.15 -sbz 18.45 --dir <dir_name>
     44
     45Assuming that uptiles is running in an IDL window on ??, this will always use the most up-to-date version of the mosaic-tiles_obstatus.fits file. Once this finishes running (usually a couple of minutes), cd to <dir_name> and execute the script:
     46
     47        prompt> ./mosaic_<UTdate>_plan.sh
     48
     49= Instructions and details for mosaicstrategy.py =
    7650'''written by A. Patej''' 
    7751
    78 nightlystrategy.py is a Python script that can generate a nightly observing plan. It requires several Python modules:
     52mosaicstrategy.py is a Python script that can generate a nightly observing plan. It requires several Python modules:
    7953    - numpy
    8054    - pylab (Matplotlib)
    8155    - ephem (Pyephem)
    8256    - pyfits
     57
     58The code is adopted from the DecamLegacy project. See those wiki pages for more information.
    8359
    8460=== Selection of Pass 1, 2, or 3 ===
     
    10076=== Running the script ===
    10177
    102 At a minimum, the nightlystrategy.py script can be run with these inputs:
     78At a minimum, the mosaicstrategy.py script can be run with these inputs:
    10379
    10480    -t *name of file with tile centers* [[BR]]
     
    10783    -pass *desired pass - input 1, 2, or 3* [[BR]]
    10884    --dir *name of the dir where the output scripts are located*
    109 For instance, to generate a pass 3 plan for the entire night of August 14, 2014, I could run:
    110 {{{
    111 python ./nightlystrategy.py -t moasic-tiles_obstatus.fits -d 2014-08-14 -p 1.0 -pass 3 --dir /home/observer/exec/mzls-utdate/
     85For instance, to generate a pass 3 plan for the entire night of April 26, 2015, run:
     86{{{
     87python ./nightlystrategy.py -t moasic-tiles_obstatus.fits -d 2015-04-26 -p 1.0 -pass 3 --dir obsScripts
    11288}}}
    11389   
    11490Additional optional arguments:
    115911. Observing conditions: [[BR]]
    116    -sb(f) *sky brightness in filter f; replace (f) = g, r, or z* [[BR]]
    117    -see(f) *seeing in filter f; replace (f) = g, r, or z* [[BR]]
    118    -transparency *value of the transparency* [[BR]]
     92   -sb(f) *sky brightness in filter f; replace (f) = g, r, or z* in mag/sqarcsec [[BR]]
     93   -see(f) *seeing in filter f; replace (f) = g, r, or z* in arcsec [[BR]]
     94   -transparency *value of the transparency* (0-1.0) [[BR]]
    11995
    120962. Observing constraints: [[BR]]
    12197   --airmass-limit *maximum allowed airmass, default is 2.3* [[BR]]
    12298   --moon-sep-limit *minimum allowed moon separation, default is 50.0 deg* [[BR]]
    123    --moon-alt-limit *moon altitude required for switching between z and g,r* [[BR]]
     99   --moon-alt-limit *moon altitude in deg required for switching between z and g,r* [[BR]]
    124100
    1251013. Custom Scheduling: [[BR]]
     
    134110Assuming 18.2 as z-band sky brightness, 1.2" seeing in z and 1.0" seeing in r, photometric conditions (pass 1) and allowing a moon separation of 40 deg and including DESI tiles:
    135111{{{
    136 python ./nightlystrategy.py -t mosaic-tiles_obstatus.fits -d 2015-03-24 -p 1.0 -pass 1 -sbz 18.2 -seer 1.0 -seez 1.2 --moon-sep-limit 40.0 --expand
     112python ./mosaicstrategy.py -t mosaic-tiles_obstatus.fits -d 2015-03-24 -p 1.0 -pass 1 -sbz 18.2 -seer 1.0 -seez 1.2 --moon-sep-limit 40.0 --expand
    137113}}}
    138114
     
    140116To generate a plan for just two hours on the night that starts on 2015-04-06 between 05:00 and 07:00 UTC:
    141117{{{
    142 python ./nightlystrategy.py -t mosaic-tiles_obstatus.fits -d 2015-04-06 -p 1.0 -pass 1 --start-date 2015-04-07 --start-time 05:00:00 --end-date 2015-04-07 --end-time 07:00:00 --moon-sep-limit 40.0
     118python ./mosaicstrategy.py -t mosaic-tiles_obstatus.fits -d 2015-04-06 -p 1.0 -pass 1 --start-date 2015-04-07 --start-time 05:00:00 --end-date 2015-04-07 --end-time 07:00:00 --moon-sep-limit 40.0
    143119}}}
    144120
    145121=== Outputs ===
    146122
    147 This script will yield 3 output data files:
    148 
    149     1. text file called "strategy_mm-dd.txt", with information about all the pointings for the night (exp time, filter, approximately when they should be run, airmass, etc). If there's a late start or gap in observing due weather, etc you can look in this file and find the tile corresponding to your revised start time and remove all the ones prior to it in the Exposure Queue. It is a good idea to check this file before accepting the plan.
     123This script will many output data files:
     124
     125    1. a single text file called "strategy_mm-dd.txt", with information about all the pointings for the night (exp time, filter, approximately when they should be run, airmass, etc). There is a corresponding plot of what has been observed previously (by pass) and what you will be observing given the plan for the night: saved as a pdf, 'plot_plan_yyy-mm-dd.pdf' (gray = z, red = r, green = g). Planets and moon are plotted as well.
    150126   
    151     2. JSON script called "decals_mm-dd_plan.json" for the night corresponding to the plan in this text file as well as some plots of what has been currently observed on previous nights and what you will be observing given the plan for the night, which are saved as a pdf, 'plot_plan_mm-dd.pdf' (gray = z, red = r, green = g)
    152 
    153    3. A FITS file called "decam-tiles_obstatus_update_yyyy-mm-dd.fits" that is a temporary updated decam obstatus file that assumes all planned observations were completed. This should not be used during the run; instead use the updated tiles file obtained from running the idl program, uptiles (see section on "Creating the Updated Observing Status File" below).
    154 
     127    2. many executable script files of the form "Mosaic_Survey<tile id>.sh" for the night corresponding to the plan. There is one script file for each pointing in the plan. A single meta-script is produced  that will call those ".sh" files: moasic_<ut date>_plan.sh.
     128 
     129
     130   3. A FITS file called "decam-tiles_obstatus_update_yyyy-mm-dd.fits" that is a temporary updated mosaic obstatus file that assumes all planned observations were completed. This should not be used during the run; instead use the updated tiles file obtained from running the idl program, uptiles (see section on "Creating the Updated Observing Status File" below).
    155131
    156132= Updating the "observed tiles" file =
     
    165141At the end of the night, you should check the updated tile file into the svn repository:
    166142{{{
    167    cd $DECALS_OBS/obstatus
    168    svn commit obstatus/decam-tiles_obstatus.fits --username <YOUR-NAME>
     143   cd $??_OBS/obstatus
     144   svn commit obstatus/mosaic-tiles_obstatus.fits --username <YOUR-NAME>
    169145}}}
    170146
    171147= Use of Steve Kent's tools for monitoring observing =
     148
     149=== Not yet installed at Mayall. ===
    172150
    173151During observing, please check frames as they read out. To use kenttools remotely, you must be
     
    193171''pgtcl>'' prompt.
    194172
    195 
    196173= Checking the Sky Brightness, Seeing and Transparency  =
    197174
     
    200177
    201178From an IDL prompt, use the DECSTAT routine to analyze the latest image on disk:
     179=== Not yet installed at Mayall ===
     180
    202181{{{
    203182   IDL> decstat
     
    214193
    215194If the telescope pointing has gone off by more than about 30 arcsec (which happens!), then
    216 this routine will likely fail to match stars.  In that case, the kenttools routine '''center''' described above
    217 should be used to find the telescope offset.  Stop observing and ask the telescope operator to apply
    218 a pointing offset.
     195this routine will likely fail to match stars.  In that case, the kenttools routine '''center''' described above should be used to find the telescope offset.  Stop observing and ask the telescope operator to apply a pointing offset.
    219196
    220197One can also check the observing progress using the almanac command:
    221198  IDL> almanac, 10001, /noprint
    222 will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window).
    223 This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether
    224 one should create and upload a new JSON observing script. The /noprint just prevents the decstat output
    225 for every frame; if you want to see that, then don't use the /noprint keyword.
     199will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window).  This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether  one should create and upload a new JSON observing script. The /noprint just prevents the decstat output for every frame; if you want to see that, then don't use the /noprint keyword.
    226200
    227201= Creating an almanac file with the observations for the night =
     
    238212For example, for the night of March 26/27, 2015, this is done with:
    239213{{{
    240 cd $DECALS_OBS/observing/logs
     214cd $??_OBS/observing/logs
    241215idl
    242216   almanac, 424978, 425143
     
    257231and svn-checked-in to  '''observing/logs/bad_expid.txt'''.
    258232
    259 Historically, we also made some files in $DECALS_OBS/logs/logfile-$DATE.txt that are
    260 output from the kenttools "inv" command.  These are probably not really needed any longer.
    261 
    262233= In case of technical difficulties =
    263234
     
    265236
    266237If this happens during your run, you have several options:[[BR]]
    267 1. First, try running nightlystrategy.py with an appropriate --airmass-limit and/or --moon-sep-limit to try to keep to the required range of tiles. [[BR]]
    268 2. If you run out of tiles with these options, add the --expand option, which allows the selection of DESI tiles in addition to SDSS tiles, in addition to any airmass-limit you need.
    269 3. '''Shorten''' your observing period (using --start-date,--start-time and --end-date,--end-time)  such that it's scheduling fewer tiles.
    270 4. Increase the assumed seeing such that it schedules longer exposures.
    271 
    272 If these options don't work, you will have to modify our observing footprint, which is encoded in decam-tiles_obstatus.fits. This file has, among other things, three columns named 'IN_SDSS', 'IN_DESI', and 'IN_DES'. For each tile the value in these columns is either 1 or 0.
    273 
    274 We define our fiducial footprint as 'IN_SDSS' = 1 & 'IN_DES' = 0. If you allow the selection of DESI tiles (using --expand), then the footprint is defined by ( 'IN_SDSS' = 1 & 'IN_DES' = 0 ) or ( 'IN_DESI' = 1 & 'IN_DES' = 0). If you have no tiles to observe that are within the region defined this way, you need to modify the tiles file.
    275 
    276 In this case, please do the following:
    277 1. Make a copy of decam-tiles_obstatus.fits (call it something like temp_decam-tiles_yyyy-mm-dd.fits)
    278 2. Open temp_decam-tiles_yyyy-mm-dd.fits using your favorite fits file editor (python/pyfits, idl, etc) and modify the file to set 'IN_SDSS' = 0 & 'IN_DESI' = 0 for all tiles that are outside the range allowed by the telescope that night. Then set 'IN_SDSS' = 1 & 'IN_DESI' = 1 for tiles that satisfy your allowed RA/dec range and also some possible quality constraints. For instance, in April 2015, David suggested adding the following tiles to satisfy dec < -5:
    279 {{{
    280 a) IN_DESI = 0
    281 b) EBV_MED < 0.1
    282 c) 120 < RA < 270
    283 d) DEC > -19
    284 }}}
    285 3. Use the modified temp_decam-tiles_yyyy-mm-dd.fits as your -t argument in nightlystrategy. For instance,
    286 {{{
    287 python ./nightlystrategy.py -t temp_decam-tiles_yyyy-mm-dd.fits -d yyyy-mm-dd -p 1.0 -pass 3 --expand
    288 }}}
    289 
    290 4. Set two versions of uptiles running, one to update the official "decam-tiles_obstatus.fits" file, so that we have an official record of the tiles being observed, the other to update 'temp_decam-tiles_yyyy-mm-dd.fits,' which you can use to recalculate the JSONs if conditions change. You can run uptiles as follows:
    291 {{{
    292 uptiles
    293 uptiles, tfile='temp_decam-tiles_yyyy-mm-dd.fits'
    294 }}}
    295 
    296 5. At the end of the night, make sure check in decam-tiles_obstatus.fits to svn; you can leave 'temp_decam-tiles_yyyy-mm-dd.fits' on observer2, or place it in a temp folder on svn.
    297 
     2381. First, try running mosaicstrategy.py with an appropriate --airmass-limit and/or --moon-sep-limit to try to keep to the required range of tiles. [[BR]]
     2392. '''Shorten''' your observing period (using --start-date,--start-time and --end-date,--end-time)  such that it's scheduling fewer tiles.
     2403. Increase the assumed seeing such that it schedules longer exposures.
     241
     242For reference, the tiles we’d be observing with MOSAIC3 in z-band are:
     243  IN_DESI = 1
     244  DEC > 30
     245  88 < RA < 301
     246  PASS <= 3
     247That’s 48617 tiles.
    298248
    299249= Example run-through for a given night =
    300250
    301 Log into the declsp account on the observer2 machine.
    302 {{{
    303 user@anyshell>ssh declsp@observer2
     251Log into the ?? account on the ?? machine.
     252{{{
     253user@anyshell>ssh ??@??
    304254}}}
    305255
     
    308258Update the code, log files and most importantly the tile file:
    309259{{{
    310 cd ~/observing
     260cd ~/mosaic3
    311261svn update
    312262}}}
     
    316266Create the nightly plan from the available tiles.  This will read the tile file to see which ones are already completed.
    317267{{{
    318 cd ~/observing
    319 python2.7 py/nightlystrategy.py -t obstatus/decam-tiles_obstatus.fits -d *UT date at ''start'' of night* -p *portion of night* -pass *pass number*
    320 }}}
    321 
    322 The output JSON file is then loaded into SISPI. The DECaLS JSONS that will be loaded into SISPI should be placed in observer2.ctio.noao.edu:/usr/remote/user/DECamObserver/ExposureScripts/User_scripts/BigProjects/DECaLS/
    323 
    324 In the observer console, select the tab "Exposure Control" and press "Load Exposure Script." Select your JSON and press "submit." The exposures will then be appended to the bottom of the Exposure Queue.
    325 
    326 You can manually modify the exposure queue. Make sure the padlock icon is unlocked if you wish to make changes. You can select a queued exposure by clicking on it; you can press "delete" to delete it, or use your cursor to click and drag it to a new position. To select a range of exposures, click on the first one, then shift+click on the last one.
     268cd ~/mosaic3
     269python2.7 py/mosaicstrategy.py -t obstatus/mosaic-tiles_obstatus.fits -d *UT date at ''start'' of night* -p *portion of night* -pass *pass number*
     270}}}
     271
     272go to /home/observer/exec, and execute the mosaic_<ut date>_plan.sh script.
    327273
    328274=== Start the automated script for updating the tile file ===
    329275
    330 From a terminal in the declsp account, start this running:
     276From a terminal in the ?? account, start this running:
    331277{{{
    332278idl
     
    334280}}}
    335281
    336 From another terminal in the declsp account, start this running:
     282From another terminal in the ?? account, start this running:
    337283{{{
    338284idl
     
    343289=== Start a window for running kenttools ===
    344290
    345 Kenttools can only be run as the "DECamObserver", and not from our "declsp" account.  (It was possible to do directly from
    346 our account before March 2015, so something changed.)
    347 
    348 Open a new window logged in as ''DECamObserver'' on observer2:
     291Kenttools can only be run as the "??", and not from our "??" account.
     292
     293Open a new window logged in as ''??'' on ??:
    349294{{{
    350295observer> observer
     
    378323=== Generating a Coverage Map ===
    379324
    380 The code pro/coverage_plot.pro will generate coverage maps for a given date range using the info in decam-tiles_obstatus.fits.  To run it, login as declsp@observer 2 and start IDL:
     325The code pro/coverage_plot.pro will generate coverage maps for a given date range using the info in mosaic-tiles_obstatus.fits.  To run it, login as ??@?? and start IDL:
    381326{{{
    382327IDL>.com coverage_plot
     
    394339IDL> coverage_plot
    395340}}}
    396 All plots will be put in $DECALS_OBS/obstatus/Plots.  The default is to output a PDF but there are flags to output other formats. 
    397 
    398 The code pro/coverage_plot_alldates.pro will produce a set of cumulative coverage plots (one per day) over a date range.  Within IDL when logged in as declsp@observer2 you enter:
     341All plots will be put in $??_OBS/obstatus/Plots.  The default is to output a PDF but there are flags to output other formats. 
     342
     343The code pro/coverage_plot_alldates.pro will produce a set of cumulative coverage plots (one per day) over a date range.  Within IDL when logged in as ??@?? you enter:
    399344{{{
    400345IDL> .com coverage_plot_alldates
     
    402347}}}
    403348
    404 All plots, consecutively numbered, will be put in $DECALS_OBS/obstatus/Plots/PDFs and can be combined into a movie, e.g with the free graphicconverter [http://www.lemkesoft.de/en/image-editing-slideshow-browser-batch-conversion-metadata-and-more-on-your-mac/] program on Macs. 
     349All plots, consecutively numbered, will be put in $??_OBS/obstatus/Plots/PDFs and can be combined into a movie, e.g with the free graphicconverter [http://www.lemkesoft.de/en/image-editing-slideshow-browser-batch-conversion-metadata-and-more-on-your-mac/] program on Macs. 
    405350
    406351Ben Weaver would prefer that we do not upload any plots to SVN but rather just the code.  So feel free to mail plots around but there is no need to check them in.