wiki:PublicPages/MayallZbandLegacy/NotesforObservers

Version 147 (modified by Jim Annis, 9 years ago) (diff)

--

OBSERVING INSTRUCTIONS FOR MzLS (PROPID 2016A-0453)

Still need to incorporate the following instructions:

  • Running the guiding loop -- although we typically do not guide on the Mayall for these 1-2 min exposures

These are instructions for observing for MzLS.


VERY VERY IMPORTANT

1. DO NOT OVEREXPOSE MOSAIC CCDs to LIGHT!

Prolonged exposure can damage and even destroy the devices. The following rules are in effect:

  1. Operation on sky is restricted to between 10 deg twilights.
  2. Operation is restricted to when OAs or other NOAO technical/scientific staff are present
  3. The cameras and MOSAIC software should be shut down when observing / NOAO personnel are not present
  4. Sky values should be kept < 40,000 adu (<30,000 adu is preferable).

2. PLEASE TALK TO THE OAs ABOUT SAFETY AT THE TELESCOPE

  1. Please read the Safety Rules carefully and FOLLOW THEM!
  2. Do not leave the control room at night without telling the OA
  3. Do not go into the dome without the OA's permission
  4. Do not go exploring by yourself
  5. Carry a flashlight and radio at night

Read the Manual for MOSAIC operations

To see the Mosaic1.1 manual, go here: http://www.noao.edu/kpno/mosaic/manual

To see the Mosaic3 manual go here: http://www.noao.edu/kpno/mosaic/manual/Mosaic3Manual.pdf

Some Preliminaries

Who is your OA for tonight? See the OA Schedule

Observing can only be done from the Mayall control room or from within the KPNO/NOAO network using a VPN.

The telescope and low-level instrument control runs on observer@mayall-2.kpno.noao.edu . Log on to the mayall-2 computer:

Username : 4meter
Password : <Ask someone on our team or the staff>

To run MOSAIC3, use the MOSAIC3 Menu GUI. Or you can also log into the mosaic3 computer from mayall-2:

ssh -XY observer mosaic3

and use command-line nocs commands.
PLEASE BE CAREFUL ON THIS COMPUTER - IT RUNS THE CONTROLLERS AND THE INSTRUMENT!!'''

Our observing scripts and data reduction runs on mzls@mayall-idl.kpno.noao.edu :

Username : mzls
Password : <Ask someone on our team or previous observers>

These environment variables define the locations of data and code:

$HOME=/home/mzls
$MOS3_DATA= -- This is where our raw data files are written. Should be something like /mosaic3/data2/observer/20160202. This is a network mounted disk on the machine mosaic3.
$MOS3_OBS=$HOME/products/mosaic3 -- This contains the observing product with code and versioned log files.
$PS1CAT_DIR=$HOME/ps1/chunks-qz-star-v2 -- Pan-STARRS1 catalogs used by the IDL MOSSTAT routine for computing astrometric offsets and photometry

These path names are set in the .bashrc file, or can be set for example with "export MOS3_DATA=/mosaic3/data2/observer". The $MOS3_DATA directory must be updated to point to the current night's data.

Other directories in the home directory are:

astrometry/, legacypipe, obsbot, tractor, wcslib -- Python code for running copilot
data/ -- Link to the top-level data directory, where each night is stored in subdirectories like "20151213"
exec/ -- Cross-mount to mosaic.kpno.noao.edu:/home/observer/exec
products/ -- Code checked out from the SDSS and the DESI svn repositories

Documentation for the IDL scripts can be printed from the IDL prompt with the DOC_LIBRARY command, for example:

      idl
      IDL> doc_library,'mosstat'

Date convention

All dates in log files are set to the local date of the beginning of the night. For example, any data taken during the night of December 13/14, 2015 will be written as 2015-12-13. This is consistent with how the NOAO Science Archive timestamps and saves the raw data files.

Are we there yet?

The tiles we are observing with MOSAIC3 in z-band are tracked in the file ~/products/mosaic3/obsstatus/mosaic-tiles_obstatus.fits with the following cuts:

  IN_DESI = 1
  DEC >= 30
  88 < RA < 301
  PASS <= 3

That’s 44,422 tiles. This will likely be further limited further to the 41,188 tiles at DEC >= 34. About 5% of the tiles have been removed from the list where there's a star brighter than V=6 within 0.35 deg of the tile centers.

The tiles are broken into three passes. Each pass, pass1, pass2, and pass3, covers the basic footprint and each is offset optimally from the others. The tile are thus now fixed or defined on the sky for the duration of the survey. By definition:

pass1 is the high quality, photometric coverage. To the greatest extent possible, we want to ensure pass one tiles are the best. As such, pass1 should be executed on photometric nights with good seeing (< 1.3"). The observer should not start pass1 if its a not a true photometric night or if 3-4 hour contiguous blocks are not stable and clear. The robot observing program will choose pass1 for seeing < 1.25", transparency > 90% and sky brightness not worse than 0.25 mag brighter than the fiducial. But the observer should really be setting pass1 based on whether its a very stable photometric night from beginning to end. Forcing the pass is described here: MosBot.

pass2 is the next best pass. Seeing should be < 1.3" or the weather is photometric (but seeing worse than 1.3"). Pass2 and 3 may alternate throughout a given night and the mosbot observing robot will do this automatically based on seeing, transparency, and sky brightness. Pass2 could also be done if no pass1 tiles are available. The robot will stick with pass2 when conditions are a little worse than this. Specifically seeing < 2" and trans > 70%. Observers can force pass3 if they think the conditions are poor, even though the robot might say pass2. This is especially encouraged if conditions are variable on short timescales and the robot is moving frequently between pass2 and pass3.

pass3 is the filler or worst pass executed when conditions are bad. Seeing > 1.3 and its not photometric (trans < 90%). Pass3 may be done in good conditions when no pass1 or pass2 tiles are available.

If you want to use only pass2 or pass3 on a given night (because, say, there is high cirrus and you are sure the night will not be photometric), then create a file called "nopass1" (by typing "touch nopass1") in the ~/exec/mosbot directory. This will restrict mosbot to choose only between passes 2 and 3.


READ THIS IN PREP FOR OBSERVING


AFTERNOON DOMEFLATS


MayallZbandLegacy/NotesforObservers/Domeflats


EXAMPLE RUN THROUGH FOR A GIVEN NIGHT


[1] Update everything

Log into the mzls account on the mayall-idl machine.

Update the code, log files and most importantly the tile file: (if people have been working on the code and tile file during the day, this step may not be necessary)

cd ~/products/mosaic3
svn up
cd ~/products/observing
svn up

[2] Set the paths

The data path must be changed to point to the current night's data. Edit the entry for MOS3_DATA ~/.bashrc file, for example if the start of the night is 13 Dec 2015 change this to

export MOS3_DATA=/mosaic3/data2/observer/20151213

Don't forget to source the file if you plan to continue to use this terminal.

[3] Create nightly plan

There are 3 nightly plan files, one with a list of tiles to observe for each of the 3 passes. These files are in JSON format, and specify the pass 1, pass 2 or pass 3 tiles to be observed at each timestamp through the night. The selection of pass number while observing will depend upon the weather conditions as described at MayallZbandLegacy/ObservingStrategy, and will either be selected automatically or can be forced by the user.

The IDL MSTRATEGY code is used to create the above plan files, as described at MayallZbandLegacy/ObservingStrategy

Arjun or David may have already provided a set of plan files for the night and posted a note to the mayall-obs e-mail list saying so. These files would be checked into the "mosaic3" product here:

   ~/products/mosaic3/json/<DATE>-p1.json
   ~/products/mosaic3/json/<DATE>-p2.json
   ~/products/mosaic3/json/<DATE>-p3.json

and copied into the observing directory where they are read by mosbot here:

   ~/obsbot/pass1.json
   ~/obsbot/pass2.json
   ~/obsbot/pass3.json

If the plan files have not been generated for the night, please do so as follows:

   idl
      mstrategy, copydir='/home/mzls/obsbot'
      exit

Then check in the plan files to the "mosaic3" product as follows:

   cd ~/products/mosaic3/json
   svn add *.json
   svn commit *.json -m "plan files for tonight"

There are plot files in that directory too. We don't check those in, but it's informative to post the planned coverage map ~/products/mosaic3/json/<DATE>.pdf to mayall-obs@desi.lbl.gov .

Deprecated instructions using a modified version of "nightlystrategy" is at MayallZbandLegacy/NightlyStrategyOld

[4] Start up mosaic control software

On mayall-2, double click on the MOSAIC3 icon, which brings up the MOSAIC3 Menu on the left edge of the screen.

  1. Start the camera control program by pressing the yellow "Start Cameras" button. Wait for this to finish, then dismiss the screen by typing any key as instructed
  2. Start the MOSAIC3 NOCS software by pressing the blue "Start MOSAIC" button. This launches a blue xterm. Move it out of the way and watch all the windows come up
  3. Rearrange the desktop as needed.

If, for some reason, the buttons do not work, you can start up the software on a command line as follows. On mayall-2 open a terminal window and:

 ssh -XY observer@mosaic3
 nocs start ccp (this is equivalent to the "Start Cameras" button on the MOSAIC3 Menu)
 nocs start all (this is equivalent to the "Start MOSAIC" button on the MOSAIC3 Menu)

Once nocs is up and running, rearrange windows as desired, and check the status of the system by typing the following in a nocs terminal window:

nocs status all
nocs fullstatus ccp

If you want to know what these commands actually do, see here: MayallZbandLegacy/NotesforObservers/MOSAICGUI_Notes

  1. Launch CCD temperature monitor from "MOSAIC Temps" icon (CCD and Dewar temps should be around 173C and 90C respectively)
  2. Launch TCS acorn monitor from "VDU" icon
  3. 4MAPS monitor from "4MAPS" icon
  4. Launch Truss temperature monitor from "Truss C" icon

All these icons are on the right side of the left hand screen of the mayall-2 computer display. Many or all of these may be running already. They can stay up. Rearrange the busy desktop as needed ...

[5] Set the PROP-ID and Project Info

On the NGUI window, press the "Set Project" button and fill out the relevant information. For example,

Principal Investigator: Arjun Dey
PIs Email Address: dey@noao.edu
Actual Observers: Tristram Shandy, Bertram Wooster
Observing Assistant: Karen Butler
OA's Email Address: 4meter@noao.edu
Proposal Identifier: 2016A-0453
Telescope System: KPNO Mayall 4m
Science Instrument: Mosaic 3

Please ensure the Proposal Identifier number is correct''

Then, in one of the NOCS xterm windows, type: "nocs set project"

[6] Take a test Zero image to ensure system is working

Check that all is well by taking a test zero exposure. On the IRAF window, check the image statistics by cd-ing to the correct directory and typing "mscstat <filename>" All rms values should be about 4-6 adu/pix; the exception is amplifier [6] which has an rms~8-10adu/pix. If any one amplifier shows very high noise, then execute the following commands in a nocs window:

nocs reset ccp
nocs init ccp

Then take two more zeros; the first one will be junk, but the second one should be OK.

[7] Take dome flats and zeros

During the afternoon (after 4pm), take dome flats with the telescope pointed at the white spot.

Instructions for Domeflats are here: MayallZbandLegacy/NotesforObservers/Domeflats

Go eat dinner

[8] Just before observing

It is best to start things up after sunset and to be ready to start observing before the 10 deg twilight mark. 10 deg twilight times for KPNO are listed here:

Before 10 deg twilight, you can do all the steps in [8] (i.e., this one)

  1. Start the mayall-idl:1 VNC session.
    A. In a terminal window from mayall-2, ssh to mzls@mayall-idl and log in using the mzls password
    B. type: ~/bin/vnc
    C. Now double click the VNC gui in the mayall-2 dock (i.e., the thing with all the icons at the bottom of the bottom left terminal). 
    This launches a dialog box asking if you want to connect to mayall-idl.kpno.noao.edu:1 . Say "connect" and type in the mzls password. 
    The VNC starts up.
    

The VNC session should automatically start MUPTILES, MOSSTAT and COPILOT. Rearrange as desired.

If it doesn't, open terminals in the VNC session and manually run the scripts for monitoring the observations and updating the tile file:

1a) The FITS file listing which tiles have been completed should be updated throughout the night.

This file is $MOS3_OBS/obstatus/mosaic-tiles_obstatus.fits. The following IDL command will monitor exposures as they are taken throughout the night, automatically updating this file.

From a terminal on the mayall-idl VNC, in the mzls account, start this running:

idl
   muptiles

At the end of the night, you should check the updated tile file into the svn repository.

1b). In a second terminal on the VNC tart the automated script for monitoring the data quality

idl
   mosstat_continuous

This will run mosstat on each frame as it shows up and display the results on the screen.

1c). In a third terminal on the VNC start running copilot:

cd ~/obsbot
python copilot.py
  1. Take a zero image to ensure everything is working. Once the image has been written, use the IRAF window and run mscstat <filename> to make sure the system came up OK. See step [6] above.
  1. Set the focus to some approximate value based on the Truss temperature and the formula:

Focus(zd) = -8400+(1.4-Ttruss)x110

where Ttruss is in deg C.

  1. Ask the telescope operator to point to the first tile in the json file in pass 2 (for normal case, or json of forced pass if starting up non standard). Then ask the telescope to go to a nearby 5th mag star and to center it on the MOSAIC3 focal plane. The coordinates of the first tile can be found by:

mayall-idl > less ~/obsbot/pass2.json

and get the RA, dec values for the OA.

  1. Wait patiently for 10 deg twilight.

[9] Get Ready to Observe - start of night

No on-sky observations are permitted before 10 deg twilight. MayallZbandLegacy/NotesforObservers/TenDegTwilight

  1. At start of night, check telescope pointing (see item [8].4 above; you should already be there!) and zero the telescope coordinates using a bright star placed on the telescope boresight (defined as the center of the mosaic3 focal plane; i.e. in the chip gaps).

When taking the image of the bright star, make sure to create an OBJECT script with NGUI (not a TEST, because MOSSTAT will ignore it), zd filter, 1 - 5 sec exposure. Watch for MOSSTAT to process the image and report the RA, Dec offsets. If mosstat fails, it could be because (a) the telescope is mis-pointed, (b) the telescope is out of focus (set the focus approx using the formula above and try again). Once you are able to determine offsets, give these offsets to the OA with opposite sign. That is, if mosstat reads "RA,Dec offsets = -15.34, 18.32", then you need to provide the OA with the offsets of +15,-18 to zero the telescope coordinates.

You can take another OBJECT frame if you want to check to make sure the offset went in the right direction.

  1. Move back to the first MzLS tile position from item [8].4.
  1. Focus the telescope
    • create a focus script using the NGUI: exposure time 5 or 10 seconds, zd filter, -100 micron focus steps, click Midpoint to *on*, 9 exposures.
    • run the focus script from the /home/observer/exec directory (./FOCUS.sh)
    • copilot will automatically analyze and report the best focus in the mayall-idl window.
    • or alternately, analyze the focus image using the IRAF script mscstarfocus
      • edit the /data/observer/mscfoc.cl script to correct the name of the image that needs to be analyzed
      • run 'cl < /data/observer/mscfoc.cl' in iraf window in data directory.
      • mark about 10 stars around the image; to get a quick idea if you have covered the right focus range mark "g" on the top star in a sequence which will pop up a graph that you will need to type "q" to get out of; mark the top star in each remaining sequence using "m"; "q" to quit; "d" to delete bad points; "q" to quit, then will get best focus value.
    • log the Truss temperature
    • set the telescope focus in the Configuration Monitor: Enter the value in Pedestal focus, hit return, and then hit Apply. If you forget the return, it will do nothing.
    • focus the guiders; this way you can use the guider images to monitor focus drifts

Example of a focus sequence

[10] Observe - all night long!

  1. From the mzls@mayall-idl window, generate the top-level observing script (tonight.sh) assuming that we have three plan files named pass1.json, etc.
       cd ~/obsbot
       python mosbot.py pass1.json pass2.json pass3.json
    

This will start the mosbot script, which will watch the $MOS3_DATA directory for new images, analyze them, and update FUTURE exposure scripts, choosing the pass number and setting the exposure time.

If the conditions look marginal, and you only want to run with pass3, say, just create an empty "forcepass3" file in the ~/exec/mosbot/ directory:

   cd ~/exec/mosbot
   touch forcepass3

If you want to allow only pass 2, then use:

   cd ~/exec/mosbot
   touch forcepass2

If you want to allow only pass 1, then use:

   cd ~/exec/mosbot
   touch forcepass1

(though pass1 should not have to be forced if conditions are good!)

If you want to allow only pass 2 and pass 3, then use:

   cd ~/exec/mosbot
   touch nopass1

Mosbot checks for these files in the order: forcepass 1,2,3 then nopass1.

Force files are not removed by tonight.sh or mosbot.py, so remove them manually to revert or change Mosbot's behavior.

For more options with mosbot, see MayallZbandLegacy/NotesforObservers/MosBot

Note that if you need to escape the script at some point during the night (see below), you will need to CTRL-C to stop mosbot.py and restart it when you are ready to start up again. If you forget this, you will be observing tiles that you already observed earlier in the night.

  1. From the observer@mosaic3 xterm window, start taking exposures using the top-level observing script (tonight.sh):
       cd ~/exec/mosbot
       ./tonight.sh
    
  1. copilot should already be running. If not, start running "copilot" in the mayall-idl VNC window. Copilot keeps a beautiful running plot of observing conditions. From any mzls@mayall-idl window:
       cd ~/obsbot
       python copilot.py
    

Whenever a new image is detected in the data directory (as defined by $MOS3_DATA), a new image is generated as ~/obsbot/recent.png.

See MayallZbandLegacy/NotesforObservers/Copilot

  1. Monitor focus by checking the image quality on each frame or keeping an eye on the mosstat PSF display. Keep track of the truss temperature variation and use the information to modify the focus as needed. Note that the focus may not respond quickly to changes in temperature, so monitor the images carefully before adjusting focus. The Mayall has astigmatism, so one can sometimes tell from the shape of the images which way to move the focus. To stop and do a focus sequence:
  • Create a file to tell obsbot to quit
       touch ~/exec/mosbot/quit
    
  • Wait for the current exposure to complete (at which point the above file is automatically removed)
  • Run a focus sequence (see link below for instructions)
  • Re-start the observing as described above. IMPORTANT make sure to re-run mosbot.py so that you don’t repeat exposures from the beginning of the night!
  1. If one wants to know where on the footprint the images being taken are, in real time:
         mayall-idl> cd ~/obsbot
         mayall-idl> eog radec.png &
    

which auto-update the display when the radec.png is overwritten.

Keep an eye on the CCD and dewar temps (should be around 173K and 90K respectively)

OK - you are off and running! Congratulations!!!

If you have problems …

See MayallZbandLegacy/NotesforObservers/Problems

Checking the Sky Brightness, Seeing and Transparency

The "copilot" python script and mosstat should already be running in a VNC. If they are not, you can launch xterms, login to the mayall-idl computer as mzls, and run these individually.

From an IDL prompt, use the MOSSTAT routine to analyze the latest image on disk:

   IDL> mosstat

There are keyword options that allow you to choose different exposure numbers or CCDs within that exposure. For example, to analyze chip 'im16' of the exposure number 12345, type:

   IDL> mosstat, 12345, ext='im16'

The full documentation can be seen with:

   IDL> doc_library,'mosstat'

To just have mosstat run continuously whenever each image appears, use

   IDL> mosstat_continuous

Please write useful human logs

Keep a log about weather conditions, which pass you observed, and telescope problems. Follow the example on the pages at MayallZbandLegacy/ObservingLogs .

Catastrophically bad frames (such as saturated frames, or where the telescope moved) are recorded and svn-checked-in to ~products/mosaic3/obstatus/bad_expid.txt .

[11] End of night

No on-sky observations are permitted with MOSAIC3 before 10 deg twilight or after 10 deg dawn. No twilight flats are allowed.

Once you exit the observing script (using "touch quit" or waiting for it to end), take a zero image. This ensures that the dark slide is put in place and the instrument is ready for shut down.

1. Shut down the software

  1. Ensure that the "Shutter" and "ready" are both in the "Dark" position on the MCCD gui
  2. Press the red "Stop MOSAIC" button on the MOSAIC Menu GUI. Wait for this to finish.
  3. Then press the yellow "Stop Cameras" button on the MOSAIC Menu GUI.

If the buttons do not work, then go to one of the NOCS xterm windows and type:

     nocs stop all (equivalent to pressing the "Stop MOSAIC" button)

Once nocs is shut down, type

     nocs stop ccp (equivalent to pressing the "Stop Cameras" button)

Once this is done, type

     nocs status all
     nocs fullstatus ccp

and make sure everything is shut down. Only when this is done, is it safe for the OAs to put on the lights in the dome.

Also, exit the window that was running mosbot, by typing CTRL-C Exit the xterm and close it.

2. Close down mayall-idl VNC session

We update the .bashrc for the next night, so it is best if you close the VNC windows and exit any xterms on mosaic3 and mayall-idl.

On a mayall-idl xterm, type:

~/bin/stopvnc

Exit from the window running mosbot by typing CTRL-C, and exit from that xterm.

2. Check in the updated tile file and updated bad exposure file

At the end of the night, you should check the updated tile file into the svn repository. Open a shell terminal on the mayall-2 desktop and log on to the mayall-idl computer:

  ssh mzls@mayall-idl

Once you have logged on to the mayall-idl computer:

   cd $MOS3_OBS/obstatus
   svn -m "observing update of tile file and bad exposure list” commit

If this does not work for some reason, try

  cd $MOS3_OBS/obstatus 
  svn -m "observing update of tile file and bad exposure list” commit -username <YOUR USERNAME>

and enter your wiki log in and password when prompted.

You can also check in the files individually as

  cd ~/products/mosaic3/obstatus
  svn -m "bad exp update" commit bad_expid.txt
  svn -m "obstatus update" commit mosaic-tiles_obstatus.fits

3. Create the Almanac and Almanac plot files

At the end of the night, create the Almanac files and check them into svn. Do this using a terminal window on the mayall-idl VNC (if it is still running), or by opening a terminal window on mayall-2, and logging on to mayall-idl by executing an "ssh mzls@mayall-idl" command :

- cd products/mosaic3/logs
- idl
- almanac
- plotalmanac,'Almanac_date.fits',ps='plot_Almanac_date'
- exit
- svn add *Almanac_date*
- svn commit *Almanac*

For example, for the night of March 26/27, 2015, this is done with:

cd products/mosaic3/logs
idl
   almanac
   plotalmanac,'Almanac_2015-03-26.fits',ps='plot_Almanac_2015-03-26'
   exit
svn add Almanac_2015-03-26.*
svn commit *Almanac_2015-03-26.*

There are actually two versions of this file, one that is an ASCII file (with .txt extension) and one that is a FITS file (with .fits extension).

4. Create the Coverage and Summary files for the Night

In a mayall-idl window on the VNC if it is still running), or by opening a terminal window on mayall-2, and logging on to mayall-idl by executing an "ssh mzls@mayall-idl" command :

- idl
- IDL> moscoverage_plot

This will produce a pdf plot of the current cumulative coverage. The plot is placed in the directory ~/products/obstatus/plots/ by the script. The file name has the form coverage.till_2016-MM-DD.pdf.

log into a mayall-idl window; cd to obsbot; then:

- python copilot.py --night
- mv night.png <date>.png (e.g., 2016-02-11.png)
- mv <dat>,png ~/products/mosaic3/logs/
- cd ~/products/mosaic3/logs/
- svn add <dat>.png
- svn -m "" commit

You can now mail these plots out to mayall-obs@desi.lbl.gov to let your collaborators know how the night went.

5. Summary of night log page

Fill out the summary log page entry for your night (i.e., add a row to the table seen at https://desi.lbl.gov/trac/wiki/MayallZbandLegacy/ObservingLogs ) You can read off the first and last image numbers from the copilot nightly summary plot, and for the total number of images taken, please subtract the bad images that you recorded.

7. Send out email to the collaboration

Please send out an e-mail to mayall-obs@desi.lbl.gov with a short report on the night. Please attach the 2016-03-17 nightly file.

End of night check list

  • Dark slides in
  • MOSAIC3 NOCS shut down
  • MOSAIC3 cameras shut down
  • Almanac file created
  • mosaic-tile_obstatus.fits, almanac, bad_expid.txt files commited to SVN
  • Log out of all mosaic3 windows
  • Log out of all mayall-idl windows
  • Mayall-idl VNC shut down
  • Observing logs and summary observing log table on wiki up-to-date and closed
  • Summary e-mail describing night sent out to mayall-obs@desi.lbl.gov

Thank the OA and go to bed


Pages linking to PublicPages/MayallZbandLegacy/NotesforObservers:

Attachments (2)

Download all attachments as: .zip