= OBSERVING INSTRUCTIONS FOR MzLS (PROPID 2016A-0453) =

[[PageOutline]]

'''Still need to incorporate the following instructions:'''
  - '''Running the guiding loop -- although we typically do not guide on the Mayall for these 1-2 min exposures'''

These are instructions for observing for MzLS. 

Additional Notes:

 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/Copilot Copilot]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/MosBot Mosbot]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/Domeflats Dome flats]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/MOSAICGUI_Notes MOSAIC GUI Notes]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/Problems Problems]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/Safety Safety]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/TenDegTwilight 10 degree Twilight]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/TenDegTwilight2017 10 degree Twilight (2017 version)]
 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/TenDegTwilight2016 10 degree Twilight (2016 version)]

== VERY VERY IMPORTANT ==

=== 1. DO NOT OVEREXPOSE MOSAIC CCDs to LIGHT! ===

Prolonged exposure can damage and even destroy the devices. The following rules are in effect: 

 1. Operation on sky is restricted to between [wiki:MayallZbandLegacy/NotesforObservers/TenDegTwilight 10 deg twilights]. 
 1. Operation is restricted to when OAs or other NOAO technical/scientific staff are present
 1. The cameras and MOSAIC software should be shut down when observing / NOAO personnel are not present
 1. Sky values should be kept < 40,000 adu (<30,000 adu is preferable). 

=== 2. PLEASE TALK TO THE OAs ABOUT SAFETY AT THE TELESCOPE ===

 1. Please read the [wiki:MayallZbandLegacy/NotesforObservers/Safety Safety Rules] carefully and FOLLOW THEM!
 1. Do not leave the control room at night without telling the OA
 1. Do not go into the dome without the OA's permission
 1. Do not go exploring by yourself
 1. Carry a flashlight and radio at night

== Read the Manual for MOSAIC operations ==

 * To see the Mosaic1.1 manual, go here: http://www.noao.edu/kpno/mosaic/manual 
 * To see the Mosaic3 manual go here: http://www.noao.edu/kpno/mosaic/manual/Mosaic3Manual.pdf

== OBSERVING TROUBLE SHOOTING == 

 * [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/Problems Observing trouble-shooting page]

== Some Preliminaries ==

Who is your OA for tonight? See [http://www.noao.edu/perl/Calcium?Op=ShowIt&CookieParams=1&CalendarName=oa&Amount=Month&NavType=Absolute&Type=Block the OA Schedule]

Observing can only be done from the Mayall control room or from within the KPNO/NOAO network using a VPN. 

The telescope and low-level instrument control runs on `observer@mayall-7.kpno.noao.edu`.

Log on to the mayall-7 computer:

  * Username : observer
  * Password : <Ask someone on our team or the staff>

To run MOSAIC3, use the MOSAIC3 Menu GUI. Or you can also log into the mosaic3 computer from mayall-7:
{{{
ssh -XY observer mosaic3
}}}
and use command-line nocs commands.

'''PLEASE BE CAREFUL ON THIS COMPUTER - IT RUNS THE CONTROLLERS AND THE INSTRUMENT!!!'''

Our observing scripts and data reduction runs on `mzls@mayall-idl.kpno.noao.edu`:

 * Username : mzls
 * Password : <Ask someone on our team or previous observers>

These environment variables define the locations of data and code:

 * `HOME=/home/mzls` -- This should be done for you by the operating system.
 * `MOS3_DATA=` -- This is where our raw data files are written. Should be something like `/mosaic3/data2/observer/20160202`. This is a network mounted disk on the machine mosaic3.
 * `$MOS3_OBS=$HOME/products/mosaic3` -- This contains the observing product with code and versioned log files.
 * `$PS1CAT_DIR=/data1/mzls/ps1/chunks-qz-star-v2` -- Pan-STARRS1 catalogs used by the IDL MOSSTAT routine for computing astrometric offsets and photometry

These path names are set in the `~/.bashrc` file, or can be set for example with `export MOS3_DATA=/mosaic3/data2/observer`.
The `$MOS3_DATA` directory must be updated to point to the current night's data.

Other directories in the home directory are:

 astrometry/, legacypipe/, obsbot/, tractor/, wcslib/:: Python code for running copilot & mosbot
 data/:: Link to the top-level data directory, where each night is stored in subdirectories like "20151213"
 exec/:: Cross-mount to mosaic.kpno.noao.edu:/home/observer/exec
 products/:: Code checked out from the SDSS and the DESI svn repositories
 
Documentation for the IDL scripts can be printed from the IDL prompt with the DOC_LIBRARY command, for example:
{{{
idl
IDL> doc_library,'mosstat'
}}}

== Date convention ==

All dates in log files are set to the local date of the '''beginning of the night'''.  For example,
any data taken during the night of December 13/14, 2015 will be written as 2015-12-13.
This is consistent with how the NOAO Science Archive timestamps and saves the raw data files.

== Are we there yet? == 

The tiles we are observing with MOSAIC3 in z-band are tracked in the file ~/products/mosaic3/obstatus/mosaic-tiles_obstatus.fits with the following cuts:
{{{
  IN_DESI = 1
  DEC >= 30
  88 < RA < 301
  PASS <= 3
}}}
That’s 44,422 tiles.
This will likely be further limited further to the 41,188 tiles at DEC >= 34.
About 5% of the tiles have been removed from the list where there's a star
brighter than V=6 within 0.35 deg of the tile centers.

The tiles are broken into three passes. Each pass, pass1, pass2, and pass3, covers the basic footprint and each is offset optimally from the others. The tile are thus now fixed or defined on the sky for the duration of the survey. By definition:

**pass1** is the high quality, photometric coverage. To the greatest extent possible, we want to ensure pass one tiles are the best. As such, pass1 should be executed on photometric nights with good seeing (< 1.3"). The observer should not start pass1 if it is not a true photometric night or if 3-4 hour contiguous blocks are not stable and clear. The robot observing program will choose pass1 for seeing < 1.25", transparency > 90% and sky brightness not worse than 0.25 mag brighter than the fiducial. But the observer should really be setting forcepass1 based on whether its a very stable photometric night from beginning to end. Forcing the pass is described here: [wiki:MayallZbandLegacy/NotesforObservers/MosBot#UseTheForce Use The Force]

**pass2** is the next best pass. Seeing should be < 1.3" or the weather is photometric (but seeing worse than 1.3"). Pass2 and 3 may alternate throughout a given night and the mosbot observing robot will do this automatically based on seeing, transparency, and sky brightness. Pass2 could also be done if no pass1 tiles are available. The robot will stick with pass2 when conditions are a little worse than this. Specifically seeing < 2" and trans > 70%. Observers can force pass3 if they think the conditions are poor, even though the robot might say pass2. This is especially encouraged if conditions are variable on short timescales and the robot is moving frequently between pass2 and pass3.

**pass3** is the filler or worst pass executed when conditions are bad. Seeing > 1.3'' and its not photometric (trans < 90%). Pass3 may be done in good conditions when no pass1 or pass2 tiles are available. 

Or, in more concise language:
{{{
Pass 1 requires transparency > 0.9 AND seeing < 1.25 AND brightness < 0.25 brighter than nominal
Pass 2 requires (transparency > 0.9 AND seeing < 2.0) OR (transparency > 0.7 AND seeing < 1.25)
}}}


== READ THIS IN PREP FOR OBSERVING ==

 * [wiki:MayallZbandLegacy/NotesforObservers/MosBot  Mosbot  notes]
 * [wiki:MayallZbandLegacy/NotesforObservers/Copilot Copilot notes]

== REMOTE OBSERVING ==

 * The fundamental rule of remote observing is: ''If there is a '''local''' observer, do not alter anything without notifying the local observer first! Even moving the mouse at the wrong time could interfere with an essential task that the local observer is performing.''
 * [wiki:MayallZbandLegacy/NotesforObservers/RemoteOps Remote Observing notes]

== AFTERNOON DOMEFLATS ==

 * [wiki:MayallZbandLegacy/NotesforObservers/Domeflats]

== EXAMPLE RUN THROUGH FOR A GIVEN NIGHT ==

=== 1 Update everything ===

Log into the mzls account on the mayall-idl machine.

Update the code, log files and most importantly the tile file: (if people have been working on the code and tile file during the day, this step may not be necessary)
{{{
ssh mzls@mayall-idl
cd ~/products/mosaic3
svn up
cd ~/products/observing
svn up
cd ~/obsbot
git pull
}}}

=== 2 Set the paths ===

The data path must be changed to point to the current night's data.
Edit the entry for MOS3_DATA ~/.bashrc file, for example if the start of the night is 13 Dec 2015 change this to
{{{
% emacs ~/.bashrc
export MOS3_DATA=/mosaic3/data2/observer/20151213
% source ~/.bashrc
}}}
Don't forget to '''''source'' the file''' if you plan to continue to use this (or any open) terminal.

=== 3 Create nightly plan ===

There are 3 nightly plan files, one with a list of tiles to observe for each of the 3 passes.
These files are in JSON format, and specify the pass 1, pass 2 or pass 3 tiles to be observed
at each timestamp through the night.  The selection of pass number while observing will depend upon the weather conditions as
described at [wiki:MayallZbandLegacy/ObservingStrategy], and will either be selected
automatically or can be forced by the user.


The IDL
[https://desi.lbl.gov/trac/browser/decam/code/observing/trunk/pro/mstrategy.pro MSTRATEGY code]
is used to create the above plan files,
as described at [wiki:MayallZbandLegacy/ObservingStrategy#NightlyStrategy]

Arjun or David may have already provided a set of plan files for the night and posted a note
to the mayall-obs e-mail list saying so.  These files would be checked into the "mosaic3" product here:
{{{
~/products/mosaic3/json/YYYY-MM-DD-p1.json
~/products/mosaic3/json/YYYY-MM-DD-p2.json
~/products/mosaic3/json/YYYY-MM-DD-p3.json
}}}
and copied into the observing directory where they are read by [wiki:MayallZbandLegacy/NotesforObservers/MosBot  mosbot] here:
{{{
~/obsbot/pass1.json
~/obsbot/pass2.json
~/obsbot/pass3.json
}}}

If the plan files have not been generated for the night, please do so as follows:
{{{
idl
IDL> mstrategy, copydir='/home/mzls/obsbot'
IDL> exit
}}}
Then check in the plan files to the "mosaic3" product as follows:
{{{
cd ~/products/mosaic3/json
svn add YYYY-MM-DD-p*.json
svn commit -m "plan files for tonight"
}}}
There are plot files in that directory too.  We don't check those in, but it's informative
to post the planned coverage map ~/products/mosaic3/json/<DATE>.pdf to mayall-obs@desi.lbl.gov .

=== 3b If you are Observing in the 2017B semester: === 
The json files that are generated by the strategy code (mstrategy) will require a very large slew near the end of the night (~09:40ut = 02:40am) which will take you from the west end of the DESI North Galactic Cap footprint to its eastern end. The problem with this is that the system times out during the long slew, and this causes it to crash.  **Note: This has also happened on occasion in 2018.**

The procedure until further notice is to break each new json file that is created in the ~mzls/obsbot/ directory into two parts at the point of this large slew. (This will not be necessary if MzLS is scheduled for a first-half night, since you will be stopping observing at ~00:30am, but it will be an issue for all full nights.) 

This means that the observing script with which you start the night will terminate at around 2:40am. At this point, ask the telescope operator to slew to the first position on the eastern side of the NGC. Once the telescope has reached there, then restart mosbot by executing 
	python mosbot.py pass1b.json pass2b.json pass3b.json —adjust —pass=3 —exptime=100
(where all the — are really two dashes). This will allow this system to continue without timing out during the slew. 

Deprecated instructions using a modified version of "nightlystrategy" is at [wiki:MayallZbandLegacy/NightlyStrategyOld]

=== 4 Start up mosaic control software ===

On mayall-7, double click on the MOSAIC3 icon, which brings up the MOSAIC3 Menu on the left edge of the screen. 

 1. **Start the camera control program** by pressing the yellow "Start Cameras" button. Wait for this to finish, then dismiss the screen by typing any key as instructed
 1. **Start the MOSAIC3 NOCS** software by pressing the blue "Start MOSAIC" button. This launches a blue xterm. Move it out of the way and watch all the windows come up
 1. Rearrange the desktop as needed. 
    If, for some reason, the buttons do not work, you can start up the software on a command line as follows.
    On mayall-7 open a terminal window and: 
    {{{
ssh -XY observer@mosaic3
nocs start ccp (this is equivalent to the "Start Cameras" button on the MOSAIC3 Menu)
nocs start all (this is equivalent to the "Start MOSAIC" button on the MOSAIC3 Menu)
    }}}
    Once nocs is up and running, rearrange windows as desired, and check the status of the   system by typing the following in a nocs terminal window:
    {{{
nocs status all
nocs fullstatus ccp
    }}}
    If you want to know what these commands actually do, see here: [wiki:MayallZbandLegacy/NotesforObservers/MOSAICGUI_Notes]
 1. '''If you are on mayall-7''' in the new U-floor control room, please wait for the system to come up completely, and then close the DHS VNC viewer window using the small red button on the top bar of the window. Then relaunch a new VNC viewer by first clicking on the blue VNC icon in the dock, and then double clicking on the "mosaic3:1" icon in the window that comes up. This should bring up a new DHS VNC window that will respond more quickly to cursor commands.
 1. Launch CCD temperature monitor from "MOSAIC Temps" icon (CCD and Dewar temps should be around 173C and 90C respectively)
 1. Launch TCS acorn monitor from "VDU" icon
 1. 4MAPS monitor from "4MAPS" icon
 1. Launch Truss temperature monitor from "Truss C" icon

All these icons are on the right side of the left hand screen of the mayall-7 computer display. Many or all of these may be running already. They can stay up. Rearrange the busy desktop as needed ...

=== 5 Set the PROP-ID and Project Info ===

On the NGUI window, press the "Set Project" button and fill out the relevant information. For example, 
{{{
Principal Investigator: Arjun Dey
PIs Email Address: dey@noao.edu
Actual Observers: Tristram Shandy, Bertram Wooster
Observing Assistant: Karen Butler
OA's Email Address: 4meter@noao.edu
Proposal Identifier: 2016A-0453
Telescope System: KPNO Mayall 4m
Science Instrument: Mosaic 3
}}}
''Please ensure the Proposal Identifier number is correct!''

Then, in one of the NOCS xterm windows, type: "nocs set project"

=== 6 Take a test Zero image to ensure system is working ===

Check that all is well by taking a test zero exposure. 

To do this, in a nocs window (an xterm connected to observer@mosaic3) do:
{{{
cd ~/exec/MzLS
./ZERO1_MzLS.sh
}}}

After the image is done, on the IRAF window, check the image statistics by cd-ing to the correct directory (e.g., /data/observer/20180105 for data from Jan. 5 2018) and typing "mscstat <filename>"
All rms values should be about 4-6 adu/pix; the exceptions are amplifier [6] or [14] which can have an rms~8-10 adu/pix. 
If any one amplifier shows very high noise, then execute the following commands in a nocs window:
{{{
nocs reset ccp
nocs init ccp
}}}
Then take two more zeros; the first one will be junk, but the second one should be OK.

=== 7 Take dome flats and zeros ===

During the afternoon (after 4pm), take dome flats with the telescope pointed at the white spot.

Instructions for Domeflats are here: [wiki:MayallZbandLegacy/NotesforObservers/Domeflats]

Go eat dinner.

=== 8 Just before observing ===

It is best to start things up after sunset and to be ready to start observing before the 10 deg twilight mark, see [wiki:PublicPages/MayallZbandLegacy/NotesforObservers/TenDegTwilight 10 deg twilight times for KPNO page].

Before 10 deg twilight, you can do all the steps in 8 (''i.e.'', this one).

 1. Start the mayall-idl:1 VNC session.
    * In a terminal window from mayall-7, ssh to mzls@mayall-idl and log in using the mzls password
    * type: `~/bin/vnc`
    * Now double click the VNC gui in the mayall-7 dock (''i.e.'', the thing with all the icons at the bottom of the bottom left terminal).  This launches a dialog box asking if you want to connect to `mayall-idl.kpno.noao.edu:1` . Say "connect" and type in the mzls password. The VNC starts up.
    The VNC session should automatically start MUPTILES, MOSSTAT and COPILOT.  Rearrange as desired.
    If it doesn't, open terminals in the VNC session and manually run the scripts for monitoring the observations and updating the tile file:
    a. The FITS file listing which tiles have been completed should be updated throughout the night.
       This file is '''$MOS3_OBS/obstatus/mosaic-tiles_obstatus.fits'''.  The following IDL command will monitor exposures as they are taken throughout the night, automatically updating this file.
       From a terminal on the mayall-idl VNC, in the mzls account, start this running:
       {{{
> idl
IDL>   muptiles
       }}}
       At the end of the night, you should check the updated tile file into the svn repository.
    a. In a second terminal on the VNC start the automated script for monitoring the data quality
       {{{
> idl
IDL> mosstat_continuous
       }}}
       This will run mosstat on each frame as it shows up and display the results on the screen. 
    a. In a third terminal on the VNC start running [wiki:MayallZbandLegacy/NotesforObservers/Copilot copilot]: 
       {{{
cd ~/obsbot
python copilot.py
       }}}
 1. Take a zero image to ensure everything is working. Once the image has been written, use the IRAF window and run mscstat <filename> to make sure the system came up OK. See step [6] above.
 1. Set the focus to some approximate value based on the Truss temperature and the formula: 
    {{{    
Focus(zd) = -8400+(1.4-Ttruss)x110
    }}} 
    where `Ttruss` is in deg C. If the temperature is >18C, the following might work better:
    {{{
Focus(zd) = -10940+(22.9-Ttruss)x110
    }}}
    For convenience, you can create these two Python functions in a python terminal:
    {{{
>>>focus = lambda x: -8400 + (1.4 - x)*110
>>>focuswarm = lambda x: -10940 + (22.9 - x)*110
>>>focus(x); where x is the current Truss temp in deg C 
    }}}
 1. Ask the telescope operator to point to an MzLS coordinate to do the pointing check. Use the first tile in the json file in pass 3 (for normal case, or json of forced pass if starting up non standard). The coordinates of the first tile can be found by:
    {{{
mayall-idl > less ~/obsbot/pass3.json
    }}}
 1. Wait patiently for 10 deg twilight. 

=== 9 Get Ready to Observe - start of night ===

'''No on-sky observations are permitted before [wiki:MayallZbandLegacy/NotesforObservers/TenDegTwilight 10 deg twilight].'''

 1. At start of night, check telescope pointing (see item 8.4 above; you should already be there)! For B-semester observing, use the pointing coordinate 280.0, +50.0) by doing the following: 
    a. create an OBJECT script with NGUI for 5 sec with the zd filter  (do not create a TEST script, because MOSSTAT will ignore it)
    a. Execute it, wait for it to readout completely, and make sure that it appears in your data directory; if it does not, use the "Update Status" button on the DHS GUI
    a. Once the image is processed by copilot, the bottom panel on the copilot plot will print the pointing offsets. Give these offsets to the OA with '''opposite''' sign.  That is, if copilot shows numbers (-15.3,+18.3), or if mosstat reads "RA,Dec offsets = -15.34, 18.32", then you need to provide the OA with the pointing offsets of +15,-18 to zero the telescope coordinates.
    a. If this process does not work for some reason, then you can also zero the telescope coordinates by using a bright star placed on the telescope boresight (defined as the center of the mosaic3 focal plane; ''i.e.'' in the chip gaps). As before, when taking the image of the bright star, make sure to create an OBJECT script with NGUI (not a TEST, because MOSSTAT will ignore it), zd filter, 1 - 5 sec exposure.  After centering the bright star, return to the first target field and use the above procedure to determine the pointing offsets - ''' do not use the copilot or mosstat results based on the bright star image! '''  Make sure you move back to the first MzLS tile position after you complete this test.
    a. If mosstat or copilot fails, it could be because (a) the telescope is mis-pointed, (b) the telescope is out of focus (set the focus approx using the formula above and try again).
    a. You can take another OBJECT frame if you want to check to make sure the offset went in the right direction.
 1. Focus the Telescope
    a. Create a focus script using the NGUI: exposure time 5 or 10 seconds, zd filter, -75 micron focus steps, click Midpoint to *on*, 9 exposures. NOTE: If observing in the 2017B semester, please make sure you are focusing at the sky location 270,+60. 
    a. run the focus script from the /home/observer/exec directory (./FOCUS.sh)
    a. copilot will automatically analyze and report the best focus in the mayall-idl window.
    a. or alternately, analyze the focus image using the IRAF script mscstarfocus 
      - edit the /data/observer/mscfoc.cl script to correct the name of the image that needs to be analyzed 
      - run 'cl < /data/observer/mscfoc.cl' in iraf window in data directory.
      - mark about 10 stars around the image; to get a quick idea if you have covered the right focus range mark "g" on the top star in a sequence which will pop up a graph that you will need to type "q" to get out of; mark the top star in each remaining sequence using "m"; "q" to quit; "d" to delete bad points; "q" to quit, then will get best focus value. 
    a. log the Truss temperature 
    a. set the telescope focus in the Configuration Monitor: Enter the value in Pedestal focus, '''hit return''', and then hit Apply.  If you forget the return, it will do nothing. 
    a. focus the guiders; this way you can use the guider images to monitor focus drifts
    a. [wiki:MayallZbandLegacy/NotesforObservers/FocusInfo Example of a focus sequence]

=== 10 Observe - all night long! ===

==== Routine Observations ====

 1. From the mzls@mayall-idl window, generate the top-level observing script (tonight.sh) assuming that we have three plan files named pass1.json, etc.
    {{{
cd ~/obsbot
python mosbot.py pass1.json pass2.json pass3.json --adjust --pass=3 --exptime=100
    }}}
    This will start the [wiki:MayallZbandLegacy/NotesforObservers/MosBot Mosbot] script, which will watch the `$MOS3_DATA` directory for new images, analyze them, and update FUTURE exposure scripts, choosing the pass number and setting the exposure time. '''The exposure script behavior can be modified by creating or removing various files in the `~/exec/mosbot/` directory on mosaic3 (nocs xterm).'''
    * If the conditions look marginal, and you only want to run with pass3, say, create an empty `forcepass3` file in the `~/exec/mosbot/` directory:
      {{{
rm ~/exec/mosbot/forcepass?
touch ~/exec/mosbot/forcepass3
      }}}
    * If you want to allow only pass 2, then use: 
      {{{
rm ~/exec/mosbot/forcepass?
touch ~/exec/mosbot/forcepass2
      }}}
    * If you want to allow only pass 1, then use: 
      {{{
rm ~/exec/mosbot/forcepass?
touch ~/exec/mosbot/forcepass1
      }}}
    * If you want to allow only pass 2 and pass 3, then use: 
      {{{
rm ~/exec/mosbot/forcepass?
touch ~/exec/mosbot/nopass1
      }}}
      Mosbot checks for these files in the order: `forcepass 1,2,3` then `nopass1`.
    * Force files are removed by `mosbot.py` on startup, so you will need to re-create those files if you restart the Mosbot.
    * For more options with mosbot, see [wiki:MayallZbandLegacy/NotesforObservers/MosBot]
    * Note that if you need to stop the script at some point during the night (see below), you will need to CTRL-C to stop mosbot.py and **restart** it when you are ready to start up again.  If you forget this, you will be observing tiles that you already observed earlier in the night.
 1. From the observer@mosaic3 xterm window, start taking exposures using the top-level observing script (tonight.sh):
    {{{
cd ~/exec/mosbot
./tonight.sh
    }}}
 1. [wiki:MayallZbandLegacy/NotesforObservers/Copilot Copilot] should already be running. If not, start running Copilot in the mayall-idl VNC window. Copilot keeps a beautiful running plot of observing conditions.  From any mzls@mayall-idl window:
    {{{
cd ~/obsbot
python copilot.py
    }}}
    Whenever a new image is detected in the data directory (as defined by `$MOS3_DATA`), a new image is generated as `~/obsbot/recent.png`.
 1. Monitor focus by checking the image quality on each frame or keeping an eye on the mosstat PSF display. Keep track of the truss temperature variation and use the information to modify  the focus as needed. Note that the focus may not respond quickly to changes in temperature, so monitor the images carefully before adjusting focus.  The Mayall has astigmatism, so one can sometimes tell from the shape of the images which way to move the focus.  Here is a focus correction cheat sheet: 
[[Image(Focus_Adjustments.png, 512px)]]

To stop and do a focus sequence:
    - Create a file to tell tonight.sh to quit. On mosaic3:
      {{{
touch ~/exec/mosbot/quit
      }}}
    - Wait for the current exposure to complete (at which point the above file is automatically removed). 
    - CTRL-C the mosbot.py session on mayall-idl.
    - Run a focus sequence (see link below for instructions).
      - [wiki:MayallZbandLegacy/NotesforObservers/FocusInfo Example of a focus sequence]
    - Re-start the observing as described above.  **IMPORTANT** make sure to re-run `mosbot.py` so that you don't repeat exposures from the beginning of the night! And **ALWAYS** check the creation time of the tonight.sh script before you start running it.
    - Make a note in the logs of the exposure number of the image frame after or before which you made the focus change. 
 1. If one wants to know where on the footprint the images being taken are, in real time:
    {{{
mayall-idl> cd ~/obsbot
mayall-idl> eog radec.png &
    }}}
    which auto-update the display when the radec.png is overwritten.
 1. Keep an eye on the CCD and dewar temps (should be around 173K and 90K respectively)
 1. OK - you are off and running! Congratulations!!!

'''''
If you need to stop an exposure do this:
'''''

Using the “Abort” button does not work. The best thing to do if you need to stop an exposure quickly is to do the following:
1. close the shutter by pressing the “Dark” button on the MCCD GUI
2. cd ~/exec/mosbot and type 'touch quit' 
3. cntrl-C out of the python mosbot.py run
This will count down the exposure to completion, but at least it won’t crash nocs or compromise the system.

==== If you have problems ... ====

See **[wiki:MayallZbandLegacy/NotesforObservers/Problems]

==== Checking the Sky Brightness, Seeing and Transparency  ====

The Copilot and mosstat programs should already be running in a VNC. 
If they are not, you can launch xterms, login to the mayall-idl computer as mzls, 
and run these individually.  

From an IDL prompt, use the MOSSTAT routine to analyze the latest image on disk:
{{{
IDL> mosstat
}}}
There are keyword options that allow you to choose different exposure numbers or CCDs
within that exposure.  For example, to analyze chip 'im16' of the exposure number 12345, type:
{{{
IDL> mosstat, 12345, ext='im16'
}}}
The full documentation can be seen with:
{{{
IDL> doc_library,'mosstat'
}}}
To just have mosstat run continuously whenever each image appears, use
{{{
IDL> mosstat_continuous
}}}


==== Please write useful human logs ====

Keep a log about weather conditions, which pass you observed, and telescope problems.  Follow the example on the pages at [wiki:MayallZbandLegacy/ObservingLogs] . Please make a note of when you make any changes to the focus (note the time and the image exposure number before/after the focus change), any changes to weather conditions (increase in wind speed, change in direction, changes in humidity, sky brightness, cloud cover, etc.), and any pointing corrections (please note the time and magnitude, and if possible the frame number). Please stay aware of the observing conditions. Go out an look at the sky yourself every few hours - do not rely solely on the computer monitors and telemetry. Don't worry - you will not be eaten; just carry a flashlight. Sing loudly, if that helps :)

Please record any catastrophically bad frames (such as saturated frames, or where the telescope moved) in the bad_expid.txt file. This file is svn-checked-in at the end of the night.
[https://desi.lbl.gov/trac/browser/decam/code/mosaic3/trunk/obstatus/bad_expid.txt ~products/mosaic3/obstatus/bad_expid.txt] .

=== 11 End of night ===

No on-sky observations are permitted with MOSAIC3 before [wiki:MayallZbandLegacy/NotesforObservers/TenDegTwilight 10 deg twilight or after 10 deg dawn]. No twilight flats are allowed. 

Once you exit the observing script (using "touch quit" or waiting for it to end), take a zero image. This ensures that the 
dark slide is put in place and the instrument is ready for shut down.

==== 1 Shut down the software ====

1. Ensure that the "Shutter" and "ready" are both in the "Dark" position on the MCCD gui
2. Press the red "Stop MOSAIC" button on the MOSAIC Menu GUI. Wait for this to finish. 
3. Then press the yellow "Stop Cameras" button on the MOSAIC Menu GUI.

If the buttons do not work, then go to one of the NOCS xterm windows and type:
{{{
     nocs stop all (equivalent to pressing the "Stop MOSAIC" button)
}}}
Once nocs is shut down, type
{{{
     nocs stop ccp (equivalent to pressing the "Stop Cameras" button)
}}}
Once this is done, type 
{{{
     nocs status all
     nocs fullstatus ccp
}}}
and make sure everything is shut down. Only when this is done, is it safe for the OAs to put on the lights in the dome. 

Also, exit the window that was running mosbot, by typing CTRL-C
Exit the xterm and close it.

==== 2 Close down mayall-idl VNC session ====

We update the .bashrc for the next night, so it is best if you close the VNC windows and exit any xterms
on mosaic3 and mayall-idl. 

'''From here to item 5, one may now execute the bash shell script bin/end_of_night to do all the book keeping and plot making. Jump to item 6 if you use this option. '''

On an '''xterm in the mayall-idl VNC window''' from any directory:

{{{
end_of_night YYYY-MM-DD &> YYYY-MM-DD-eon.log
}}}

Where DD is the day at the beginning of the night. Using the error redirect to a log file allows one to check for errors. One can watch as the script is running by:
{{{
tail -f YYYY-MM-DD-eon.log
}}}
in a separate xterm.

To continue instead in manual mode, keep going:

On a mayall-idl xterm, type: 
{{{
~/bin/stopvnc
}}}

Exit from the window running mosbot by typing CTRL-C, and exit from that xterm.

==== 3 Check in the updated tile file and updated bad exposure file ====

At the end of the night, you should check the updated tile file into the svn repository. Open a shell terminal 
on the mayall-7 desktop and log on to the mayall-idl computer: 
{{{
  ssh mzls@mayall-idl
}}}

Once you have logged on to the mayall-idl computer: 
{{{
   cd $MOS3_OBS/obstatus
   svn -m "observing update of tile file and bad exposure list” commit
}}}
If this does not work for some reason, try 
{{{
  cd $MOS3_OBS/obstatus 
  svn -m "observing update of tile file and bad exposure list” commit -username <YOUR USERNAME>
}}}
and enter your wiki log in and password when prompted.

You can also check in the files individually as

{{{
  cd ~/products/mosaic3/obstatus
  svn -m "bad exp update" commit bad_expid.txt
  svn -m "obstatus update" commit mosaic-tiles_obstatus.fits
}}}


==== 4 Create the Almanac and Almanac plot files  ====

At the end of the night, create the Almanac files and check them into svn. Do this using 
a terminal window on the mayall-idl VNC (if it is still running), or by opening a terminal window
on mayall-7, and logging on to mayall-idl by executing an "ssh mzls@mayall-idl" command : 
{{{ 
- cd products/mosaic3/logs
- idl
- almanac
- plotalmanac,'Almanac_date.fits',ps='plot_Almanac_date'
- psfmovie
- exit
- svn add *Almanac_date*
- svn commit *Almanac*
}}}

For example, for the night of March 26/27, 2015, this is done with:
{{{
cd products/mosaic3/logs
idl
   almanac
   plotalmanac,'Almanac_2015-03-26.fits',ps='plot_Almanac_2015-03-26'
   exit
svn add *Almanac_2015-03-26.*
svn commit *Almanac_2015-03-26.*
}}}

There are actually two versions of this file, one that is an ASCII file (with .txt extension)
and one that is a FITS file (with .fits extension).

==== 5 Create the Coverage and Summary files for the Night  ====

In a mayall-idl window on the VNC if it is still running), or by opening a terminal window
on mayall-7, and logging on to mayall-idl by executing an "ssh mzls@mayall-idl" command : 
{{{
- idl
- IDL> moscoverage_plot
}}}
This will produce a pdf plot of the current cumulative coverage. The plot is placed in the directory ~/products/mosaic3/obstatus/plots/ by the script. The file name has the form coverage.till_2016-MM-DD.pdf.

log into a mayall-idl window; cd to obsbot; then:
{{{
- python copilot.py --night
- mv night.png <date>.png (e.g., 2016-02-11.png)
- mv <date>.png ~/products/mosaic3/logs/
- cd ~/products/mosaic3/logs/
- svn add <date>.png
- svn -m "" commit
}}} 

You can now mail these plots out to mayall-obs@desi.lbl.gov (YYYY-MM-DD.png will suffice as mayall-desi@ does not like large attachments) to let your collaborators know how the night went.  

==== 6 Summary of night log page ====

Fill out the summary log page entry for your night (i.e., add a row to the table seen at 
https://desi.lbl.gov/trac/wiki/MayallZbandLegacy/ObservingLogs )
You can read off the first and last image numbers from the ~/products/mosaic3/logs/Almanac-2016-02-11.txt, and for the 
total number of images taken, please subtract the bad images that you recorded. 

==== 7 Send out email to the collaboration ====

Please send out an e-mail to mayall-obs@desi.lbl.gov with a short report on the night. Please attach the YYYY-MM-DD.png nightly file. 

==== Summary: End of night check list ====

* Dark slides in
* MOSAIC3 NOCS shut down
* MOSAIC3 cameras shut down
* Almanac file created
* mosaic-tile_obstatus.fits, almanac, bad_expid.txt files commited to SVN
* Log out of all mosaic3 windows
* Log out of all mayall-idl windows
* Mayall-idl VNC shut down
* Observing logs and summary observing log table on wiki up-to-date and closed
* Summary e-mail describing night sent out to mayall-obs@desi.lbl.gov
* Chat with the OA about the mirror cooling set temperature for the next night

=== 12 Thank the OA and go to bed ===

[[BackLinks]]