Copilot

copilot.py is a script that watches for new images, measures them, and produces a plot to help observers understand what the conditions are doing.

Start it from the mzls@mayall-idl.kpno.noao.edu account,

cd ~/obsbot
python copilot.py &

It will monitor the $MOS3_DATA directory, and whenever a new image comes it, it will measure it, update a little database, and create a "strip plot" showing the conditions. The plot should be shown in a python plot window. The plot is also saved to the file recent.png; you could for example view it using

eog recent.png &

When a focus sequence is taken, it will run an analysis on the image and show another plot (also saved as focus.png).

The database is in ~/obsbot/obsdb/mosaic3.sqlite3 for MzLS.

If you want to make a plot showing your whole night, you can do

python copilot.py --night

and it will save the plot night.png.

Copilot Options (not usually needed)

• -h, --help show this help message and exit
• --ext=EXT Extension to read for computing observing conditions: default "N4" for DECam, "im4" for Mosaic3
• --extnum=EXTNUM Integer extension to read
• --rawdata=RAWDATA Directory to monitor for new images: default $MOS3_DATA if set, else "rawdata"
• --n-fwhm=N_FWHM Number of stars on which to measure FWHM
• --no-db Do not append results to database
• --no-focus Do not analyze focus frames
• --fits=FITS Write database to given FITS table
• --plot Plot recent data and quit
• --plot-filename=PLOT_FILENAME Save plot to given file, default recent.png
• --nightplot, --night Plot tonight's data and quit
• --qa-plots Create QA plots
• --keep-plots Do not remove PNG-format plots (normally merged into PDF)
• --mjdstart=MJDSTART MJD (UTC) at which to start plot
• --mjdend=MJDEND MJD (UTC) at which to end plot (default: now, which is 57440.940)
• --skip Skip images that already exist in the database
• --threads=THREADS Run multi-threaded when processing list of files on command-line
• --fix-db
• --tiles=TILES Tiles table, default obstatus/mosaic-tiles_obstatus.fits
• --no-show Do not show plot window, just save it.

Things to Watch For on the Copilot Plot

• Vertical red line labelled *MISSING IMAGE!*

This can be BAD -- it signifies that no images have been written out since the last purple point. When this occurs, check if you have been taking data. It means that while the telescope may be offsetting and camera opening and closing the shutter properly, no files are being written out! Need to stop (using "touch ~/exec/mosbot/quit") and restart MOSAIC3 software. Sigh.

• Vertical red line in top panel accompanied by "pixcnt" error message

This means that the frame taken has shifted pixels. You do not need to interrupt observing. Please log the exposure number of the bad frame both in the observing logs and in the bad_expid.txt file.

• Variations in transparency / seeing

May indicate that one has to change to using only pass2 and pass3 or just pass 3.

An example plot:

The top panel shows the seeing. Any problems with the data we know how to detect are marked with red vertical bands. These include:

• md5sum -- this image contains identical pixels as a previous image. This requires action! Please see the problem description https://desi.lbl.gov/trac/wiki/PublicPages/MayallZbandLegacy/NotesforObservers/Problems#a5.Repeatimages HERE.
• pixcnt -- the image readout is shifted. Please have a look at this exposure and note in the logs which CCD has the shift. The shifts are pretty easily recognized in the *mscexam* IRAF window -- blocks of one of the images will be shifted to the left or right. Do not stop observing. This problem is described https://desi.lbl.gov/trac/wiki/PublicPages/MayallZbandLegacy/NotesforObservers/Problems#a4.Shiftedimages here
• MISSING IMAGE -- it has been a while since the last image has been seen. Have images stopped being written? See problem description https://desi.lbl.gov/trac/wiki/PublicPages/MayallZbandLegacy/NotesforObservers/Problems#a0.Noimagesbeingwritten here

The second panel shows the Sky brightness relative to a nominal value. Larger values means the sky is darker than nominal. This panel also shows the pass number of previous exposures. This is the pass number of the tiles that were observed.

(Note, this does *not* come from an assessment of how good the conditions were, it comes from looking at what tile we chose to observe. Note also that the "pass2.json" file may contain tiles from pass 3, so even if you have set *forcepass2*, you may see pass 3 labels in the copilot plot.)

The "transparency" panel shows what fraction of the light from stars is getting through the atmosphere to us. Values above 90% usually indicate clear photometric conditions.

The "exposure time" panel is complicated. The filled magenta dots show the actual exposure time of the images we took. The black circles show the exposure time we should have taken, given the measured conditions in that image; it's post-facto what we should have done. We have lower and upper bounds on the exposure times, 80 and 250 seconds. If the "post-facto" target exposure time is below or above the bounds, it is shown with a triangle, and a circle will also be plotted at the lower or upper bound line. Triangles above the upper bound indicate images that might not be as deep as desired. Sometimes a line is visible in this plot (e.g. see 10hr - 11hr above 250 sec). This is the predicted exposure time to get 20,000 ADU from the sky background. Mosbot will try to limit the exposure times to avoid exposing for too long, but if conditions are changing rapidly, it may not realize soon enough that the sky has become too bright. The detectors can be damaged by too much light, so if you see this line, you should monitor the conditions very carefully! You may need to stop tonight.sh and take stock. This most easily happens with moon and clouds which make the background high.

The numbers on the exposure time panel are the "depth factor" of the images. A value of "1.50" indicates that we exposed for 1.5 times as long as was necessary to achieve our target depth. Note that the Mosbot does try to adapt its exposure times based on the depth achieved in other passes, so sometimes it may be shooting for depth values greater than 1.0. And when exposure times are clipped up to 80 seconds, we expect the depth factor to be greater than 1. Similarly, when exposure times are clipped down to 250 seconds, or to limit the sky background counts, then we expect the depth factor to be smaller than 1.

[depth factor has been removed from the plot. This description is useful when looking at plots created before about February 2016] The "depth factor" plot just shows our actual exposure time divided by the post-facto target exposure time. 1 indicates that we're doing exactly what we should be doing. Greater than 1 indicates we're over-exposing, and less than 1 indicates that we're under-exposing. The vertical labels show the exposure numbers.

The final panel is a strip chart showing the RA and DEC offsets from tile center determined post facto. When offsets are greater than a total of 10", the telescope zero point should be updated. Pointing and tracking are currently excellent at the Mayall. The actual telescope pointing updates should be made using output from mosstat. Thus the copilot plot may show non zero, but constant offsets. The goal is to get the mosstat and copilot zero points to agree.

Focus frames will be marked with a dot labelled "F", and Zero frames will be marked "Z".

Notes

Normally, copilot.py just waits for new images to appear, but you can also tell it to process a set of images (eg, if it crashed and you want it to run the backlog) by specifying the files on the command line. You can also tell it to skip files that already exist in the database via --skip, eg,

python copilot.py --skip $MOS3_DATA/mos3_*.fits

You can also do this to re-run on a focus frame:

python copilot.py $MOS3_DATA/mos3_86308.fits
eog focus.png

Or the slightly more pared-down script:

python mosaic_focus.py $MOS3_DATA/mos3_86308.fits --no-plots
eog focus.png

The database just records our CCD measurements, so it's not valuable. If things somehow go wrong and the database gets corrupted or you get git wedged or something, you can always delete it and start over via:

python obsdb/manage.py flush
python obsdb/manage.py migrate