== Problems During Observing .. And What to Do ==

=== 0. No images being written ===

'''Symptom''': No new images are showing up on disk or in the obsbot display.  The control system and readout timer,
however, had no indication that data wasn't being written.

'''Frequency:''' This appeared as a new problem starting on 5/6 Feb 2016.

'''Fix:''' 
   - In a mosaic3 xterm window type
{{{
      touch ~observer/exec/mosbot/quit
}}}
   - Wait for the exposure to complete and the observing script to stop
   - Stop NOCS using the red "Stop MOSAIC" button on the MOSAIC 3 MENU gui
   - Once NOCS has shutdown, find the blue xterm window which came up when NOCS was launched and type
{{{
       nocs nuke pana 
       nocs nuke dhs
       nocs status all
}}}
   - The output from nocs status all should show that all processes are STOPPED and only 3 pvm processes RUNNING
   - Restart NOCS using the blue "Start MOSAIC" button on the MOSAIC 3 MENU gui
   - Once the system is back up, take a ZERO image to make sure everything is working
   - When the image appears, in the IRAF window type 
{{{
       mscstat <file name> 
}}}
   - If the results of the mscstat show the noise (in the Standard Deviation column) to be 4<rms<10, then ...
   - restart the observing script by following the instructions here

=== 1. Detector timeouts ===

'''Symptom''': Count-down timer in NMSL GUI turns red keeps counting down negative numbers (to -105 or so), and no new images are read.

'''Frequency''': Several times per night in the early Feb 2016 run.

'''Fix:'''
- Stop the observing script with cntrl-C
- Restart the controllers
{{{
       ditscmd nohs nohs_endobs
       nocs nmsl reset
}}}
- Take a zero image, make sure it reads out, and check the image by running "mscstat"
- Re-start the observing script


=== 2. Images show bad amp ===

'''Symptom:''' One amp has extremely large noise, looks awful the display of the image with "mcsdisplay" with a noise > 1000 ADU/pix from the "mscstat" command.

'''Frequency:''' Several times in Dec 2015 run, never in early Feb 2016 run.

'''Fix:'''
- Stop the observing script with cntrl-C
- Reset the NOCS controller by typing the following in a NOCS xterm window:
{{{
	nocs reset ccp
	nocs init ccp
}}}
- Take two zero images. The first image is always bad. The second should be OK.
   Check it using  "mscstat <filename>" once it has read out. All rms values should be between 4-7 ADU except for amp 6 with will be 8-10 ADU.
- Re-start the observing script

=== 3. 4MAPS primary mirror support  goes “OFF AIR” during observing ===

'''Symptom:''' The OA should notice when this happens, but you can also see if the mirror support is listed as "OFF AIR" on the 4MAPS GUI.
The images will also appear to have very large and strangely-shaped PSFs.

[[Image(4MAPS_gui.tiff, 400px)]]
[[Image(4MAPS_OFF_AIR.tiff, 400px)]]

'''Frequency:''' This happened several times per night in the Dec 2015 run, but with reduced frequency in the early Feb 2016 run.

'''Fix:'''
- Stop the observing script with cntrl-C
- Wait for it to finish and for “DONE” to appear on the NMSL GUI
- Type “ditscmd nohs nohs_endobs” in the xterm window in which the script was stopped. This should readout the last image and display it on the real-time display ximtool window
- Tell the OA they can move the telescope to zenith and reset 4MAPS
- Check the last image taken when 4MAPS failure occurred - the image quality is probably bad.  Add this as a bad exposure in the file ~products/mosaic3/logs/bad_expid.txt
- Once the OA has reset 4MAPS, re-start the observing script

=== 4. Shifted images ===

'''Symptom:''' Sub-blocks of an image have been shifted in the readout, and is obvious on the display as the overscan region being in a different place.
The readout system detects when this happens, adds a set of PIXCNT* cards in the image header, and a red line shows up on the obsbot/copilot strip chart.
Every now and then, an image appears which shows a shifted region in one or more CCDs. Please log this. You can identify images with shifts by running 

'''Frequency:''' A couple of times per night in the early Feb 2016 run.

'''Fix:'''
- Don't stop taking data.  This one exposure will be bad, but it should recover on its own
- Add this as a bad exposure in the file ~products/mosaic3/logs/bad_expid.txt

=== 5. Repeat images ===

'''Symptom:''' The image data is an exact duplicate of a previous image, usually the one from 6 images prior.
The headers are not duplicated, only the data, although the READTIME card will be the same to all decimal places
to the preceding image.

'''Frequency:''' Several times per night during the early Feb 2016 run, but should be less often now. 

'''Fix:'''
- In a NOCS xterm window, type: 
{{{
		nocs nuke pana
		nocs nuke dhs
}}}
- Start MOSAIC (from the startup GUI)
- Take a zero frame and make sure all the amps are OK and have reasonable noise levels using "mscstat"
- Re-start the observing script
- List as bad exposures in the file ~products/mosaic3/logs/bad_expid.txt


=== 6. Focus drifting ===

If you lose track of the focus and don’t know whether you are below or above the focus value, wait for the readout countdown to start 
and then type CTRL-c  in the window executing the script. Wait for the readout to complete and the “DONE” to appear on the NMSL window.
Then execute
{{{
	ditscmd nohs nohs_endobs
}}}
Then do a focus sequence. 
	- Create a focus script using the “Focus” button on the NGUI
          This pops up the following window: 
[[Image(NOCS_FOCUS.tiff, 600px)]]
	- use a good guess for the mid-point: 
                 focus in zd filter = -8400 + (1.4C - Ttruss)*110
          * an instrument focus step of -100 (or -70, if the seeing is good)
          * at least 7 focus iterations and 
          * a detector pixel shift of 30; 
          * Make sure you have chosen the zd filter
	- execute the focus sequence
- Edit the /data2/observer/mscfoc.cl script to change the file name of the focus image
- In the IRAF window, type "mscfoc.cl" and follow the instructions
- Once you have determined the best focus: 
		- set the focus value in the MCCD window and hit “Apply”
		- log the Truss temperature and focus values. 
- Create a new observing script
- Tell the OA that you have determined a new focus and changed the focus. 
- Restart observing

=== 7. Images of stars on the guider are bouncing around ===

'''Symptoms:''' Guider images are visibly bouncing around on the TV guider image, and the seeing is poor, with the image extension being 
primarily in the EW (i.e., up-down) direction: 
[[Image(windshake.tiff, 200pix)]]
'''Fix:''' If the wind is high, ask the OA to close the louvers that allow additional outside air to circulate into the dome.
If observing near zenith, the lower part of the dome slit can also be closed.  That lower part of the slit only starts
vignetting the telescope if observing to airmasses larger than 2.