== Problems During Observing .. And What to Do ==


=== 1. Count-down timer counting negative and no new images read ===

'''Symptom''': Count-down timer keeps counting down negative numbers, and no new images are read.

'''Frequency''': Several times per night in the early Feb 2016 run.

'''Fix:''' Arjun types magic things.

=== 2. Images show bad amp ===

'''Symptom:''' One amp has extremely large noise, looks awful the display of the image with "mcsdisplay" with a noise > 1000 ADU/pix from the "mscstat" command.

'''Frequency:''' Several times in Dec 2015 run, never in early Feb 2016 run.

'''Fix:''' Reset the NOCS controller by typing the following in a NOCS xterm window:
{{{
	nocs reset ccp
	nocs init ccp
}}}
Then take two zero images. The first image is always bad. The second should be OK.
Check it using  "mscstat <filename>" once it has read out. All rms values should be between 4-7 ADU except for amp 6 with will be 8-10 ADU.

=== 3. 4MAPS primary mirror support  goes “OFF AIR” during observing ===

'''Symptom:''' The OA should notice when this happens, but you can also see if the mirror support is listed as "OFF AIR" on the 4MAPS GUI.
The images will also appear to have very large and strangely-shaped PSFs.

[[Image(4MAPS_gui.tiff, 400px)]]

'''Frequency:''' This happened several times per night in the Dec 2015 run, but never in the early Feb 2016 run.

'''Fix:'''
- Type Control-C to exit out of the observing script
- Wait for it to finish and for “DONE” to appear on the NMSL GUI
- Type “ditscmd nohs nohs_endobs” in the xterm window in which the script was stopped. This should readout the last image and display it on the real-time display ximtool window
- Tell the OA they can move the telescope to zenith and reset 4MAPS
- Check the last image taken when 4MAPS failure occurred - the image quality is probably bad.  Add this as a bad exposure in the file ~products/mosaic3/logs/bad_expid.txt
- Create a new observing script
- Once the OA has reset 4MAPS, start the new script

=== 4. Shifted images ===

'''Symptom:''' Sub-blocks of an image have been shifted in the readout, and is obvious on the display as the overscan region being in a different place.
The readout system detects when this happens, adds a set of PIXCNT* cards in the image header, and a red line shows up on the obsbot/copilot strip chart.
Every now and then, an image appears which shows a shifted region in one or more CCDs. Please log this. You can identify images with shifts by running 

'''Frequency:''' A couple of times per night in the early Feb 2016 run.

'''Fix:''' 
- Add this as a bad exposure in the file ~products/mosaic3/logs/bad_expid.txt

=== 5. Repeat images ===

This is very hard to diagnose, and requires you to pay attention to the obsbot/copilot output, the individual images, and occasionally running the command: 

hselect mos3*fits[0] $I,READTIME yes 

in the IRAF window. 

If the READTIME values begin to repeat, this is a sign that new images are not being written out. Another clue is that the read out counter on the NMSL gui goes straight from 44 to 0. A third clue is that the MOSSTAT command is failing to find any stars in the image, even though the offsets were small when the sequence was started. 

If any of these are true, then it means that the images are just duplicates of previous images. You need to stop the sequence using CTRL-C in the window where the script is being exectuted, and restart everything: 
	- Stop MOSAIC (from the startup gui)
	- In a NOCS xterm window, type: 
{{{
		nocs nuke pana
		nocs nuke dhs
}}}
	- Start MOSAIC (from the startup gui)
Once everything comes up and you have rearranged windows how you like, take a ZERO frame to make sure that the amps are all OK. Check the noise on the zero image using mscstat. 
If this is not OK, then see instructions for Problem 1. 

From here, it is best to make a new plan (strategy file) and start it up. 

Log all the images that had identical READTIME values. 

=== 6. Detector Timeouts ===

Occasionally, the numbers on the countdown timer on the NMSL GUI will turn red during readout, and the timer will read down to -105 or so. 
When this happens, it usually means that the image is lost. You can try the following:
       - ditscmd nohs nohs_endobs
       - nocs nmsl reset
Then try taking a zero image. See if it reads out, and check the image using mscstat. 
If it looks ok, you can start a new observing script. 
However, pay attention to make sure that the next few images are written out.

=== 7. Focus drifting ===

If you lose track of the focus and don’t know whether you are below or above the focus value, wait for the readout countdown to start 
and then type CTRL-c  in the window executing the script. Wait for the readout to complete and the “DONE” to appear on the NMSL window.
Then execute
{{{
	ditscmd nohs nohs_endobs
}}}
Then do a focus sequence. 
	- Create a focus script using the “Focus” button on the NGUI
          This pops up the following window: 
[[Image(NOCS_FOCUS.tiff, 600px)]]
	- use a good guess for the mid-point: 
                 focus in zd filter = -8400 + (1.4C - Ttruss)*110
          * an instrument focus step of -100 (or -70, if the seeing is good)
          * at least 7 focus iterations and 
          * a detector pixel shift of 30; 
          * Make sure you have chosen the zd filter
	- execute the focus sequence
- Edit the /data2/observer/mscfoc.cl script to change the file name of the focus image
- In the IRAF window, type "mscfoc.cl" and follow the instructions
- Once you have determined the best focus: 
		- set the focus value in the MCCD window and hit “Apply”
		- log the Truss temperature and focus values. 
- Create a new observing script
- Tell the OA that you have determined a new focus and changed the focus. 
- Restart observing

=== 8. Images of stars on the guider are bouncing around ===
	- If the wind is high, ask the OA to close the louvers