= Problems During Observing .. And What to Do =

[[PageOutline]]


== -1. To stop an exposure ==
If you need to stop the current exposure and the system is running fine, do not hit Abort!

Do this instead:

Using the “Abort” button does not work. The best thing to do if you need to stop an exposure quickly is to do the following:

 1. close the shutter by pressing the “Dark” button on the MCCD GUI
 1. cd ~/exec/mosbot and type touch quit 
 1. cntrl-C out of the python mosbot.py 

This will count down the exposure to completion, but it won’t crash nocs or compromise the system.

== 0. No images being written ==

 Symptom:: No new images are showing up on disk or in the obsbot display, and the real time display does not update.  The control system and readout timer, however, had no indication that data wasn't being written.
 Frequency:: This appeared as a new problem starting on 5/6 Feb 2016.
 Fix:: 
       - In a mosaic3 xterm window type
         {{{
touch ~observer/exec/mosbot/quit
         }}}
       - Wait for the exposure to complete and the observing script to stop
       - Stop NOCS using the red "Stop MOSAIC" button on the MOSAIC 3 MENU gui
       - Once NOCS has shutdown, find the blue xterm window which came up when NOCS was launched and type
         {{{
nocs nuke pana 
nocs nuke dhs
nocs status all
         }}}
       - The output from nocs status all should show that all processes are STOPPED and only 3 pvm processes RUNNING
       - Restart NOCS using the blue "Start MOSAIC" button on the MOSAIC 3 MENU gui
       - Once the system is back up, take a ZERO image to make sure everything is working
       - When the image appears, in the IRAF window type 
         {{{
mscstat <file name> 
         }}}
       - If the results of the mscstat show the noise (in the Standard Deviation column) to be 4<rms<10, then ...
       - restart the observing script by following the instructions here

== 1. Images appear on the real time display, but are not written immediately to the data directory ==

 Symptom:: The real time display shows updated images, but files are very slow to appear in the data directory.  The MOSSTAT and copilot displays are slow to update; copilot may eventually flag the image with a "readtime" failure.
 Frequency:: Frequent as of early February 2016.
 Fix:: 
       * In the Data Handling System window in the mosaic3:1 VNC session
         - select the Shared Memory Cache tab
         - Click Process All
         - Click Update Status
       * The image should then appear in the data directory.  

== 2. Detector timeouts ==

 Symptom:: Count-down timer in NMSL GUI turns red keeps counting down negative numbers (to -105 or so), or hangs during exposure, and no new images are read.[[BR]]
           [[Image(nocs_timeout.png, 400px)]]
 Frequency:: Several times per night in the early Feb 2016 run. No timeouts on Feb 5, one on Feb 6.
 Fix::
       - Stop the observing script using "touch quit" (don't ctrl-C unless desperate)
       - Make sure that the NMSL text window is visible. Wait for the text window to finish counting up.
       - before stopping Mosaic, double check that "Shutter" and "ready" are both in the "Dark" position
       - Stop Mosaic, and issue the following commands on mosaic3 :
          {{{
nocs nuke pana 
nocs nuke dhs
          }}}
      - Start Mosaic. The nuke commands will clear issues with stale PAN processes.
      - Take a zero and see if it looks ok by running mscstat. If good, restart the bot and keep going!
      - There is (should be) no need to Stop/Start Cameras. However, if it doesn't look ok, then you may have to do a full restart.
        * Stop Mosaic
        * Stop Cameras
        * Start Cameras
        * Start Mosaic
      - Take a zero image, make sure it reads out, and check image by running "mscstat" If the zero image looks ok, you can restart the observing script.

== 3. Images show bad amp ==

 Symptom:: One amp has extremely large noise, looks awful the display of the image with "mcsdisplay" with a noise > 1000 ADU/pix from the "mscstat" command.
 Frequency:: Several times in Dec 2015 run, never in early Feb 2016 run.
 Fix::
       - Stop the observing script with cntrl-C
       - Reset the NOCS controller by typing the following in a NOCS xterm window:
         {{{
nocs reset ccp
nocs init ccp
         }}}
       - Take two zero images. The first image is always bad. The second should be OK. Check it using  "mscstat <filename>" once it has read out. All rms values should be between 4-7 ADU except for amp 6 with will be 8-10 ADU.
       - Re-start the observing script

== 4. 4MAPS primary mirror support  goes “OFF AIR” during observing ==

 Symptom:: The OA should notice when this happens, but you can also see if the mirror support is listed as "OFF AIR" on the 4MAPS GUI. The images will also appear to have very large and strangely-shaped PSFs.[[BR]]
           [[Image(4MAPS_gui.tiff, 400px)]] [[Image(4MAPS_OFF_AIR.tiff, 400px)]]
 Frequency:: This happened several times per night in the Dec 2015 run, but with reduced frequency in the early Feb 2016 run.
 Fix::
       - Stop the observing script with cntrl-C
       - Wait for it to finish and for “DONE” to appear on the NMSL GUI
       - Type “ditscmd nohs nohs_endobs” in the xterm window in which the script was stopped. This should readout the last image and display it on the real-time display ximtool window
       - Tell the OA they can move the telescope to zenith and reset 4MAPS
       - Check the last image taken when 4MAPS failure occurred - the image quality is probably bad.  Add this as a bad exposure in the file ~products/mosaic3/obstatus/bad_expid.txt
       - Once the OA has reset 4MAPS, re-start the observing script

== 5. Shifted images ==

 Symptom:: Sub-blocks of an image have been shifted in the readout, and is obvious on the display as the overscan region being in a different place. The readout system detects when this happens, adds a set of PIXCNT* cards in the image header, and a red line shows up on the obsbot/copilot strip chart. Every now and then, an image appears which shows a shifted region in one or more CCDs. Please log this. You can identify images with shifts by watching  the copilot plots (for points with vertical red lines in the seeing plot), looking for the ERROR messages on the mosstat output, or just doing 
           {{{
hselect mos*fits[0] $I,pixcnt1 yes
           }}}
           in the IRAF window.[[BR]]
           [[Image(imageshift.png,400)]]
 Frequency:: A couple of times per night in the early Feb 2016 run.
 Fix::
       - Don't stop taking data.  This one exposure will be bad, but it should recover on its own
       - Add this as a bad exposure in the file ~products/mosaic3/obstatus/bad_expid.txt
       - And add a note to the logs

== 6. Repeat images ==

 Symptom:: The image data is an exact duplicate of a previous image, usually the one from 6 images prior. The headers are not duplicated, only the data, although the READTIME card will be the same to all decimal places to the preceding image. You can see whether images are repeating by typing the following command into the IRAF cl window:
           {{{
hselect mos3*fits[0] $I,readtime,exptime,title yes
           }}}
           and look at the "readtime" column. If the numbers are the same in all decimal places and repeating, then the images are repeats. You will have to stop taking data and restart the system. 
 Frequency:: Several times per night during the early Feb 2016 run, but should be less often now. 
 Fix::
       - If you are running a mosbot script, stop it by going to the directory in which the script is running (~/exec/mosbot/) and  type "touch quit" and then wait for the script to end. 
       - use the MOSAIC startup GUI and Stop MOSAIC
       - In a NOCS xterm window, type: 
         {{{
nocs nuke pana
nocs nuke dhs
         }}}
       - Start MOSAIC (from the startup GUI)
       - Take a zero frame and make sure all the amps are OK and have reasonable noise levels using "mscstat"
       - Re-start the observing script
       - List as bad exposures in the file ~products/mosaic3/obstatus/bad_expid.txt

== 7. Focus drifting ==

 Symptom:: If you lose track of the focus and don’t know whether you are below or above the focus value.
 Frequency:: Unknown.
 Fix::
       - Wait for the readout countdown to start and then type CTRL-c  in the window executing the script. Wait for the readout to complete and the “DONE” to appear on the NMSL window.
       - Then execute
         {{{
ditscmd nohs nohs_endobs
         }}}
         Then do a focus sequence. 
       - Once you have determined the best focus: 
         - set the focus value in the MCCD window and hit “Apply”
         - log the Truss temperature and focus values. 
       - Create a new observing script
       - Tell the OA that you have determined a new focus and changed the focus. 
       - Restart observing

== 8. Images of stars on the guider are bouncing around ==

 Symptom:: Guider images are visibly bouncing around on the TV guider image, and the seeing is poor, with the image extension being primarily in the EW (''i.e.'', up-down) direction:[[BR]]
           [[Image(windshake.tiff, 300px)]]
 Frequency:: Unknown
 Fix:: If the wind is high, ask the OA to close the louvers that allow additional outside air to circulate into the dome. If observing near zenith, the lower part of the dome slit can also be closed.  That lower part of the slit only starts vignetting the telescope if observing to airmasses larger than 2.
 Fix 2:: If accounting for the wind does not fix the problem, it may be related to an issue of telescope tracking, as was the case in February 8-9 2016.  You can try a test of moving to a different part of the sky and taking a 60-second exposure.  If the stars are round, but are elongated when you move back to the field, there may not be much you can do other than report the problem and continuing to observe.  The problem seen in February 8-9 2016 was traced to the RA axis encoder tape.  Shelby Gott cleaned the tape, which fixed the problem.

== 9. Bad telescope pointing ==

 Symptom:: The reported telescope offsets from the python "obsbot" or the IDL "mosstat" show that the pointing has gone off by more than 20 arcsec.  The "mosstat" script will likely fail to match stars if this offset is larger than 20 arcsec.
 Frequency:: This happened many times per night in 2015, but should happen only rarely now.
 Fix:: The operator will need to recenter the telescope. One can do this by telling the OA to offset the telescope in the '''OPPOSITE''' direction of the computed mosstat positional offsets, ''i.e.'', if you see: 
       {{{
RA, Dec offsets: -15.230 14.756
       }}}
       ask the OA to offset the telescope +15" in RA (i.e., east) and -15" in DEC (i.e., south) and zero the coordinates there. Only apply these offsets when the exposure is done (''i.e.'', during readout) and when the telescope is not moving! One can also check the observing progress using the almanac command:
       {{{
IDL> almanac, 10001, /noprint
       }}}
       will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window).  This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether  one should create and upload a new JSON observing script. The /noprint just prevents the decstat output for every frame; if you want to see that, then don't use the /noprint keyword. 

== 10. tonight.sh script stops for no apparent reason ==

 Symptom:: Things go idle and quiet.  No exposures are being taken or read out, the telescope is not moving, and the software does not indicate any activity.  The shell window where you are running tonight.sh returns to a prompt, and has no error message of any kind.    
 Frequency:: This happened > 5 times on the night of 06/07 March.  The tonight show just stopped after one exposure on multiple occasions.
 Fix:: Check telescope drives are active and dome is tracking. Re-run the mosbot -py command with the appropriate pass?.json files.  Check that the tonight.sh script has been rebuilt anew.  Start the new tonight.sh script.

== 11. MOSSTAT or MUPTILES don't see data ==

 Symptoms:: mosstat or muptiles can’t see the data as its being taken.
 Frequency:: Rare.
 Fix:: Check the MOS3_DATA environment variable and change as necessary in ~/.bashrc file on mayall-idl. Then: 
       {{{
export MOS3_DATA=/mosaic3/dataX/observer/DATE
       }}} 
       or 
       {{{
source .bashrc
       }}}
       Note "dataX" is usually data2 or data3 and DATE is the UT Date of the morning on which we are observing.

== 12. Semaphore Failure ==

 Symptoms::
            - The tonight.sh script ends without writing the last image. There is a message in the NOCS xterm that tonight was executed in that says "FAILURE" and refers to a semaphore failure.
            - NMSL window shows countdown green 0.0
 Frequency:: Rare. Several times in a three night run in March 2016.
 Fix::
       - First kill mosbot with ctrl C in the mzls@mayall-idl xterm.
       - Try to read out the last image. 
         {{{
> ditscmd nohs nohs_endobs
         }}}
         This should work. The image may be ok, or it may show all 0's
       - Take a zero, and check with mscstat. Usually ok. (if not follow [0] above for full restart of Mosaic3. 
       - restart mosbot.
       - restart tonight.sh.

== 13. Elongated images due to 4MAPS valve problems ==

 Symptoms::
            - Images appear elongated. In April 2016 the PA was ~20 - 30 and possibly due to a valve issue in the 4MAPS system.
            - 4MAPS o10, o11 might show blue signifying a delta psi > 1 indicating an open valve. The valve stuck at blue (delta psi > 1) during an exposure usually, but not always, leads to elongated images since forces on the mirror may be bad.
 Frequency:: Rare. Occurred on several nights in April, 2016.
 Fix::
       - You can tell if the valve is stuck open if the “light” on the 4MAPS display of the airbags turns blue (for, ''e.g.'', o11 or i11) for an extended period of time (more than 30 sec, say). Flickering between red/yellow/green does not count; However, if the valve is stuck open, the color on the light should go from green to yellow to red to blue. 
       - Looking at the engineering GUI for 4MAPS will show the pressure reading dropping. If it is stuck on blue, then the fix Bob Marshall recommends is the following: 
       - manually command (using the 4MAPS gui) the pressure on the problematic airbag '''lower than the reading.''' This should try and force the exhaust valve to open. 
       - then set the pressure back '''up to the nominal reading'''; this should try and force the exhaust valve to close. 
       - This can be done without moving the telescope, but you probably want to stop the mosbot script, since you don’t want the script to command the telescope to move while you are repressurizing the bag. 
       - If this test does not work with the telescope in its current position, then the next step is to take the telescope to zenith and repeat the procedure.

== 14. Focus frames are causing timeouts ==

 Symptoms:: The focus frames tend to cause timeouts of the instrument. 
 Frequency:: Can be as large as 3/4 focus frames encounter this problem. 
 Fix:: We do not yet know the cause of this problem (it is a relatively new “feature”, as of the start of the 2017B observing semester). For now, please set up focus scripts with only 5 steps, which for some reason tends to increase the probability that they will complete. Run the first sequence with a pedestal focus step size of 125 units, and then run a second focus sequence with a smaller step size of 75 units once you have a good first-guess for the focus.  The problem with the focus frames also means that we should minimize the number of focus frames we run through the night - it is better to just try and “eyeball” the focus based on the contour plots of the PSF and estimates based on the temperature variations of the telescope truss. 

== 15. XQuartz-related windows on mayall-7 freeze up ==

 Symptom:: We have had recent (Aug 2017) issues with the X-windows on mayall-7 freezing up. When this happens, the observing appears to continue without interruption, but all windows that use XQuartz (like the NMSL and NGUI guis, NOCS xterms, MCCD gui from which you can change the focus, etc.) all freeze up.
 Workaround:: You can keep observing in this mode even with these windows frozen; you just have to ask the OA when you want to change focus, etc. In this mode, the “Terminal” command on the mac system still works and launches windows, so you can use this to ssh mosaic3 and execute any commands needed to interact with the exposure queue (''e.g.'', put in a “touch forcepass3" command, or "touch quit”).
 Fix:: If you really need to restart x-windows, you will have to stop the mosaic software - in this case, follow this procedure:
       a. open a Terminal window
       a. ssh mosaic3
       a. cd exec/mosbot/
       a. touch quit
       a. wait for the exposure to stop, the tonight.sh script to stop, and the exposure to display in the ximtoo window
       a. type “nocs stop all”; once this is done, type “nocs nuke pana” and “nocs nuke dhs”
       a. Go to the top menu bar and kill XQuartz - this will also kill a bunch of other windows and menus
       a. Restart Xquartz by clicking on the “Mosaic3” icon on the right hand side of the bottom left screen on Mayall-7
       a. Restart mosaic by using the “Star Mosaic” button


== 16. Sudden drop in transparency ==

 Symptom:: Sudden drop in transparency when the skies are clear. If this is caused by occultation of the telescope pupil, the pupil ghost image will show distortions (see image)
 Fix:: Tell the OA and have her/him fix it. In the past the pupil has been occulted by the mirror covers not retracting fully, or the dome drives tripping and resulting in occulting the telescope (check the dome drive error angle on the black VDU monitor), or the lower shutter being in the way.

[[Image(pupilghost_obscura_137456.jpg,400)]]



[[BackLinks]]