Changes between Version 67 and Version 68 of PublicPages/MayallZbandLegacy/NotesforObservers/Problems


Ignore:
Timestamp:
Jan 30, 2017 2:27:14 PM (7 years ago)
Author:
Benjamin Alan Weaver
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PublicPages/MayallZbandLegacy/NotesforObservers/Problems

    v67 v68  
    146146== 7. Focus drifting ==
    147147
    148 If you lose track of the focus and don’t know whether you are below or above the focus value, wait for the readout countdown to start
    149 and then type CTRL-c  in the window executing the script. Wait for the readout to complete and the “DONE” to appear on the NMSL window.
    150 Then execute
    151 {{{
    152         ditscmd nohs nohs_endobs
    153 }}}
    154 Then do a focus sequence.
    155 - Once you have determined the best focus:
    156                 - set the focus value in the MCCD window and hit “Apply”
    157                 - log the Truss temperature and focus values.
    158 - Create a new observing script
    159 - Tell the OA that you have determined a new focus and changed the focus.
    160 - Restart observing
     148 Symptom:: If you lose track of the focus and don’t know whether you are below or above the focus value.
     149 Frequency:: Unknown.
     150 Fix::
     151       - Wait for the readout countdown to start and then type CTRL-c  in the window executing the script. Wait for the readout to complete and the “DONE” to appear on the NMSL window.
     152       - Then execute
     153         {{{
     154ditscmd nohs nohs_endobs
     155         }}}
     156         Then do a focus sequence.
     157       - Once you have determined the best focus:
     158         - set the focus value in the MCCD window and hit “Apply”
     159         - log the Truss temperature and focus values.
     160       - Create a new observing script
     161       - Tell the OA that you have determined a new focus and changed the focus.
     162       - Restart observing
    161163
    162164== 8. Images of stars on the guider are bouncing around ==
    163165
    164 '''Symptoms:''' Guider images are visibly bouncing around on the TV guider image, and the seeing is poor, with the image extension being
    165 primarily in the EW (i.e., up-down) direction:
    166 
    167 [[Image(windshake.tiff, 300px)]]
    168 
    169 '''Fix:''' If the wind is high, ask the OA to close the louvers that allow additional outside air to circulate into the dome.
    170 If observing near zenith, the lower part of the dome slit can also be closed.  That lower part of the slit only starts
    171 vignetting the telescope if observing to airmasses larger than 2.
    172 
    173 '''Fix 2:''' If accounting for the wind does not fix the problem, it may be related to an issue of telescope tracking, as was the case in February 8-9 2016.  You can try a test of moving to a different part of the sky and taking a 60-second exposure.  If the stars are round, but are elongated when you move back to the field, there may not be much you can do other than report the problem and continuing to observe.  The problem seen in February 8-9 2016 was traced to the RA axis encoder tape.  Shelby Gott cleaned the tape, which fixed the problem.
     166 Symptom:: Guider images are visibly bouncing around on the TV guider image, and the seeing is poor, with the image extension being primarily in the EW (''i.e.'', up-down) direction:[[BR]]
     167           [[Image(windshake.tiff, 300px)]]
     168 Frequency:: Unknown
     169 Fix:: If the wind is high, ask the OA to close the louvers that allow additional outside air to circulate into the dome. If observing near zenith, the lower part of the dome slit can also be closed.  That lower part of the slit only starts vignetting the telescope if observing to airmasses larger than 2.
     170 Fix 2:: If accounting for the wind does not fix the problem, it may be related to an issue of telescope tracking, as was the case in February 8-9 2016.  You can try a test of moving to a different part of the sky and taking a 60-second exposure.  If the stars are round, but are elongated when you move back to the field, there may not be much you can do other than report the problem and continuing to observe.  The problem seen in February 8-9 2016 was traced to the RA axis encoder tape.  Shelby Gott cleaned the tape, which fixed the problem.
    174171
    175172== 9. Bad telescope pointing ==
    176173
    177 '''Symptoms:''' The reported telescope offsets from the python "obsbot" or the IDL "mosstat" show
    178 that the pointing has gone off by more than 20 arcsec.  The "mosstat" script will likely fail to match
    179 stars if this offset is larger than 20 arcsec.
    180 
    181 '''Frequency:''' This happened many times per night in 2015, but should happen only rarely now.
    182 
    183 '''Fix:'''
    184 The operator will need to recenter the telescope. One can do this by
    185 telling the OA to offset the telescope in the *OPPOSITE* direction of the computed mosstat positional offsets, i.e., if you see:
    186 {{{
    187    RA, Dec offsets: -15.230 14.756
    188 }}}
    189 ask the OA to offset the telescope +15" in RA (i.e., east) and -15" in DEC (i.e., south) and zero the coordinates there. Only
    190 apply these offsets when the exposure is done (i.e., during readout) and when the telescope is not moving!
    191 
    192 One can also check the observing progress using the almanac command:
    193 {{{
    194   IDL> almanac, 10001, /noprint
    195 }}}
    196 will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window).  This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether  one should create and upload a new JSON observing script. The /noprint just prevents the decstat output for every frame; if you want to see that, then don't use the /noprint keyword.
     174 Symptom:: The reported telescope offsets from the python "obsbot" or the IDL "mosstat" show that the pointing has gone off by more than 20 arcsec.  The "mosstat" script will likely fail to match stars if this offset is larger than 20 arcsec.
     175 Frequency:: This happened many times per night in 2015, but should happen only rarely now.
     176 Fix:: The operator will need to recenter the telescope. One can do this by telling the OA to offset the telescope in the '''OPPOSITE''' direction of the computed mosstat positional offsets, ''i.e.'', if you see:
     177       {{{
     178RA, Dec offsets: -15.230 14.756
     179       }}}
     180       ask the OA to offset the telescope +15" in RA (i.e., east) and -15" in DEC (i.e., south) and zero the coordinates there. Only apply these offsets when the exposure is done (''i.e.'', during readout) and when the telescope is not moving! One can also check the observing progress using the almanac command:
     181       {{{
     182IDL> almanac, 10001, /noprint
     183       }}}
     184       will print out a summary of all the frames from 10001 to the present in a nice tabular form (widen the window).  This is useful for checking how the seeing and sky brightness have varied since frame 10001, and whether  one should create and upload a new JSON observing script. The /noprint just prevents the decstat output for every frame; if you want to see that, then don't use the /noprint keyword.
    197185
    198186== 10. tonight.sh script stops for no apparent reason ==
    199187
    200 '''Symptoms:''' Things go idle and quiet.  No exposures are being taken or read out, the telescope is not moving, and the software does not indicate any activity.  The shell window where you are running tonight.sh returns to a prompt, and has no error message of any kind.   
    201 
    202 '''Frequency:''' This happened > 5 times on the night of 06/07 March.  The tonight show just stopped after one exposure on multiple occasions.
    203 
    204 '''Fix:''' Re-run the mosbot -py command with the appropriate pass?.json files.  Check that the tonight.sh script has been rebuilt anew.  Start the new tonight.sh script.
     188 Symptom:: Things go idle and quiet.  No exposures are being taken or read out, the telescope is not moving, and the software does not indicate any activity.  The shell window where you are running tonight.sh returns to a prompt, and has no error message of any kind.   
     189 Frequency:: This happened > 5 times on the night of 06/07 March.  The tonight show just stopped after one exposure on multiple occasions.
     190 Fix:: Re-run the mosbot -py command with the appropriate pass?.json files.  Check that the tonight.sh script has been rebuilt anew.  Start the new tonight.sh script.
    205191
    206192== 11. MOSSTAT or MUPTILES don't see data ==
    207193
    208 '''Symptoms:''' mosstat or muptiles can’t see the data as its being taken.
    209 
    210 '''Frequency:''' Rare.
    211 
    212 '''Fix:''' check the MOS3_DATA environment variable and change as necessary in ~/.bashrc file on mayall-idl. Then:
    213 
    214  {{{export MOS3_DATA=/mosaic3/dataX/observer/DATE}}}
    215 or
    216  {{{source .bashrc}}}
    217 
    218 Note "dataX" is usually data2 or data3 and DATE is the UT Date of the morning on which we are observing.
     194 Symptoms:: mosstat or muptiles can’t see the data as its being taken.
     195 Frequency:: Rare.
     196 Fix:: Check the MOS3_DATA environment variable and change as necessary in ~/.bashrc file on mayall-idl. Then:
     197       {{{
     198export MOS3_DATA=/mosaic3/dataX/observer/DATE
     199       }}}
     200       or
     201       {{{
     202source .bashrc
     203       }}}
     204       Note "dataX" is usually data2 or data3 and DATE is the UT Date of the morning on which we are observing.
    219205
    220206== 12. Semaphore Failure ==
    221207
    222 '''Symptoms:'''
    223 - The tonight.sh script ends without writing the last image. There is a message in the NOCS xterm that tonight was executed in that says "FAILURE" and refers to a semaphore failure.
    224 - NMSL window shows countdown green 0.0
    225 
    226 '''Frequency:''' Rare. Several times in a three night run in March 2016.
    227 
    228 '''Fix:'''
    229 - First kill mosbot with ctrl C in the mzls@mayall-idl xterm.
    230 - Try to read out the last image.
    231 
    232  {{{ > ditscmd nohs nohs_endobs }}}
    233 
    234  This should work. The image may be ok, or it may show all 0's
    235 
    236 - Take a zero, and check with mscstat. Usually ok. (if not follow [0] above for full restart of Mosaic3.
    237 - restart mosbot.
    238 - restart tonight.sh.
     208 Symptoms::
     209            - The tonight.sh script ends without writing the last image. There is a message in the NOCS xterm that tonight was executed in that says "FAILURE" and refers to a semaphore failure.
     210            - NMSL window shows countdown green 0.0
     211 Frequency:: Rare. Several times in a three night run in March 2016.
     212 Fix::
     213       - First kill mosbot with ctrl C in the mzls@mayall-idl xterm.
     214       - Try to read out the last image.
     215         {{{
     216> ditscmd nohs nohs_endobs
     217         }}}
     218         This should work. The image may be ok, or it may show all 0's
     219       - Take a zero, and check with mscstat. Usually ok. (if not follow [0] above for full restart of Mosaic3.
     220       - restart mosbot.
     221       - restart tonight.sh.
    239222
    240223== 13. Elongated images due to 4MAPS valve problems ==
    241224
    242 '''Symptoms:'''
    243 - Images appear elongated. In April 2016 the PA was ~20 - 30 and possibly due to a valve issue in the 4MAPS system.
    244 - 4MAPS o10, o11 might show blue signifying a delta psi > 1 indicating an open valve. The valve stuck at blue (delta psi > 1) during an exposure usually, but not always, leads to elongated images since forces on the mirror may be bad.
    245 
    246 '''Frequency:''' Rare. Occurred on several nights in April, 2016.
    247 
    248 ''' Fix '''
    249  You can tell if the valve is stuck open if the “light” on the 4MAPS display of the airbags turns blue (for, e.g., o11 or i11) for an extended period of time (more than 30 sec, say). Flickering between red/yellow/green does not count; However, if the valve is stuck open, the color on the light should go from green to yellow to red to blue.
    250 
    251  Looking at the engineering GUI for 4MAPS will show the pressure reading dropping. If it is stuck on blue, then the fix Bob Marshall recommends is the following:
    252  - manually command (using the 4MAPS gui) the pressure on the problematic airbag '''lower than the reading.''' This should try and force the exhaust valve to open.
    253 
    254  - then set the pressure back '''up to the nominal reading'''; this should try and force the exhaust valve to close.
    255 
    256  This can be done without moving the telescope, but you probably want to stop the mosbot script, since you don’t want the script to command the telescope to move while you are repressurizing the bag.
    257 
    258  If this test does not work with the telescope in its current position, then the next step is to take the telescope to zenith and repeat the procedure.
     225 Symptoms::
     226            - Images appear elongated. In April 2016 the PA was ~20 - 30 and possibly due to a valve issue in the 4MAPS system.
     227            - 4MAPS o10, o11 might show blue signifying a delta psi > 1 indicating an open valve. The valve stuck at blue (delta psi > 1) during an exposure usually, but not always, leads to elongated images since forces on the mirror may be bad.
     228 Frequency:: Rare. Occurred on several nights in April, 2016.
     229 Fix::
     230       - You can tell if the valve is stuck open if the “light” on the 4MAPS display of the airbags turns blue (for, ''e.g.'', o11 or i11) for an extended period of time (more than 30 sec, say). Flickering between red/yellow/green does not count; However, if the valve is stuck open, the color on the light should go from green to yellow to red to blue.
     231       - Looking at the engineering GUI for 4MAPS will show the pressure reading dropping. If it is stuck on blue, then the fix Bob Marshall recommends is the following:
     232       - manually command (using the 4MAPS gui) the pressure on the problematic airbag '''lower than the reading.''' This should try and force the exhaust valve to open.
     233       - then set the pressure back '''up to the nominal reading'''; this should try and force the exhaust valve to close.
     234       - This can be done without moving the telescope, but you probably want to stop the mosbot script, since you don’t want the script to command the telescope to move while you are repressurizing the bag.
     235       - If this test does not work with the telescope in its current position, then the next step is to take the telescope to zenith and repeat the procedure.
    259236
    260237[[BackLinks]]