LWS
Troubleshooting

Summary

Computers

Datataking system hung

Symptom
System claims that it is acquiring data (e.g., Xpose/LWS says acquiring) but images are not being read out and no saveindex messages are being written in the coyote tip window.
Problem
The datataking system is hung.
Solution
  1. Do not attempt to start another exposure! This will only succeed in hanging the instrument control computer (coyote), requiring a full system reboot to recover.
  2. From a keokea window, type ltp (load timing pattern) to reboot the DSPs and load the default timing pattern. Wait for the command to complete. If it does not complete within one minute, you have a severe crash which requires further action; see these instructions on how to proceed.
  3. In Xpose/LWS, click Abort to halt exposure in progress. Click Set Mode and select the appropriate observing mode for your observations (required to load the proper timing pattern).
  4. Issue the command
    5. md nodtel=off
    to return to the "nod on" beam.
  5. Resume data acquisition.

Coyote is hung

Symptom
Datataking system is hung; ltp will not complete successfully; TkLogger complains about RPC: program not registered
Problem
The device driver on coyote is hung.
Solution
This is a severe crash and you should consider contacting your I.S. immediately for help. Follow these instructions to reboot the system:
  1. Try to get a prompt in the coyote tip window by typing the following magic incantation:
    2. ~#
    You should get a response ok from coyote. If not, try the magic incantation a couple more times (if still no ok prompt, your I.S. will need to rlogin to coyote directly using the root password and initiate a reboot with the reboot command).
  2. Once you get the ok prompt on coyote, initiate a reboot by typing
    3. boot
    . If you had to rlogin directly as root, uset he reboot command.
  3. While coyote reboots, kill your instrument control windows by de-iconifying the lwsStart window and typing Ctrl-C. This should kill the window and all of your GUIs.
  4. Manually quit QuickView and any remaining GUIs which were not terminated already (XLWS, Xpose/LWS, XCALCT, etc.)
  5. Wait for coyote to complete its reboot sequence and print a login: prompt in the coyote tip window. You do not need to login to coyote, just wait for the prompt to appear.
  6. Re-start the LWS software by selecting the LWS Start Up item from the LWS Control submenu on the OpenWindows pulldown menu.
  7. When the GUIs have returned, you may resume observing. Note that some keywords (e.g., grating name, central wavelength, etc.) may show illegal values and will be reported incorrectly in your image headers, but the instrument components will be in the same state they were in prior to the crash. Once you make a grating move, the keyword values will be correct.

Collect deamons dead

Symptom
Upon instrument startup or a motor move, TkLogger complains about RPC: program not registered
Problem
The collect deamons on coyote are not running
Solution
Reboot coyote as decribed in these instructions.

LWS software hung with the message -- IO to DSP has faulted

Symptom
LWS software hung with the message IO to DSP has faulted.
Problem
Communications with the DSPs has failed.
Solution
Reboot coyote as decribed in these instructions.

Couldn't get a coyote tip window

Symptom
Could not establish a tip connection to coyote.
Problem
Either coyote is down, the serial port is broken, or the connecting wires are damaged.
Solution
  1. Try the command
    2. ping coyote
    to determine whether it is reachable. If you can't ping or rlogin to coyote, then it has crashed badly. Best action in this case is to return the telescope to horizon and have summit personnel check electricity and connections to the forward cass module's electronics bay. If connections are fine then they can manually power cycle the electronics rack.
  2. If coyote is alive, the problem is in the serial port or wiring. Have electronic staff diagnose problems.

No output in coyote tip window

Symptom
coyote tip window comes up, but the only message printed is
connected
. Motor moves and data acquisition do not generate output in the window, but motor moves and image acuisition appears to complete.
Problem
The cable between the coyote serial port and keokea is not properly connected.
Solution
  1. Have electronic staff inspect serial line connections both at keokea and at coyote.
      2. Serial connectors may need to be reseated.
      Check the color coding on "TIP" serial lines and connectors that indicate LWS and NIRC.The "TIP" lines may be swapped.
  2. Try another motor move to see whether output is generated on the tip window.

Can't get keywords

Symptom
XLWS program can't get keywords; motor moves are not reflected in chanegs in the XLWS GUI; log tail window shows errors such as:
Nov 29 15:47:01 keokea lws_klib: extract.c,231(make_nonblocking): Unable to set socket flags, errno 9
Problem
The keyword library is not responding to keyword requests.
Solution
Shut down and restart all LWS software.

Keokea filesystem full

Symptom
Message
write failed: "/" filesystem full
appears in coyote tip window or a keokea window.
Problem
The root partition on keokea is full, probably because the logfile /var/log/local0 has gotten too large.
Solution
  1. Shut down all LWS software.
  2. Log in to keokea as user lws.
  3. Go to directory /var/log.
  4. Copy the log file to headquarters:
    5. cp local0 /net/papahaku/instrhome1/lws/logs/local0.DDmmmYY
    This operation may take a couple of minutes if the file is many megabytes.
  5. Log into keokea as user root.
  6. Go to directory /var/log.
  7. Remove the logfile:
    8. rm local0
  8. Create a replacement logfile:
    9. touch local0
  9. Locate the syslog daemon:
    10. keokea# ps -auxww | fgrep syslogd
root       120  0.0  0.3   72   88 ?  S    Dec 21  1:01 syslogd
root     12205  0.0  0.7  132  184 q0 S    18:26   0:00 fgrep syslogd
    In this case, the pid is 120.
  10. Re-start the log daemon:
    11. kill -HUP pid
    You may now log out of the root session.
  11. From the observing account, re-start datataking system and verify that the log tail window comes up.
  12. Perform an operation which will generate log output (e.g., home or move a motor via XLWS, or select Video Mode via Xpose/LWS). Verify that the log tail window generates output.
  13. Compress the old logfile by logging into any HQ machine as lws, going to the directory ~lws/logs, and typing:
    14. compress local0.DDmmmYY

Can't start exposure - outdir invalid

Symptom
Observer attempts to start an exposure, but even though the acquire keyword goes to TRUE, images are not being read out and no saveindex messages are being written in the coyote tip window. The following message is seen in the coyote tip window when one tries to start an exposure:
    Oct 14 00:55:28 coyote acquire16: RFits_create: create_fits_1: User 
    lwsdev not in passwd file.
    Oct 14 00:55:28 coyote acquire16: RFits_create: create_fits_1: User 
    lwsdev not in passwd file.
    Oct 14 00:55:28 coyote rpc.collectd: Caught acquire's death with 
    ACQERR="RFits_create: create_fits_1: User lwsdev not in passwd file." 
    exit_code=255.
Problem
The output directory keyword outdir is set to an illegal value. This can happen after a coyote reboot or when restart_collectd was run. These actions cause outdir to default to an illegal value, generally lwsdev@keokea:/s/sdata42/...
Solution
  1. Issue the command
    2. s outdir
    and verify that the username (prepended to the directory name) is valid and that the data directory exists.
  2. If the outdir is invalid, execute the lws_init script which will prompt for the data directory number and will then set the data directory appropriately.

Chopper

Chopper will not start chopping

Symptom
Guider shows no chopping and the chptrig keyword on the Chop-Nod GUI reads disabled.
Problem
The chopper needs to be initialized.
Solution
  1. Have the OA check for alarms and re-initialize the chopper from the DCS GUI if needed.
  2. If Chop Nod Control GUI shows that chpstyle = automatic, then click the Init Chopper button.

Chopper will not start chopping

Symptom
Guider shows no chopping and the chpcycle keyword on the Chop-Nod GUI is not incrementing.
Problem
The DSPs needs to be reset by reloading the timing pattern.
Solution
Type the following command at a keokea prompt to re-load the default timing pattern:
       ltp

Chopper will not start chopping

Symptom
Guider shows no chopping but the chpcycle keyword on the Chop-Nod GUI is incrementing.
Problem
The chopper trigger is set to the wrong value.
Solution
Click the Chop Init button on the Chop-Nod GUI to reset the chop trigger style to LEVEL.

Chopper is out of sync with images

Symptom
Guider shows chopping, but quicklook shows the object changing from positive to negative.
Problem
Either the chop trigger generator is set incorrectly or the chop frequency is incorrect.
Solution
  1. Halt the chopper.
  2. Have the OA verify that the chop trigger selector switch on the connect panel in the summit compute rroom is set to LWS and not NIRC.
  3. Verify the chop frequency listed in the Chop-Nod Gui.
  4. Restart the chopper.
  5. Start acquiring data.

Exposure starts before chopper settles

Symptom
Object is smeared on image.
Problem
The exposure started before the chopper had settled.
Solution
  1. Halt the chopper via the Chop Nod Control window.
  2. Restart the chopper.
  3. Wait 15 seconds for chopper to settle.
  4. Start acquiring data.
  5. As long as the chopper remains running at the same chopper frequency, stopping and restarting the chopper should not be required before subsequent images. But if the chop frequency changes or the chopper is halted (to slew the telescope, for example), then you may need to repeat this procedure.

Guider

Guider does not follow offset

Symptom
When executing a dither pattern, the guider does not follow the guider star.
Problem
Xguide is experiencing a "race condition" well known as a bug. This problem is caused when the "resume" directive is received before the "resumex" and "resumeY" position are sent. This can be verified in the guider log file (/nightly/log/autoguider.syslog).
Solution
  1. Have the OA stop and re-start the Xguide software.
  2. If problems persist, have the OA reboot TDC.

Guiding is lost when chopper starts chopping

Symptom
When the chopper starts, the guide star moves out of the guide box resulting in the two chop beams adjacent to the guide box.
Problem
The chopper parameters are not set to the LWS default values. For normal LWS operations  the chopper keyword chopstyle is set to manual. The manual chopstyle keeps one chop beam in the rest frame and the other chopbeam a full chopthrow away, typically 10 arcsec. The guide box will remain in the rest frame and the guide star will remain in the box when chopstyle is set to manual. Chopstyle is set from clicking on the the chop-nod gui of XLWS
Solution
  1. Abort exposure.
  2. Halt chopper so the OA can set up on the correct chop beam.
  3. Click the Chop Init button on the Chop-Nod GUI to reset the chop  style to MANUAL
  4. Click the Start button on the Chop-Nod GUI and verify guide star remains in guide box.
Start data acquisition.

Guiding is lost during chop-nod

Symptom
During chop-nod observations, guiding is lost when the telescope nods.
Problem
Either the OA set up the guider on the wrong chop beam, or the nod parameters are incorrectly set. If set correctly, then the nod is in the opposite direction to the chop and the amplitude is the same; hence, the guider should not have to move during the nod, because one of the chop beams should always remain in the box.
Solution
  1. Stop exposure.
  2. Halt chopper to ensure that the OA sets up on the correct chop beam.
  3. Issue the command
    4. md nodtel=off
    to return to the "nod on" beam.
  4. Allow OA to choose a guide star. Note that one must not choose a star whose image will conflict with another object upon chopping; i.e., if the chop direction is north-south at 10 arcsec amplitude and another star lies 10 arcsec south of the guide star, the guider will see a "double star" when it nods and this will cause guider failure.
  5. Use the Chop-Nod GUI to set up chopping and nodding correctly, or use a facility script such as quad_chop to take data. User scripts cannot yet set up chopping and nodding appropriately.
  6. Start data acquisition.

Guider wanders during chop-nod exposures

Symptom
QuickView shows image wandering over the detector, although there is always a target in the guidebox.
Problem
There is another star in the location of the "chop off" image of the guide star. Note that one must not choose a star whose image will conflict with another object upon chopping; i.e., if the chop direction is north-south at 10 arcsec amplitude and another star lies 10 arcsec south of the guide star, the guider will see a "double star" when it nods and this will cause guider failure.
Solution
  1. Stop data acquisition and halt guider.
  2. Issue the command
    3. md nodtel=off
    to return to the "nod on" beam.
  3. Allow OA to choose a new guide star.
  4. Use the Chop-Nod GUI to set up chopping and nodding correctly.
  5. Start data acquisition.

Guider jumps during chop or chop-nod exposures

Symptom
The guide star is carefully centered before chopping begins, but then immediately jumps out of the guide box.
Problem
The chopper is being run in automatic mode (chpstyle=automatic) instead of manual mode. This can happen if the observer does not perform a Chop Setup and the OA initialized the chopper.
Solution
  1. Stop data acquisition and halt guider.
  2. Issue the command
    3. md nodtel=off
    to return to the "nod on" beam.
  3. On the Chop-Nod GUI, click the Chop Setup button to reset the chopper settings to the LWS defaults. Verify that the value of the chpstyle keyword is manual.
  4. Allow OA to re-acquire the guide star.
  5. Start chopper and verify that guider continues to track star.
  6. Start data acquisition.

Instrument

No signal at 20 µm

Symptom
Exposures taken in the 20µm regime seem to contain only noise.
Problem
Either the image is saturated, atmospheric transmission is rotten, the ZnSe window is in, or the KBr dewar window has been damaged by water vapor.
Solution
  1. Use the background monitor box on the QuickView GUI to check that the image is not saturated. If the background level is above 90%, consider switching filters or reducing the frmtime via XCALCT.
  2. Check the CSO tau plots to determine the amount of precipitable water vapor. If tau is currently above 0.05 then atmospheric transmission at 20 µm will be poor.
  3. If tau is below 0.05 (1mm or less of precipitable water vapor), then verify that the ZnSe window is out. Note: you are allowed to remove the window ONLY if the dewpoint is at least 10 degrees below the current dome temperature.
  4. If problems persist, it may be that the hygroscopic KBr dewar window has been damaged by water vapor. This is very bad news, and can mean the end of your observing run. Determining the status of the dewar window requires pulling the forward cass module and inspecting the dewar window with a flashlight for evidence of misting or fogging. Replacing it requires warming the dewar, a multi-day procedure.

No signal at 23.1 and 24.5 µm

Symptom
Exposures acuired of a bright object at 23.1 and 24.5 µm contain only noise, but at 17.65 and 18.75 µm the object is easily detected.
Problem
Either the image is saturated or the ZnSe window is in.
Solution
  1. Use the background monitor box on the QuickView GUI to check that the image is not saturated. If the background level is above 90%, reduce the frmtime either using one of the supplied custom files for imaging or by creating one of your own..
  2. If the background is low, verify that the ZnSe window is out. The background level is higher with the ZnSe window in position. If the background doesn't change with the window out, then there is a mechanical problem with the window mechanism. Try issuing up to 10 window out commands to move the window. If the window is stuck, call the SA on duty. Note: you are allowed to remove the window ONLY if the dewpoint is at least 10 degrees below the current dome temperature.

Saturated signal

Symptom
Image looks saturated; background meter in QuickView is close to or at 100%.
Problem
Too much signal on the detector, due to either:
  1. Clouds;
  2. High amounts of airborne water vapor when observing near 20µm;
  3. Forgot to remove ZnSe filter for 20µm observing;
  4. Sector wheel in the beam.
Solution
  1. Halt data acquisition.
  2. Home the sector wheel by typing the command
    3. m sechome=1
    This will remove it from the beam.
  3. Verify that the ZnSe filter is out if observing near 20 µm. Note: you are allowed to remove the window ONLY if the dewpoint is at least 10 degrees below the current dome temperature.
  4. Try using a different filter with lower throughput, or reducing the frametime in the current filter (by using XCALCT).

Sector wheel runaway

Symptom
Images are contaminated by excess light and the coyote tip window shows continuous askmotor messages scrolling by.
Problem
The sector wheel is cycling continuously.
Solution
  1. Halt data acquisition.
  2. Click the red Motor Halt on the XLWS GUI to stop the sector wheel.
  3. Issue the keyword command
    4. m sechome=1
    to re-home the sector wheel.
  4. Re-start data acquisition.

Filter/grating/aperture wheel won't home

Symptom
When commanded to home, the filter, grating, or aperture wheel returns immediately, as if already at home. However, no motors appear to have moved.
Problem
The cable carrying the home signal is loose.
Solution
Have summit staff check the connection on the cable between LWS and the breakout box (B.O.B.).

Wheel will not stop rotating

Symptom
After initiating a home command, the mechanism never stops moving. Output on the coyote tip window repeats message: Askmotor
Problem
The homing switch failed to trigger during the mechanism movements.
Solution
In the keokea window, issue the command 
	tellmotor K
Try homing the mechanism again. If the problem persists, call your SA.

Telescope

Telescope will not offset

Symptom
When running a dither script or issuing moves from the command line or QuickView, the telescope does not move; i.e., FACSUM shows no change in position and the stars on the guider do not move.
Problem
The DCS simulator software is running.
Solution
Type the following command in a keokea window to halt the simulator:
make_dcs_real

Telescope will not nod

Symptoms
  1. When taking a chop-nod exposure, the telescope does not nod (i.e., FACSUM offset coordinates do not change and the images of the guide star do not move).
  2. In chop-nod mode, the images taken in the alternate nod beam are displayed as negative, rather than positive images.
Problem
The chop and nod parameters are not set appropriately.
Solution
  1. Shut down and re-start the XLWS gui.
  2. Bring up the chop-nod control window by clicking the Chop/Nod button on the XLWS gui.
  3. Halt the chopper by pressing the Stop button on the Chop/Nod gui.
  4. Verify that chop angle and throw are set properly.
  5. Re-start the chopper by clicking the Start button.
  6. Verify that the chop angle is correct by comparing the relative orientation of the chop beams seen on the guider eavesdrop display with the orientation of the field given by the guider compass rose.
  7. Take a chop-nod exposure and verify that the telescope now nods properly.

QuickView

QuickView will not respond

Symptom
QuickView does not respond to mouse input; e.g., selecting different tools has no effect, can't make vector plots with middle mouse, etc.
Problem
QuickView has crashed.
Solution
  1. Raise the IDL window behind the QuickView display and check for evidence of a fatal error.
  2. If QuickView has crashed, type exit to quit IDL.
  3. Re-start QuickView from the LWS Control menu on the OpenWindows pulldown menu.

Quickview will not update

Symptom
Quickview responds to commands but the image is not updating.
Problem
Either:
Solution
  1. Verify that you are acquiring data by checking the Xpose-LWS GUI; the box at upper left should read "Acquiring." If not, then you are not taking data and must start and exposure.
  2. If you are acquiring data but the coyote tip window is not giving saveindex messages on a regular basis (at least every nod cycle) then the data-taking system has crashed; go here for further help.
  3. Verify that QuickView is in "dynamic mode" by selecting the Buffer Control menu and clicking the dynamic mode button.
  4. If you have changed the output directory then you must quit and restart QuickView.

QuickView image shows only vertical stripes

Symptom
The image on quickview shows only vertical stripes, rather than a subtracted image.
Problem
Either the detector is off or QuickView is displaying raw data.
Solution
  1. Verify that detector power is on by checking the XLWS GUI.
  2. Select the Buffer Control menu in QuickView and ensure that the Chop Difference option is selected.

Image appears contaminated

Symptom
Image displayed by QuickView seems to include multiple objects or multiple spectra.
Problem
Either QuickView is in "accumulate" mode or the colormap stretch is inappropriate.
Solution
  1. Select the Buffer Control menu in QuickView and ensure that the Non-Accumulate option is selected.
  2. Adjust the contrast and brightness by clicking and dragging right mouse in the image window.

Xpose/LWS

Xpose will not start an exposure

Symptom
Observer clicked Start on Xpose/LWS but the exposure did not start.
Problem
Data-taking system was already acquiring data when Start was pressed. New exposure can't be started until the previous one is stopped.
Solution
  1. Use Abort to quit the exposure in progress, then click Start again.
  2. If this fails, you may have a problem with the datataking system hung or an invalid outdir.

XLWS

XLWS shows wrong settings

Symptom
The XLWS GUI displays incorrect values for the current aperture, grating, filter, etc.
Problem
XLWS has lost communications with the keyword library.
Solution
  1. Sometimes XLWS is just a bit slow; waiting a minute may clear the problem.
  2. Check the keywords from the command line to ensure that they are really begin reported incorrectly by XLWS. For example, if the grating name is bad on XLWS, type the following command in a keokea window to determine whether the problem lies in the keyword library or in XLWS:
    3. s graname
    If the reported value is the same as that given by XLWS, then they keyword library is at fault. If they differ, XLWS is the problem.
  3. If incorrect settings persist beyond a minute, kill and re-start the XLWS GUI.
Go to: LWS Home Page - Instruments Home Page - Keck Home Page

Last modified: Sat Oct 14 23:17:21 HST 2000