DEIMOS
Troubleshooting

Summary

Images

No light in images

Symptom
Images taken using a grating (spectral or zeroth order image) look like darks.
Problem
The grating is at the wrong tilt, or the grating cover was not removed when the grating was installed.
Solution
  1. Re-home both grating tilts with the following commands on polo: 
    	unstick_grating 3
	unstick_grating 4
  2. Have summit support personnel check for and, if necessary, remove the cover from the grating.

Images contaminated by red light

Symptom
Images show excess red light
Problem
A red LED inside or outside of the instrument has been left on.
Solution
Send someone up to locate the source of the illumination. One likely place is a red LED on the DEIMOS handpaddle which is lit when the paddle is left in MANUAL mode. Switch it to COMPUTER mode in order to disable the LED. Another thing to check is the plastic housing enclosing the optical encoders on the grating tilt stages, which can be deformed and thus leak light.

Direct images and spectra shifted vertically

Symptom
Spectra and/or direct images taken on the grating appear to be shifted vertically on the CCD images relative to their customary position.
Problem
The grating tilt encoder has lost its positional reference, most likely due to momentary loss of power to the tilt ensoder.
Solution
Re-home both tilt encoders by running the following commands on polo: 
	unstick_grating 3
	unstick_grating 4

Bias jumps in images

Symptom
Images show a significant jump in the bias level in one or more amplifier.
Problem
The CCD readout electronics are too warm.
Solution
  1. Check DEIMOS temperatures by issuing the temps command on polo.
  2. If electronics temps are too high, then have summit staff remove covers on the electronics as needed to improve airflow.

Computers & Software

FACSUM and compass roses wrong

Symptom
Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.
Problem
The DEIMOS rotator control computer (roto) has lost its DCS connection.
Solution
See corrective actions described below.

ds9 will not load images

Symptom
Newly acquired images do not appear on the the ds9 image display GUI, even though the GUI appears to be alive and responds to mouse input. The ds9relay widget is present and active.
Problem
The X server you are sitting at is no longer secure, possibly because someone has used the xhost command to open an insecure line of communication.
Solution
  1. Try issuing the command 
    	\xhost -
    to disable all insecure xhost access.
  2. If this does not work, log out and log in again.

ds9 will not load images

Symptom
Newly acquired images do not appear on the the ds9 image display GUI, even though the GUI appears to be alive and responds to mouse input. The ds9relay widget is not visible on the screen with ds9.
Problem
The ds9relay task has died and needs to be restarted.
Solution
  1. Run the ctx command on polo and verify that ds9relay is not running.
  2. From the OpenWindows menu, select the item DEIMOS Control Menu > Subcomponents... > Restart ds9.
  3. Verify that a ds9relay appears. If it does not appear or dies, see the item above regarding xhost problems.

Images are not saved to proper directory

Symptom
An output directory is defined (outdir commands returns valid result) but images are not going there.
Problem
The output directory does not exist.
Solution
Create the output directory using the mkdir command.

Images will not start

Symptom
Clicking the Expose button on the Dashboard window appears to have no effect, although testAll indicates that all systems are functional. Watching the output on the science CCD crate reveals the message: 
Image aborted: insufficient memory on host computer
Problem
The host computer is improperly configured.
Solution
  1. Have the sysadmin add the following lines to the /etc/system file on polo: 
    *       Below are changes added locally

*       set the maximum shared memory segment size up for DEIMOS
*       images to be passed from lickserv to other apps
set shmsys:shminfo_shmmax=268435456

*       set the maximum number of users up to a useful level
set maxusers=128

*       by default system uses 2% of RAM; or, as reported by "sysdef -i"
*       47415296        maximum memory allowed in buffer cache (bufhwm)
*       which matches what is expected for a system with 2.25 Gbyte RAM
*       we need enough for a DEIMOS image, and some room to spare
*       this is measured in kbyte
set bufhwm=163840

*       We believe that the default paging algorithm causes amazing
*       slowdowns to happen when writing files that exceed about 80 MB.
*       If we understand correctly, this may be a remedy.
*       see   http://www.sun.com/sun-on-net/performance/priority_paging.html
set priority_paging=1
  2. Reboot the host computer.

DCS keywords missing in image headers

Symptom
Keywords such as RA, Dec, ROTPPOSN, etc., are not found in the image headers. These keywords can still be obtained from the polo command line via show -s dcs commands.
Problem
The watch_ccd process has lost its connection to DCS and needs to be restarted.
Solution
Use this procedure to restart watch_ccd:
  1. Wait for the exposure to end; do not begin another
  2. rlogin keamano -l kics
  3. deimos restart watch_ccd
Newly-acquired images should now have these keywords in the image header. You can verify this by using the fitshead command: 
	polo{deimos}112: fitshead d0812_0081.fits rotposn
	d0812_0081.fits:  ROTPOSN = 270.00146389
If a ROTPOSN value is shown, that's good news. If no output is received, then the keyword is missing from the header and the problem has not been fixed.

Keywords missing from image headers

Symptom
Keywords are found (or reported) to be missing from image headers.
Problem
Communication with a keyword service is interrupted.
Solution
  1. View logfile. In general, if there is a problem with keywords missing from a DEIMOS FITS header, the best place to start is to look at the appropriate watch_ccd log file in /local/kroot/var/log on keamano:
    • If it's an FCS image that is missing one or more keywords in its FITS header, look at the logfile df_watch_ccdlog by issuing the command 
      	view_logfile df_watch_ccd
      or by selecting the item DEIMOS Engineering > Log files... > Log view... > df_watch_ccd log from the OpenWindows pulldown menu.
    • If it's a science CCD image, look at the logfile ds_watch_ccdlog by issuing the command 
      	view_logfile ds_watch_ccd
      or by selecting the item DEIMOS Engineering > Log files... > Log view... > ds_watch_ccd log from the OpenWindows pulldown menu.
  2. Locate errors. Use the timestamp of the FITS file to key to the appropriate timestamped messages in the log file. In such cases where keywords are missing from a FITS file, the corresponding watch_ccdlog file should have error messages about either
    • reading the missing keyword, or
    • reading some other keyword from the same service.
  3. If the former: then try to show the missing keyword; e.g., type the following at a polo prompt: 
    	show -s deimot tmirrval
    If that fails, then there is probably a problem either with the dispatcher that serves that keyword or the underlying hardware that that dispatcher controls.
  4. In the latter case: when the FITS file is missing keyword X from service Y but the watch_ccdlog file instead reports an error reading keyword Y from service Y, then the problem may be more indirect. Recall that once watch_ccd gets an error reading any keyword from service Y, it will not attempt to read any more keywords from that service for that image. Thus, an error reading keyword Z will not only cause keyword Z (from service Y) to be missing from that FITS file but any other keywords from that same service which have not yet been collected by watch_ccd for this image will also be missing. In such cases, you should then check the dispatcher that serves keyword Z along with its underlying hardware.

CCD crate will not boot

Symptom
CCD crate is unresponsive. Attempts to reboot via RESET button or power cycling the unit do not succeed. Connecting via telnet to the crate shows the following message printed out every few seconds: 
	Timeout waiting for ARP/RARP packet
Problem
The crate's is asking for an IP address and is not being given one because the YP service doies not know about it.
Solution
  1. Ensure that the appropriate entry for the VME crate appears in the master YP database from which the 'ethers' database is pushed. For the FCS crate, that entry is: 
    	00:80:42:0b:4e:8c fcsvmep.keck.hawaii.edu
  2. Change the nsswitch.conf file on keamano so that it resolves hardware ethernet addresses directly from the /etc/ethers file.

Dashboard will not run

Symptom
Dashboard window comes up with many imcomplete components listed, and numerous complaints about keywords it can't access.
Problem
One or more computers or keyword libraries which server keywords are not running.
Solution
Use the testAll or checktl commands to test keyword libraries. You may need to reboot CCD crates or restart software.

Mechanism Moves

Move times out

Symptom
Mechanism move fails with error message "Error code -1: Someone needs to define an error message for this condition"
Problem
A motor move timed out or otherwise failed in a probably innocuous manner.
Solution
Re-try the move.

No light from grating

Symptom
Spectrum appears to be a dark image, although the shutter appeared to open.
Problem
Grating tilt encoder controller lost power thus sending grating to the wrong tilt for imaging, or technician forgot to remove the grating cover when inserting the grating into the instrument.
Solution
  1. Re-home both grating tilts with the following commands on polo: 
    	unstick_grating 3
	unstick_grating 4
  2. Have summit support personnel check for and, if necessary, remove the cover from the grating.

Dewar filter wheel will not rotate

Symptom
Attempts to change the dewar filter wheel to another position fail. Inspection of the Filter Wheel Select window show that the filter wheel is stuck between two positions and cannot reach either of them.
Problem
A filter has not been loaded properly, and the filter holder is protruding from the wheel.
Solution
Have technician remove and inspect the offending filter, then re-install it properly.

Grating stage will not tilt

Symptom
Attempts to change tilt of a grating stage fail. Trying to home the stage using the modify -s deimot G3TLTCAL=1 command yields a timeout error.
Problem
The motor control process for this stage is hung.
Solution
Type the command unstick_grating n, where n is the number of the misbehaing grating (3 or 4), in a polo window to return the grating to normal operation.

Tent mirror not positioning reliably

Symptom
The tent mirror does not position reliably; the piezo controller may not appear to be functioning; FCS may not be tracking well.
Problem
The piezo dispatcher and/or controller operating the tent tent mirror may be in a bad state, possibly due to a power glitch.
Solution
First, restart the piezo dispatcher following the instructions obtained from the intrcon GUI:
  1. Log into keamano as kics
  2. Issue the command 
    	deimos restart dispatcher.piezo
If this fails to fix the problem, have summit technician cycle power on the piezo controller as follows:
  1. The tent mirror actuator is driven by a Physik Instrument P-845 Piezo controller. This controller is located in DEIMOS electronics bay #2; see this photo that shows the equipment located in this bay.
  2. The piezo controller is located in the outboard position (i.e., farthest from the center of the rotation axis) of this bay while the DEIMOS PXL camera electronics unit is located in the inboard position.
  3. To access the piezo controller, rotate DEIMOS so that bay #2 is accessible, then remove the panel from the rear of the bay (i.e., the panel on the side of the electronics ring that faces away from the telescope).
  4. The main power button is the square red pushbutton located in the lower right corner of the piezo controller's front panel. Switch the controller's power off for 30 seconds, then turn it back on.
Note that you can watch the output of the tent mirror control software (dispatcher.piezo) using the following command from a polo window: 
view_logfile -tail tmirr

Filter move succeeds but filter is "Unknown"

Symptom
Following a filter move, the filter name turns to Unknown. Inspecting the filter wheel position on the filter detail panel shows that the wheel appears to be at the proper position.
Problem
The filter wheel is unbalanced and thus when the filter rotation motor is depowered at the end of the move, the wheel turns enough that its raw encoder position no longer matches the defined position within the tolerance of 1000 units.
Solution
Possible solutions are:
  1. Have summit staff exchange filters in an attempt to balance the wheel.
  2. Have Lick software personnel modify the configuration file to increase the tolerance to an acceptable level.

Loss of air pressure

Symptom
Dashboard pops up a warning message "Air pressure has been lost!"
Problem
The pneumatic system on DEIMOS lost pressure, and pressure may still be out.
Solution
  1. If an exposure is in progress, abort it immediately. The loss of air pressure has likely caused the grating to unclamp, and if the exposure continues it will be ruined. If air pressure remains off, the CCD shutter will not close and the exposure will be ruined anyway. But if you're near the end of a long exposure, it's worth trying to salvage it.
  2. On the main Dashboard, open the Tricorder panel and check the barrel and cradle air pressure indicators at OKAY, then a problem exists. Have summit staff investigate the loss of air pressure.
Note that the air pressure is interrupted momentarily once per day around 8:30am by the air water trap purge system. Occasionally, this system may trigger one or more additional purges following at 30 minute intervals if the original purge did not clear the trap.
See also: Notes on air purge

Grating tilt in reverse limit

Symptom
Attempts to move the grating tilt stage fail. The G3TLTLIM or G4TLTLIM keyword indicates motor in secondary limit. The grating subpanel on the dashboard indicates that the tilt of the grating stage is at a limit.
Problem
The tilt stage has exceeded the allowed range of travel and has hit the secondary limit switch. This cuts power to the motor and prevents further motor moves.
Solution
It is necessary for summit personnel to assist in moving the stage out of a limit. This operation presents a physical hazard to the operator because the stage could move suddenly when it is backed off the limit switch. Accordingly, this task should only be performed by a qualified instrument technician!
  1. Send the grating to the unload position, either by opening the grating access hatch and pressing the appropriate button, or by issuing the command 
    	modify -s deimot gratepos=-3
    to send slider 3 to the unload position, or 
    	modify -s deimot gratepos=-3
    to send slider 4 to the unload position. The technician can then access the stage through the grating changeout hatch.
  2. Open the grating changeout hatch. De-power the motor by unplugging the black plastic connector carrying DC power to the motor.
  3. Manually rotate the wheel on the fiction drive mechanism to move the stage in the appropriate direction. An indication of the positive direction should be written on the wheel. If the grating tilt stage is in the negative secondary limit, you will need to move in the positive direction, and vice versa. You can monitor the current position using the G3TLTRAW or G4TLTRAW keywords to verify that the stage is moved in the proper direction. The technician may be able to detect the "click" of the mechanical limit switch when the stage moves off the limit.
  4. With hands clear of the tilt stage, restore power the motor by plugging in the black plastic connector carrying DC power to the motor. BEWARE! The stage may begin a tilt move when power is restored.
  5. When the stage is returned to a working state, re-home the tilt stage from the Initialize panel on the main DEIMOS dashboard GUI.

Grating tilt in "bogus" secondary limit

Symptom
Grating stage(s) fails to move, reporting a secondary tilt limit. However, the grating subpanel on the dashboard indicates that the stage isn't close to a tilt limit.
Problem
The secondary limit switch is falsely triggered.
Solution
Ensure via visual inspection that the stage is not in a limit. If needed, manually tilt the stage via the drive wheel to move the tilt for verification. This can be done by sending the affected grating to the unload position (e.g., modify -s deimot gratepos=-3 for the grating 3 unload position) and accessing it through the grating access hatch at the side of the instrument. If the limit switch is giving a bogus error, you can jumper out the secondary limit switch as follows:
  1. Pull back DEIMOS from the science position.
  2. Remove front cladding to access the grating slide area.
  3. Locate the slider in question.
  4. Locate the wiring interconnect panel on teh grating slider.
  5. Disconnect the appropriate wires (detailed description TBD -- ask Wagner).

Grating will not unclamp

Symptom
Repeated attempts to unclamp the current DEIMOS grating fail.
Problem
The pneumatic clamps holding the grating will not release.
Solution
  1. Try using the unclamp_grating command to unclamp the grating. By default, this will try 25 times to unclamp. If no success after 25 tries, go to next step.
  2. Have summit personnel disconect the air supply line at the rear of the instrument via this procedure. This should remove the air pressure and allow the clamps to release.

Grating will not move to unload position

Symptom
The grating will not move to the unload position, because the select stage hits a limit and the tilt move does not execute. GRSELORD=-3 as desired, but GRSELLIM shows a primary limit.
Problem
The move cannot complete due to the limit switch striker being out of position.
Solution
Use keyword-level commands to move the tilt to the unload position as described in this procedure.

Grating select move fails (pin not in)

Symptom
The grating fails to move into position. Move Failed is written below the grating box on the DEIMOS dashboard. Under the grating diagnostics subpanel, the Grating Message reads: 
 Can't set GRATEPOS=4, grating locating pin not in
And the pin is out, the position reads -999, and the named position is unknown.
Problem
We are at an unfavorable rotator postion where the gravity load prevents the grating from moving into position.
Solution
If you are trying to move to slider 4, execute the command 
clampup_slider4
to attempt to clamp up. Otherwise, simply re-try the move from the dashboard GUI. If the pin will not engage, then rotate to a PA that is 90 degrees away from this location and retry the grating move. If this fails, rotate another 90 and try again. If it is still failing, there may be additional problems.

Grating select move fails (positioning error)

Symptom
Grating move fails with error: 
Can't set GRATEPOS=3, motor shut off - positioning error.
Problem
The move failed, most likely due to excessive vibration in the lead screw generating an overtorque condition in the controller. This can happen in the vicinity of the slider 3 load position at certain DEIMOS position angles.
Solution
  1. Rotate DEIMOS by 90 degrees.
  2. Home the grating select stage using the command 
    modify -d deimot GRSELCAL=1
    or alternatively use the Initialize panel on the dashbaord GUI and click the Grating selectbutton.
  3. If the calibration succeeds, then re-try the original move. If it fails, suspect a mechanical problem.

Slitmask: select move fails with motor powered off error

Symptom
Slitmask move fails; SLMSKMSG and/or SLSELMSG keywords report that move failed and motor is powered off.
Problem
The motor move has failed due to a bad reading on a switch, usually either the one on the slitmask arm or on the door comb.
Solution
Based on one successful recovery from this error, the following procedure is suggested:
  1. Attempt to calibrate (home) the stage. This may succeed, in which case you can quit.
  2. Have summit support attempt to move the slitmask selector motor manually.
  3. Attempt to calibrate the stage.
  4. Use Galil-level commands to move the selector slightly.
  5. Attempt to calibrate the stage.
  6. Use Galil-level commands to insert and retract the slitmask.
  7. Attempt to calibrate the stage.

Slitmask: selected slitmask reads "UNKNOWN" after requesting a slitmask

Symptom
A slitmask move fails. The slitmask status reads "UNKNOWN" after trying to select a slitmask and move it into the optical path. Slitmask detail panel on DEIMOS dashboard indicates comb error when attempting to move a mask.
Problem
A slitmask is sticking on the hotdog mechanism when the select stage is slewed, triggering the comb.
Solution
Summit staff will need to inspect the comb on the instrument.
  1. Have summit staff send rotate the telescope to a position allowing access to the Keck II right Nasmyth platform.
  2. On the Dashboard's rotator subpanel, set allow DCS control to no to permit local control.
  3. On this same panel, click the Unlock button to allow the rotator to move.
  4. Using the deimos rotator sub-dashboard, enter a rotator position of -75 (or 285) to allow summit staff to access the slitmask hatch, or have the OA rotate to stationary position 0 deg.
  5. Have summit staff open the access hatch using the screwdriver at the instrument (pictures needed).
  6. Have summit staff inspect the slitmasks for signs of a mask stuck on the comb (pictures needed).
  7. If a mask is sticking, have summit staff remove this mask by undoing the 2 thumbscrews on the slitmask door, flipping down the door, and pulling out the mask.
  8. Close the slitmask door and shut the access hatch.
  9. Verify that the slitmask select stage will now operate freely by by selecting mask 2 and mask 12, thus running the system from end to end.
  10. Allow DCS control of the rotator.
  11. Have the OA run a stand-by init on the rotator.
  12. Resume observing.

Slitmask move fails

Symptom
Attempting a slitmask move fails.
Problem
Slitmask may be jammed.
Solution
See procedure for dealing with jammed slitmask.

Electronics

Motor controller fault

Symptom
The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running.
Problem
The motor controller has faulted.
Solution
Attempt to reset the motor controller as follows:
  1. From the main DEIMOS dashboard, click the daemons button to raise the daemon subpanel.
  2. Use the displaeyd status information on the daemons subpanel to verify the fault reportd by testAll.
  3. Use the Ctrlr 1 Reset or Ctrlr 2 Reset buttons as required to attempt to reset the controller.
If this fails, proceed to the next item.

Motor controller hung

Symptom
The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running. Attempting to reset the controller from the dashboard's daemons subpanel (as described above) fails.
Problem
The motor controller has hung.
Solution
Attempt to reset the motor controller as follows:
  1. Log in as kics to keamano.
  2. Use the reset_motor_controller script to reset the appropriate motor controller: 
    	reset_motor_controller n
    where n=1 or 2.
  3. After the script completes, use the command 
    	deimos status dispatcher2.n
    (where n=1 or 2) to verify that the appropriate dispatcher was restarted properly.
  4. If the dispatcher is reported to be not running, use the command 
    	deimos start dispatcher2.n
    (where n=1 or 2) to restart.
If this fails, proceed to the next item.

Ion pump is off

Symptom
The testAll command reports that one or both of the ion pumps is off.
Problem
The ion pumps need to be turned on.
Solution
State the solution to the problem here, numbering the steps to be taken:
  1. Go to the DEIMOS dashbaord GUI and click the Tricorder panel.
  2. Verify that the status of IONPUMP1 and/or IONPUMP2 is OFF. Also check the dewar temp; if the dewar temp is well above -100 C then you may not be able to start the ion pump.
  3. Click the Restart button under the appropriate ion pump to attempt to restart the ion pump. The status should change from OFF to starting.
  4. If the startup is successful then the status will eventually change to on. If the startup fails, the status will return to off, and you should try again to start the ion pump. Even if the dewar is cold, numerous restarts may be required before the ion pump will stay on.

Problem communicating with Lantronix unit

Symptom
Motor moves fail. Running testAll indicates a problem communicating to the Lantronix (IP 192.168.6.4) (Lantronix-barrel).

  • testAll indicates ... 
      Checking deits2..................OK (Lantronix - cradle)
  Checking deits3..................ERROR! (Lantronix - barrel)

    deits3 has the IP 192.168.6.4

  • Can't telnet to IP 192.168.6.4
  • Can't ping 192.168.6.4
Problem
The Lantronics unit is not functioning.
Solution
Follow the procedure for resetting the Lantronix unit.

Rotator

Rotator won't go into DCS Control Mode

Symptom
Rotator will not go into DCS control mode; attempt to modify ROTATMOD to DCS yields an error about INSTRUME and FOCALSTN not being set properly for DEIMOS.
Problem
The DEIMOS rotator control computer (roto) has lost its DCS connection.
Solution
Follow these steps to re-establish the DCS connection:
  1. Enter this command at a polo prompt: 
    	deirot restart
    You will be prompted to enter the kics password (which is different on DEIMOS computers than on others).
  2. Once the restart completes, verify that all is well by issuing the command deirot status from roto as kics; the output should resemble this.
  3. The observer must click the Allow DCS Control button and then select the yes option to enable DCS control of the rotator.
  4. The OA must do a Standby/Init on the rotator to re-initialize the rotator.
  5. Note: If restarting the deirot process fails to fix the problem, the real problem may be that keamano has stopped forwarding IP traffic to roto. See this document for details.

Rotator reporting wrong angle

Symptom
Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.
Problem
The DEIMOS rotator control computer (roto) has lost its DCS connection.
Solution
See corrective actions described above.

Can't find stars

Symptom
At the beginning of the night, pointing appears to be way off; stars not visible in guider.
Problem
Either the rotator has not been initialized and placed into DCS mode, or the DCS does not know the rotator position.
Solution
  1. Run testAll to verify that the instrument and telescope settings are correct (e.g., selected instrument is DEIMOS).
  2. If the rotator was not initialized at the start of the night, then follow these instructions to do so.
  3. If you have not yet commanded a rotator move, have the OA send the rotator to any angle (e.g., 0°).
  4. Verify on FACSUM that the rotator is in position angle mode.

Rotator spontaneously homes itself

Symptom
When sending a rotator move, the rotator unexpectedly executes a large slew to the home position.
Problem
The rotator software determined that the discrepancy between the readings on its two absolute encoders was too large and decided to home itself in order to reset the encoders before proceeding.
Solution
Allow the move to complete, then proceed with observing. Note that properly initializing the rotator at the start of each night should generally prevent this from happening.

Can't do STANDBY/INIT on rotator

Symptom
At the start of the night, the OA cannot perform the STANDBY/INIT operation on the rotator.
Problem
The rotator software on roto is not communicating properly with DCS.
Solution
  1. Restart rotator software as kics on keamano via 
    	deirot restart
    and attempt another STANDBY/INIT.
  2. If that fails, try 
    	deirot stop
    then wait 5 - 10 seconds, then issue the command 
    	deirot start
    and attempt another STANDBY/INIT.
  3. If that fails, restart DCSGUI.
  4. If that fails, reboot TDC and AUX, then restart DCSGUI and perform another deirot restart.

DCS Rotator Fault

Symptom
At the start of a telescope slew, the rotator faults. The rotator logfile indicates 
	Ignoring bad DCS demand #####,D (#successive bad demands=1): Demand - NNN.N (physical angle) is outside limits
Problem
The DEIMOS rotator received a request to go to an invalid rotator position. Usually this indicates a problem in the rotator angle interpolation routine.
Solution
  1. Issue the following command in order to ensure that the rotator is now giving appropriate demands: 
    	modify -s dcs rotdest=0 rotmode=stationary
    Failing to issue this command could cause the rotator to fault again after re-initializing.
  2. Perform an INIT (not necessary to do STANDBY/INIT) on the rotator.

Rotator moves are all outside of limits

Symptom
When the OA attempts to send a rotator move, SKY complains that the requested angle is out of range for the DEIMSO rotator although it is clearly a legal request.
Problem
The rotators limits are not properly defined.
Solution
  1. Verify the problem by displaying the keyword values: 
    	show -s dcs rotccwlm rotcwlm
    The correct values are: 
                          rotccwlm = -330.00 deg
                       rotcwlm = 402.00 deg
  2. If you get zero values back, have the OA re-load DEIMOS as the current instrument.

Rotator repeatedly starts spontaneous slews

Symptom
While attempting to track on sky, the DEIMOS rotator repeatedly goes into ACQUIRE mode and moves the rotator, causing guiding to be lost. Inspection of the rotator logfile (keamano:$KROOT/var/log/deirotlog) reveals many entries similar to this: 
	Ignoring bad demand #180846,D:  x_dmd=-2.666305 v_dmd=nan  a_dmd=nan
Problem
watch_rotator, the program that monitors DCS position broadcasts of ROTPDEST and extrapolates the required (x, vel, accel), is issuing velocity and accel values (NaN's). Typically, this occurs when watch_rotator is receiving bad values from DCS (perhaps due to one of those infamously sketchy EPICS/CA connections), and therefore it cannot fit a curve with appropriate velocity and acceleration.
Solution
  1. Enter this command at a polo prompt: 
    	deirot restart
    You will be prompted to enter the kics password (which is different on DEIMOS computers than on others).
  2. After 30 seconds, use the testAll command to check the status of the system. If problems are shown, issue the command deirot status from roto as kics; the output should resemble this.
  3. The observer must click the Allow DCS Control button and then select the yes option to enable DCS control of the rotator.
  4. The OA must do a Standby/Init on the rotator to re-initialize rotator control.

Flashing handpaddle lights

Symptom
LEDs on the DEIMOS rotator handpaddle are flashing.
Problem
The rotator needs to be re-initialized.
Solution
Re-initialize the rotator:
  1. Close all DEIMOS hatches and release the band brake.
  2. Switch the toggle switch on the DEIMOS handpaddle to COMPUTER.
  3. Open an xterm on polo.
  4. Type the following commands: 
    	modify -s deirot rotatlck=unlocked
	rotatcal
    Answer 'y' when prompted whether you are sure you want to home the rotator.
  5. Allow the homing operation to complete.

Rotator disabled due to ESTOP or power outage

Symptom
Rotator will not move or initialize. The Rotation subpanel on the DEIMOS dashboard indicates ESTOP from DCS.
Problem
The rotator is disabled because the Keck II facility ESTOP circuit was de-activated, either because someone pressed a facility ESTOP button or because of a power outage.
Solution
  1. Follow the DEIMOS Rotator Controller Reset Procedure.
  2. Verify that the ESTOP from DCS indicator on the dashboard's Rotation subpanel is no longer lit.

Flexure Compensation System

No light on FCS images

Symptom
No light is visible in an FCS exposure
Problem
Probably one of the following:
  1. CCD shutter is not open
  2. I or Z-band filter is in use
  3. CuAr lamp burned out
  4. Grating cover left on
  5. Grating is tilted side-to-side
Solution
Follow these steps to recover:
  1. Verify that the CCD shutter is open.
  2. If the I or Z band filter is in use, ensure that teh CuAr lamp is turned on; the LED does not emit at these wavelengths.
  3. If still no light, try switching to the LED lamp by requesting a direct-mode calibration image on the FCS Setup GUI.
  4. If still no light, acquire a spectrum with the science array and check whether any light is received. If not, send someone up to check whether the grating cover has mistakenly been left in place.
  5. If still no light, check whether the science spectra appear shifted to the left or right relative to the expected positions based on the slit layout. If so, the grating may not be properly seated.

FCS analysis fails

Symptom
FCS reports "cross-correlation failed" or "FCS disagree" errors.
Problem
Potential causes are:
  1. The grating tilt encoder has lost its position so the actual position does not match the reported position.
  2. Either the FCS reference image or the new FCS image contains a substantial cosmic ray.
  3. Grating became unclamped due to a loss in DEIMOS air pressure.
  4. Grating was not properly installed in its mount by summit technician.
Solution
  1. Re-home the grating tilt mechanism by either clicking on the RE-Cal button in the grating detail panel or typing "mmot g#tltcal=1" at a polo prompt, and where "#" is either "3" or "4".
  2. Inspect the FCS reference image for cosmic rays and if needed, re-take the reference image.
  3. Unclamp and re-clamp the grating via the dashboard.
  4. Have summit staff remove and re-install the suspect grating to ensure that it is seated properly in the slider.

FCS cannot match previous

Symptom
An attempt to use the "Match Previous" feature of FCS fails with the error Dewar X translation stage hit a limit.
Problem
The grating is either improperly seated now, or was improperly seated when the original FCS reference exposure was taken.
Solution
  1. Compare Y positions of the same reference spots on the original FCS reference image and the new reference image. Verify that they are separated significantly in Y. If not, then the problem is elsewhere.
  2. Have summit staff unload and re-install the grating.
  3. Re-try the "Match Previous" operation.

FCS ping-pongs in X

Symptom
FCS issues moves but does not converge on the target, which keeps bouncing back and forth in the X direction.
Problem
The piezo controller/actuator is not functioning properly.
Solution
  1. Inspect cabling at the FCS controller (located in the same bay as the PXL controller), particularly the high voltage line coming out the center of the controller.
  2. If cabling looks fine, check that the FCS controller channel is working properly by using one of the other channels on the controller (consult Lick first).

FCSTRACK script dies on startup

Symptom
When attempting to run fcstrack, the script dies with an Subscript out of range. error.
Problem
The script attempted to read an undefined keyword.
Solution
Locate the offending passage as follows
  1. Use the which fcstrack command to locate the executable file.
  2. Use cp to copy the file to a local directory.
  3. Use an editor to modify the first line of the file by adding the -x flag to print lines as they are executed.
  4. Determine the line at which the error occurs and use this information to isolate the offending keywords.
  5. Modify the keyword to have a non-null value.
In one case, the offending keyword was fcscusel, the name of the selected FCS lamp. The solution was simply to type 
	modify -s fcs fcscusel=Cu1

FCS focus tolerance too tight

Symptom
FCS refuses to lock in due to a lockout condition, which is claimed to be a focus mismatch although the focus is correct.
Problem
The focus tolerance keywords are not initialized properly.
Solution
  1. Check the keyword values via: 
    	show -s deifcs fcsfoto1 fcsfoto2
  2. If they are zero, kill and restart the FCSTRACK script, which should reset them to reasonable values like 200 units. If not, then you can also perform this step manually: 
    	modify -s deifcs fcsfoto1=200 fcsfoto2=200
  3. Kill and restart FCSGUI.

FCS Lockout (cannot track)

Symptom
FCS remains in lockout condition even when the grating, central wavelength, filter and focus values all look correct. It complains that there is no setup for the current configuration. Beware especially after a mask alignment.
Problem
Usually, the central wavelength of the grating is slightly off from reference value. There is finite resolution in the tilt mechanism and sometimes the "Go Back" function of the grating does not set the exact wavelength.
Solution
Move off the central wavelength, then back again.
  1. In the main Dashboard GUI, change the central wavelength by 1 Angstrom.
  2. Return to your desired central wavelength
This should solve the problem.

FCS CCD failure

Symptom
FCS cannot track. Looking at FCS images reveals that one of the two FCS CCDs is not reading out (i.e., half of the FCS image displayed on FIGDISP contains all zero-valued pixels), although the other CCD looks normal.
Problem
The signal chain for one of the FCS CCDs is interrupted.
Solution
An expert technician at the summit will need to check and repair the cabling to make a permanent repair. As a temporary measure, as long as one of the CCDs is reading out the support astronomer can enable partial FCS compensation by telling the FCS software to use only the one working CCD. To do this:
  1. Log in to polo as user kics.
  2. Edit the file /local/kroot/bin/fcstrack.offline_select. Under normal operation (both CCDs working), line 53 of this script reads: 
    	set offline = 0
    If CCD 1 (left-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read: 
    	set offline = 1
    If CCD 2 (right-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read: 
    	set offline = 2
  3. Kill and restart the fcstrack script. You should get a warning message: 
    	*** FCS CCD #1 has been marked offline ***
    Verify that FCS is now able to track.

FCS dewar stage at negative limit

Symptom
FCS complains of a lockout situation because the dewar stage is at a negative limit. FCS cannot re-position the dewar stage to correct for differences between the FCS reference image and the current FCS image.
Problem
One problem may be that the grating is improperly clamped either currently or when the reference image was acquired in the afternoon. If the grating is improperly clamped, the result could be large shifts between FCS images.
Solution
  • Try simply unclamping and reclamping the grating. If it doesn't work the first time, try it once more.
  • Rotate the instrument 90 degrees from the current position or to -30 deg (about the rotator value set by FCS setup in the afternoon). Then try clamping and reclamping the grating.
  • If that doesn't work, the current position of the grating is likely not at fault. It may have been that the grating was improperly clamped when the FCS reference images were acquired. If this is indeed the problem, then an FCS image will need to be retaken as a reference. You will need to unlock the rotator and dis-allow DCS control of the rotator. Next, reacquire FCS images following the afternoon procedures. When finished acquiring new FCS images, you will need put the rotator back into DCS control and have the OA do a standby-init on the rotator.

FCS cannot match previous

Symptom
When attempting to use the "Match Previous" function on the FCS setup GUI to match to the previous night's FCS reference image, the attempt fails with an error "unable to match". Comparing the current FCS image to the previous reference image shows a large spatial offset.
Problem
The grating tilt encoder has slipped and the new image is not properly registered.
Solution
  1. Recalibrate the grating tilt by typing the following command in a polo window: 
    modify -s deimot g3tltcal=1
    Note: if using slider 4, specify g4tltcal=1 instead.
  2. Re-try the match previous operation.
  3. If FCS still cannot match, use the fcsmov command to steer the current image so that it matches the previous reference image.

FCS fails to cross correlate images

Symptom
FCS images are flipped left/right or the spectrum appear shifted left/right relative to the FCS reference image. When FCS is tracking, large cross correlation errors are reported by the FCS error handler.
Problem
This problem is attributed to attenuated signal in the fiber optic digital communications link between the FCS CCD controller (inside DEIMOS) and FCS CCD VME crate (in the Keck-2 computer room).
Solution
This problem may not be solved at night. A technician will need to clean the FCS fibers and reseat the boards during the day.

TV Guider

Guider is not responsive

Symptom
Guider cannot be selected from Xguide.
Problem
Guider is powered off or requires a power cycle.
Solution
Cycle power to the guider off and on by executing the following command on polo: 
	tvpower cycle
This should take about one minute to complete.

Bad guider images

Symptom
Guider can be selected from Xguide but images from the guider do not show stars or only a portion of the field (top 1/3) show stars.
Problem
Either the guider is not properly configured or the shutter is broken.
Solution
Follow these steps to determine source of problem:
  1. Verify that the DEIMOS hatch is open from the main Dashboard panel.
  2. On the Dashboard's TV Guider subpanel, verify that the TV filter is set to a reasonable value (e.g., R). If it is set to Home, select R.
  3. On the Dashboard's TV Guider subpanel, verify that the TV focus is set to a reasonable value for the selected filter (e.g., 970 for R). Note that 1 is not a reasonable value.
  4. Note that the default value for the guider filter and focus is R & 970. If you suspect that the defaults should be in use, reselect these values.
  5. If still no good image, close the hatch, turn on the Qz lamp, and see whether any light is seen on the image.
  6. If no light is seen, the guider shutter may be inoperative. Replacing this is an extensive job for the day crew.

Bad guider images

Symptom
Light is seen on the guider images, but the images appear "smeared" horizontally.
Problem
The serial clock card in the guider controller is malfunctioning.
Solution
Repairing this is a full day's work for a talented daycrew. Have daytime summit staff pull DEIMOS, remove access panels on front of instrument, and check serial card in the guider head.

AUTOFOC and Mira

AUTOFOC returns abnormal telescope focus value

Symptom
After running AUTOFOC to focus the telescope, the secondary mirror position is more than a few tenths of a mm away from the nominal value of 0.8 mm.
Problem
The TV camera focus is not set correctly.
Solution
  1. On Dashboard, click on the TV Guider button to call up the guider window.
  2. Check the displayed TV camera focus value, which should be around 1000 units.
  3. Change this value as needed and click the MOVE button to make the change.
  4. Re-run AUTOFOC and verify that the value has changed.

Mira images are badly out of focus

Symptom
When running Mira, the images are badly out of focus.
Problem
The telescope is badly out of focus, possibly because the DEIMOS TV camera focus is not set correctly.
Solution
See the description for AUTOFOC returns abnormal telescope focus value.

AUTOFOC crashes

Symptom
Autofoc crashes with the following error message: 
	can't connect autofoc to aut_srv_task
Problem
The wrong version of AUTOFOC was run. There is a SunOS version for use with the SunOS guider, and a Solaris version for use with the Solaris guider.
Solution
Run the correct version of AUTOFOC.

MIRA fails to reset parameters

Symptom
Data directory, filename prefix, etc., are not properly reset after running MIRA.
Problem
The MIRA script failed to complete.
Solution
Have the OA or SA restore parameters as follows:
  1. Open a polo window as dmoseng
  2. Execute the command 
    	restore_state -verify
    This should restore the pre-MIRA settings.

Slitmask Alignment

Xbox Fails (Image not Written)

Symptom
When xbox is run, it fails before analyzing the first box, with an error akin to 
	extension BluSlits not found
Subsequent attempts to analyze this same image yield the same error.
Problem
The program started reading the image before the header was fully written. Due to an IRAF feature, the buffer is not flushed until you request it with teh flpr command.
Solution
  1. Type flpr and re-run xbox.
  2. If that fails, restart IRAF and re-run xbox.
  3. If that fails, inspect the image header (use fv -cmap 2 filename) and verify that the specified extention exists in the image header.

xbox Fails (Missing FITS Chunks)

Symptom
When xbox is run, it fails before analyzing the first box, with an error akin to 
	extension DESIslits not found
Subsequent attempts to analyze this same image yield the same error.
Problem
The FITS header extension called DESIslits is not being read by the software, eithe because it does not exist in the image header or because IRAF can't read it.
Solution
If you are trying to get a slitmask aligned and are pressed for time, proceed straight to the procedure for aligning slitmasks without FITS chunks. Otherwise, you can proceed as follows:
  • Use fv to view the header: 
    	cdata ; fv -cmap 2 filename
    Use the FV scroll bar to check for the DESIslits extension.
  • If DesiSlits is found, then try restarting IRAF and re-reading the file.
  • If DesiSlits is not found, then run the command check_fits_chunks to determine whether the FITS chunks are on disk.
  • If all indications are OK, then restart the write_image process as the current observer on polo: 
    	deimos restart write_image
If FITS chunks are missing, it is likely that something went wrong in the process of scanning the masks. Follow the procedure for dealing with a mask scan error, then take these additional steps:
  • Re-start dashboard GUI.
  • Verify that slitmask list agrees with expectation.
  • Run the command check_fits_chunks to determine whether the FITS chunks are on disk. If not, then seek help.
  • Then try taking a new alignment image and running the alignment software.

DREMEL

Bad response when scanning barcodes

Symptom
When scanning a filter or grating barcode, the instrument responds with a "bad" signal (3 short honks).
Problem
The stage was not at a defined position. Unless the position is defined, DREMEL will not accept the barcode.
Solution
Verify that you have a green light from the stage's position indicator before you scan. If not, re-select the position using the local controls on the instrument.

No response when scanning barcodes

Symptom
When a barcode is scanned using the handheld barcode scanner, the scanner beeps an acknowledgement but DEIMOS does not give the "honk" response.
Problem
Either DREMEL or the bargun dispatcher is hung or otherwise unresponsive.
Solution
  1. Verify that the DREMEL and Hand Held Barcode Scanner lights on the intrcon GUI are on. If not, restart these processes using the instructions available by clicking with the left mouse on the name of the process.
  2. Verify that DREMEL is really responsive:
    • On the DEIMOS dashboard, double-click on the Slitmask box to launch the Slitmask Panel.
    • On the Slitmask Panel, click the dremel button at the lower left to launch the dremel monitor panel.
    • Click the Hello? button and verify that the message Hello! appears at the top of the panel. If not, restart dremel per the instructions on intrcon.
  3. Watch the dremel and bargun logs by issuing the following commands in a polo window: 
    	view_logfile -tail dremel
	view_logfile -tail bargn
    Also, issue the following command on polo to monitor broadcasts from the bar gun dispatcher: 
    	cshow -s deimot bargnget
  4. Have a summit technician scan the barcode on a grating or filter, or scan one of the barcodes on the laminated sheets attached to the instrument.
  5. If the bar gun beeps but no entry appears in the bar gun log, restart the bargun dispatcher as follows:
    • Log into keamano as kics
    • Type the command: 
      	deimos restart dispatcher.bargun
  6. If the bar gun is functional but no entry appears in the dremel log, restart dremel as follows:
    • Log into keamano as kics
    • Type the command: 
      	deimos restart dremel
  7. Repeat the above steps and verify that dremel now recognizes the bar gun signals.

Slitmask scanning error (bad barcodes)

Symptom
The automated Count/Scan operation to get the slitmask barcodes terminates with an error about mask scanning, indicating either that an incorrect number of barcodes were received or that invalid barcodes were scanned.
Problem
The barcode reader incorrectly scanned one or more barcodes.
Solution
Follow these steps to manually fix the barcode list. Note: unless you really know what you're doing, it is best to leave this to a DEIMOS expert.
  1. In the main DEIMOS dashboard, double-click in the brown slitmask box to load the slitmask subpanel.
  2. On the slitmask subpanel, click the dremel button to load the DREMEL subpanel.
  3. On the DREMEL subpanel, click the Emergency/Diagnostic Tools button to launch the dremel Emergency/Diagnostic Tools subpanel.
  4. On the dremel Emergency/Diagnostic Tools subpanel, click the Get barcodes button. The current barcode list will be copied into the text box below that button.
  5. In the text box, enter a comma-separated list of the correct slitmask barcodes starting from slot 2 and continuing through the last occupied slot. Be sure there is a one-to-one correspondence between barcodes and occupied slots (as indicated by the DREMPOPS keyword in the DREMEL subpanel). Do not press the Tell dremel button yet!
  6. On the DREMEL subpanel, verify that the DREMFITS keyword reads 
    	Completed around...
    which indicates that the FITS-writing process has run to completion. If not, then the process may be hung and you have a bigger problem which may require a keamano reboot.
  7. On the dremel Emergency/Diagnostic Tools subpanel, click the Tell dremel button to force regeneration of FITS chunks.
  8. On the dremel sunpanel, click Commit! to save changes.
  9. Wait for the DREMFITS and DREMCMIT keywords to read 
    	Completed around...
  10. Remember that the dashboard must be restarted in order to show the revised list of gratings, filters, and slitmasks.

Review Config shows wrong data

Symptom
When the Review Config (aka tkremel) widget is launched from the DREMEL panel, it does not show the expected changes to the current configurations on filters, gratings, and slitmasks.
Problem
This is a known feature of tkremel with a simple solution.
Solution
  1. Use the Quit button to exit tkremel.
  2. Re-launch tkremel and all should now be well.

Review Config fails to launch

Symptom
Clicking the Review Config button on the DEIMOS does not bring up the tkremel window previewing the new configuration.
Problem
A possible cause is that the resulting configuration is invalid and cannot be displayed, perhaps because the same name appears in two different positions.
Solution
  1. Inspect the file /local/kroot/data/deimot/dyna/dremel.cfg. Check the listings for filters, gratings, and slitmasks looking for identically-named elements in two positions of the same sytem.
  2. If identical entries are found, fix the problem by re-scanning elements in the affected stage.
  3. From the main DEIMOS dashboard, double-click in the slitmask box to bring up the slitmask panel.
  4. Click the dremel button to bring up the DREMEL panel.
  5. Click on Review Config and verify that it runs and displays the new configuration properly.

DREMEL can't access database

Symptom
DREMEL dies while attempting to initialize, or at the completion of a slitmask scan when attempting to read recods from slitmask database. The DREMEL panel on the DEIMOS dashboard indicates an unusual failure on the top status line.
Problem
The summit instrument Sybase server is not responding properly and requires a reboot.
Solution
  1. Use ping waiaha to determine whether waiaha is alive.
  2. Have summit staff determine whether waiaha is functioning properly, and reboot if needed.
  3. Restart DREMEL by logging into keamano as kics and running: 
    	deimos stop dremel
	deimos start dremel

Go to: DEIMOS Home Page - Instruments Home Page - Keck Home Page
Last modified: Tue Nov 25 00:09:22 HST 2008