Images

No light in images

Symptom
Images taken using a grating (spectral or zeroth order image) look like darks.
Problem
The grating is at the wrong tilt, or the grating cover was not removed when the grating was installed.
Solution
  1. Re-home both grating tilts with the following commands on polo:
    unstick_grating 3
    unstick_grating 4
  2. Have summit support personnel check for and, if necessary, remove the cover from the grating.

Images contaminated by red light

Symptom
Images show excess red light
Problem
A red LED inside or outside of the instrument has been left on.
Solution
Send someone up to locate the source of the illumination. One likely place is a red LED on the DEIMOS handpaddle which is lit when the paddle is left in MANUAL mode. Switch it to COMPUTER mode in order to disable the LED. Another thing to check is the plastic housing enclosing the optical encoders on the grating tilt stages, which can be deformed and thus leak light.

Direct images and spectra shifted vertically

Symptom
Spectra and/or direct images taken on the grating appear to be shifted vertically on the CCD images relative to their customary position.
Problem
The grating tilt encoder has lost its positional reference, most likely due to momentary loss of power to the tilt ensoder.
Solution
Re-home both tilt encoders by running the following commands on polo:
unstick_grating 3
unstick_grating 4

Bias jumps in images

Symptom
Images show a significant jump or drop in the bias level in one or more amplifier.
Problem
The CCD readout electronics may be too warm.
Solution
  1. Check DEIMOS temperatures by issuing the temps command on polo.
  2. If electronics temps are too high, then have summit staff remove covers on the electronics as needed to improve airflow.

NaN values in images

Symptom
DEIMOS images show data values as NaN.
Problem
The CCD readout electronics are malfunctioning, perhaps because they are too warm.
Solution
  1. Check DEIMOS temperatures by issuing the temps command on polo.
  2. If electronics temps are too high, then have summit staff remove covers on the electronics as needed to improve airflow.
  3. If data values are bad in only one amplifier, it may be a malfunction on the Leach board. Try increasing the bias level by decreasing the value of the VIDOFFn keyword for the affected amplifier.

NaN or zero values in images only affecting one amplifier type

Symptom
DEIMOS images show data values as NaN or zeros (depending on which version of DS9 is used to open the images). The section of the image affected corresponds only to amplifier A or B.
Problem
The CCD readout electronics are malfunctioning.
Solution
  1. Change the default readout mode. If, for instance, the problem appears in A amplifiers in Direct mode when the default amplifier mode was set to DUAL:A+B, then change the default amplifier mode to SINGLE:B by selecting the corresponding Amp mode on the Detector control panel of the DEIMOS instrument control GUI
  2. Take an image in Direct mode by either hitting the Expose button on the DEIMOS instrument control GUI, or by using the command goi on a polo or deimos terminal to check that the amplifier mode has actually changed.

Stray light or light source detected

Symptom
Stray light detected in images
Problem
It may be that a light source was left on in DEIMOS. An example is when a grating system errors and the error is not cleared because it is not the primary grating in use.
Solution
  1. use checkLamps to see if any of the lamps is currently on.
  2. If a lamp is on, turn it off.
    • For arclamps, you can simply use the Internal lamps → Details... button on the DEIMOS instrument control GUI and then turn off the corresponding lamps.
    • If it is a motor lamp, you will first need to halt the stage and then turn the lamp off. We will use the grating 4 tilt mechanism as an example:
      mmot g4tltmod=halt   # halts the stage
      mmot g4tltfop=off    # turns off the lamp.
  3. Note that the keywords in the output from checkLamps shows what keywords to modify to turn the lamps off.

Focus images with very low counts

Symptom
Focus images show very low counts during the focusloop (R+dome) script.
Problem
The tertiary mirror in not facing the correct Nasmyth focus.
Solution
Contact your Support Astronomer, who will take the proper actions to ensure the tertiary mirror is rotated to face DEIMOS Nasmyth focus.

Computers & Software

FACSUM and compass roses wrong

Symptom
Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.
Problem
The DEIMOS rotator control computer (roto) has lost its DCS connection.
Solution
See corrective actions described below.

Ginga will not load images

Symptom
Newly acquired images do not appear on the the Ginga display GUI
Problem
The Ginga display GUI failed to gather the information to create the DEIMOS mosaic. This can be caused by a temporary communication glitch with keyword services such as dcs.
Solution
Use the ds9_mosaic script to diplay the image:
  1. Display the image on a polo or deimos terminal as follows:
    cdata
    ds9_mosaic image-name
If the problem persists over multiple images, then there might be an issue with the Ginga display GUI, in which case it is recommended to restart it:
  1. Exit the Ginga display GUI by selecting File → Exit
  2. Restart the Ginga display GUI on the VNC background menu: DEIMOS Control Menu → Subcomponents... → Start Ginga display GUI.
If there are missing dcs keywords in multiple images, then it is possible that there is a serious communication issue. In this case it is recommended to call SWOC, although observing can continue as soon as images displayed with the ds9_mosaic script look fine.

Images are not saved to proper directory

Symptom
An output directory is defined (outdir commands returns valid result) but images are not going there.
Problem
The output directory does not exist.
Solution
Create the output directory using the command newday -test.

Images are written incorrectly

Symptom
Images are saved with strange filenames (e.g., d0217_0001.fits.old.PdGWS5) and FRAMENO keyword does not increment between frames.
Problem
Either the ADDFRAME keyword is set incorrectly or the DEIMOS daemons are in a bad state.
Solution
  1. Check the value of the ADDFRAME keyword with gshow -s deiccd addframe.
  2. If the keyword value is false, then set it to true with modify -s deiccd addframe=1

DCS keywords missing in image headers

Symptom
Keywords such as RA, Dec, ROTPPOSN, etc., are not found in the image headers. These keywords can still be obtained from the polo command line via gshow -s dcs commands.
Problem
The watch_ccd process has lost its connection to DCS and needs to be restarted.
Solution
  1. Wait for the exposure to end; do not begin another.
  2. On a polo terminal:
    deimos status watch_ccd
    deimos restart watch_ccd
Newly-acquired images should now have these keywords in the image header. You can verify this by using the command:
fitsheader -e 0 -k rotposn image.fits
If a ROTPOSN value is shown, that's good news. If no output is received, then the keyword is missing from the header and the problem has not been fixed.

Keywords missing from image headers

Symptom
Keywords are found (or reported) to be missing from image headers.
Problem
Communication with a keyword service is interrupted.
Solution
  1. View logfile. In general, if there is a problem with keywords missing from a DEIMOS FITS header, the best place to start is to look at the appropriate watch_ccd log file in /kroot/var/log on polo:
    • If it is a FCS image that is missing one or more keywords in its FITS header, look at the logfile df_watch_ccdlog by selecting the VNC menu item DEIMOS Engineering → Log files... → Log view... → FCS watch_ccd log
    • If it is a science CCD image, look at the logfile ds_watch_ccdlog by selecting the VNC menu item DEIMOS Engineering → Log files... → Log view... → Science watch_ccd log
  2. Locate errors. Use the timestamp of the FITS file to key to the appropriate timestamped messages in the log file. In such cases where keywords are missing from a FITS file, the corresponding watch_ccdlog file should have error messages about either
    • reading the missing keyword, or
    • reading some other keyword from the same service.
  3. If the former: then try to show the missing keyword; e.g., type the following at a polo prompt:
    gshow -s deimot tmirrval
    If that fails, then there is probably a problem either with the dispatcher that serves that keyword or the underlying hardware that that dispatcher controls.
  4. In the latter case: when the FITS file is missing keyword X from service Y but the watch_ccdlog file instead reports an error reading keyword Y from service Y, then the problem may be more indirect. Recall that once watch_ccd gets an error reading any keyword from service Y, it will not attempt to read any more keywords from that service for that image. Thus, an error reading keyword Z will not only cause keyword Z (from service Y) to be missing from that FITS file but any other keywords from that same service which have not yet been collected by watch_ccd for this image will also be missing. In such cases, you should then check the dispatcher that serves keyword Z along with its underlying hardware.

CCD crate will not boot

Symptom
CCD crate is unresponsive. Attempts to reboot via RESET button or power cycling the unit do not succeed (see the CCD powerup procedure). Connecting via telnet to the crate shows the following message printed out every few seconds:
	Timeout waiting for ARP/RARP packet
Problem
The crate is asking for an IP address and is not being given one because the YP service does not know about it.
Solution
  1. Ensure that the appropriate entry for the VME crate appears in the master YP database from which the 'ethers' database is pushed. For the FCS crate, that entry is:
    	00:80:42:0b:4e:8c fcsvmep.keck.hawaii.edu
  2. Change the nsswitch.conf file on keamano so that it resolves hardware ethernet addresses directly from the /etc/ethers file.

No Sounds from Speakers

Symptom
The eventsounds utility does not echo event sounds.
Problem
The soundplay/aplay utility may not be running on your machine, the eventsounds may not be up and running or the deisoundboard process running in deimos. Of course, it might also be the case that your computer's volume is too low.
Solution
  1. Check if the deisoundboard process is running and restart it if needed. On the deimos machine:
    deimos status deisoundboard
    deimos restart deisoundboard
  2. Kill the eventsounds GUI and restart it with the VNC background menu DEIMOS Control Menu → Subcomponents... → Start EventSounds GUI
  3. If the process soundplay is not running in the local machine, open a terminal in the local machine and restart the process with the command soundplay.

Mechanism Moves

Move times out

Symptom
Mechanism move fails with error message "Error code -1: Someone needs to define an error message for this condition"
Problem
A motor move timed out or otherwise failed in a probably innocuous manner.
Solution
Re-try the move.

No light from grating

Symptom
Spectrum appears to be a dark image, although the shutter appeared to open.
Problem
Grating tilt encoder controller lost power thus sending grating to the wrong tilt for imaging, or technician forgot to remove the grating cover when inserting the grating into the instrument.
Solution
  1. Re-home both grating tilts with the following commands on either polo or deimos:
    unstick_grating 3
    unstick_grating 4
  2. Have summit support personnel check for and, if necessary, remove the cover from the grating.

Dewar filter wheel will not rotate

Symptom
Attempts to change the dewar filter wheel to another position fail. The filter wheel is stuck between two positions and cannot reach either of them.
Problem
A filter has not been loaded properly, and the filter holder is protruding from the wheel.
Solution
Have technician remove and inspect the offending filter, then re-install it properly.

Tent mirror not positioning reliably

Symptom
The tent mirror does not position reliably; the piezo controller may not appear to be functioning; FCS may not be tracking well.
Problem
The piezo dispatcher and/or controller operating the tent tent mirror may be in a bad state, possibly due to a power glitch.
Solution
First, restart the piezo dispatcher following the instructions obtained from the intrcon GUI:
  1. Log into keamano as kics
  2. Issue the command
    	deimos restart dispatcher.piezo
If this fails to fix the problem, have summit technician cycle power on the piezo controller as follows:
  1. The tent mirror actuator is driven by a Physik Instrument P-845 Piezo controller. This controller is located in DEIMOS barrel electronics bay #2.
  2. The piezo controller is located in the outboard position (i.e., farthest from the center of the rotation axis) of this bay while the DEIMOS PXL camera electronics unit is located in the inboard position.
  3. To access the piezo controller, rotate DEIMOS so that bay #2 is accessible, then remove the panel from the rear of the bay (i.e., the panel on the side of the electronics ring that faces away from the telescope).
  4. The main power button is the square red pushbutton located in the lower right corner of the piezo controller's front panel. Switch the controller's power off for 30 seconds, then turn it back on.
Note that you can watch the output of the tent mirror control software (dispatcher.piezo) using the following command from the VNC background menu DEIMOS Engineering → Log files... → Log tail... &rarr tmirr.log

Filter move succeeds but filter is "Unknown"

Symptom
Following a filter move, the filter name turns to Unknown. Inspecting the filter wheel position on the DWFIL detail panel shows that the wheel appears to be at the proper position.
Problem
The filter wheel is unbalanced and thus when the filter rotation motor is depowered at the end of the move, the wheel turns enough that its raw encoder position no longer matches the defined position within the tolerance of 1000 units.
Solution
Possible solutions are:
  1. Have summit staff exchange filters in an attempt to balance the wheel.
  2. Have Lick software personnel modify the configuration file to increase the tolerance to an acceptable level.

Loss of air pressure

Symptom
Problem
The pneumatic system on DEIMOS lost pressure, and pressure may still be out.
Solution
  1. If an exposure is in progress, abort it immediately. The loss of air pressure has likely caused the grating to unclamp, and if the exposure continues it will be ruined. If air pressure remains off, the CCD shutter will not close and the exposure will be ruined anyway. But if you are near the end of a long exposure, it is worth trying to salvage it.
  2. On the main Dashboard, open the Tricorder panel and check the barrel and cradle air pressure indicators at OKAY, then a problem exists. Have summit staff investigate the loss of air pressure.
Note that the air pressure is interrupted momentarily once per day around 8:30am by the air water trap purge system. Occasionally, this system may trigger one or more additional purges following at 30 minute intervals if the original purge did not clear the trap.
See also
Notes on air purge

Grating stage will not tilt (timeout)

Symptom
Attempts to change tilt of a grating stage fail. Trying to home the stage using the modify -s deimot G3TLTCAL=1 command yields a timeout error.
Problem
The motor control process for this stage is hung, or the drive belt has snapped.
Solution
  1. Type the command unstick_grating n, where n is the number of the misbehaing grating (3 or 4), in a polo window to return the grating to normal operation.
  2. If this times out, then suspect that the drive belt on the titl stage is broken.

Grating stage will not tilt - Galil controller needs reset

Symptom
Attempts to change tilt of a grating stage fail. Trying to home the stage using the modify -s deimot G3TLTCAL=1 command yields a timeout error.
Problem
The motor control process for this stage is hung.
Solution
  1. Type the command unstick_grating n, where n is the number of the misbehaing grating (3 or 4), in a polo window to return the grating to normal operation.
  2. If this fails several times, then you can try the Galil controller reset procedure.

Grating will not go To Zero

Symptom
Attempts to change tilt of a grating stage to zeroth order imaging position fail. Issuing the gozero command results in an error indicating that the move would exceed the allowable software range.
Problem
Accumulated tilt corrections are too large.
Solution
  1. Check the value of the grating tilt offsets by issuing the command
    smot gXtltoff
    where X is the grating numger (3 or 4). Typically these parameters are 100 or less.
  2. If the offset is large, (i.e., absolute value greater than 100), then reset the tilt offset and recalibrate the tilt on the stage by issuing the command;
    gratecal
  3. Re-try the To Zero operation.

Gratings will not move

Symptom
Attempts to move the grating fail. Both gratings will not tilt. A review of the grating detail panel on the DEIMOS control GUI reveals that there is an "ERROR: Controller CTRL2 is in MANUAL control mode." The rotator control GUI shows a message indicating that a panel is open.
Problem
The grating sub system is lock out because the grating hatch is open. It may have popped open when the instrument was rotated.
Solution
Have the summit crew rotate deimos to the grating load position and inspect the hatch. They may need to open and close the hatch again to get it to properly seat with the switches set. Then retry the grating moves.

Grating tilt in reverse limit

Symptom
Attempts to move the grating tilt stage fail. The deimot.G3TLTLIM or deimot.G4TLTLIM keywords indicate motor in secondary limit. The deimot.G3TLTMSG or deimot.G4TLTMSG keywords indicate that one of the grating tilt stages is at a limit.
Problem
The tilt stage has exceeded the allowed range of travel and has hit the secondary limit switch. This cuts power to the motor and prevents further motor moves.
Solution
It is necessary for summit personnel to assist in moving the stage out of a limit. This operation presents a physical hazard to the operator because the stage could move suddenly when it is backed off the limit switch. Accordingly, this task should only be performed by a qualified instrument technician!
  1. Send the grating to the unload position, either by opening the grating access hatch and pressing the appropriate button, or by issuing the command
    	unload_grating 3
    to send slider 3 to the unload position, or
    	unload_grating 4
    to send slider 4 to the unload position. The technician can then access the stage through the grating changeout hatch.
  2. Open the grating changeout hatch. De-power the motor by unplugging the black plastic connector carrying DC power to the motor.
  3. Manually rotate the wheel on the friction drive mechanism to move the stage in the appropriate direction. An indication of the positive direction should be written on the wheel. If the grating tilt stage is in the negative secondary limit, you will need to move in the positive direction, and vice versa. You can monitor the current position using the G3TLTRAW or G4TLTRAW keywords to verify that the stage is moved in the proper direction. The technician may be able to detect the "click" of the mechanical limit switch when the stage moves off the limit.
  4. With hands clear of the tilt stage, restore power the motor by plugging in the black plastic connector carrying DC power to the motor. BEWARE! The stage may begin a tilt move when power is restored.
  5. When the stage is returned to a working state, re-home the tilt stage by clicking on either G3 tilt or G4 tilt on the Init/Stow panel on the DEIMOS control GUI.

Grating tilt in "bogus" secondary limit

Symptom
Grating stage(s) fails to move, reporting a secondary tilt limit. However, the grating detail panel on the DEIMOS control GUI indicates that the stage is not close to a tilt limit.
Problem
The secondary limit switch is falsely triggered.
Solution
Ensure via visual inspection that the stage is not in a limit. If needed, manually tilt the stage via the drive wheel to move the tilt for verification. This can be done by sending the affected grating to the unload position (e.g., modify -s deimot gratepos=-3 for the grating 3 unload position) and accessing it through the grating access hatch at the side of the instrument. If the limit switch is giving a bogus error, you can jumper out the secondary limit switch as follows:
  1. Pull back DEIMOS from the science position.
  2. Remove front cladding to access the grating slide area.
  3. Locate the slider in question.
  4. Locate the wiring interconnect panel on teh grating slider.
  5. Disconnect the appropriate wires (detailed description TBD.

Grating will not unclamp (clamps won't release)

Symptom
Repeated attempts to unclamp the current DEIMOS grating fail.
Problem
The pneumatic clamps holding the grating will not release.
Solution
  1. Try using the unclamp_grating command to unclamp the grating. By default, this will try 25 times to unclamp. If no success after 25 tries, go to next step.
  2. Have summit personnel disconect the air supply line at the rear of the instrument via this procedure. This should remove the air pressure and allow the clamps to release.

Grating will not unclamp (pin still in)

Symptom
Repeated attempts to unclamp the current DEIMOS grating fail.
Problem
The grating pin remains in even after all clamps are released.
Solution
This is a tricky condition, since with the pin in you can't clamp up, nor can you unclamp. Here's one thing that has worked.
  1. Rotate DEIMOS to ROTATVAL=-90°.
  2. Follow the instructions on the grating clamps page to do the following:
    • close clamps 1, 2, 4, and 5
    • open clamps 5, 4, 2, and 1
    then re-start the motor control software.
This should release the pin.

Grating will not move to unload position

Symptom
The grating will not move to the unload position, because the select stage hits a limit and the tilt move does not execute. The keyword deimot.GRSELORD=-3 as desired, but deimot.GRSELLIM shows a primary limit.
Problem
The move cannot complete due to the limit switch striker being out of position.
Solution
Use keyword-level commands to move the tilt to the unload position as described in this procedure.

Grating select move fails (pin not in)

Symptom
The grating fails to move into position. In the grating detail panel on the DEIMOS control GUI, the message reads:
 Can't set GRATEPOS=4, grating locating pin not in
And the pin is out on the grating clamps status panel.
Problem
We are at an unfavorable rotator postion where the gravity load prevents the grating from moving into position.
Solution
If you are trying to move to slider 4, execute the command
clampup_slider4
to attempt to clamp up. Otherwise, simply re-try the move from the dashboard GUI. If the pin will not engage, then rotate to a PA that is 90 degrees away from this location and retry the grating move. If this fails, rotate another 90 and try again. If it is still failing, there may be additional problems.

Grating select move fails (positioning error)

Symptom
Grating move fails with error:
Can't set GRATEPOS=3, motor shut off - positioning error.
Problem
The move failed, most likely due to excessive vibration in the lead screw generating an overtorque condition in the controller. This can happen in the vicinity of the slider 3 load position at certain DEIMOS position angles.
Solution
  1. Rotate DEIMOS to the position at which the grating system is horizontal. This corresponds to a Rotator angle (deirotg.ROTATVAL) of -90 or +90 on the DEIMOS rotator GUI.
  2. Home the grating select stage using the command
    modify -d deimot GRSELCAL=1
    or alternatively use the Initialize panel on the dashboard GUI and click the Grating selectbutton.
  3. If the calibration succeeds, then re-try the original move. If it fails, suspect a mechanical problem.
  4. If the calibration does not succed, try to reset the Galil controller.

Grating select move fails - Galil controller needs reset

Symptom
Grating move fails.
Problem
The Galil controller may need a reset.
Solution
Try the Galil controller reset procedure.

Timeout error after executing Recalibrate all

Symptom
The button Recalibrate All on the Init/Stow control GUI fails to home most of the mechanisms.
Problem
The mechanisms recalibration procedure failed most likely because too much air pressure was built in the DEIMOS pneumatic system.
Solution
Cycle the shutter mechanism several times to ensure that some of the air pressure is released. To do this bring up an xterm on polo as user and run the command cycle_shutter -100. This will keep trying to cycle the shutter until it finally opens and closes.

Troubleshooting motors

See Motor troubleshooting

Slitmask: select move fails with motor powered off error

Symptom
Slitmask move fails; SLMSKMSG and/or SLSELMSG keywords report that move failed and motor is powered off.
Problem
The motor move has failed due to a bad reading on a switch, usually either the one on the slitmask arm or on the door comb.
Solution
Based on one successful recovery from this error, the following procedure is suggested:
  1. Rotate DEIMOS to the slitmask unload position (ROTATVAL=-90) and try the move again. If this works, then quit.
  2. Attempt to calibrate (home) the stage (mmot slselcal=1). This may succeed, in which case you can quit.
  3. Have summit support attempt to move the slitmask selector motor manually.
  4. Attempt to calibrate the stage.

Slitmask: comb missaligned killed motor power

Symptom
The following are symptoms of a comb missalgnment:
Problem
Two possibilities exist.
  1. A slitmask is sticking on the mask pusher mechanism, triggering the comb sensor. The slitmask system cannot be moved until the mask is freed from the pusher bar.
  2. The comb was bumped in the process of moving a slitmask into beam. The comb needs to be re-aligned.
Solution

Slitmask move fails - slitmask jammed

Symptom
Attempting a slitmask move fails.
Problem
Slitmask may be jammed.
Solution
See procedure for dealing with jammed slitmask.

Slitmask move fails - fiducial hole blocked

Symptom
Attempting a slitmask move fails, but inspection of the masks shows that the masks are not jammed.
Problem
A slitmask may be missing its fiducial hole.
Solution

Slitmask move fails - mask position is read both at beginning and end

Symptom
Attempting a slitmask move fails, error says that slitmask position is detected both at beginning and end)
Problem
Slitmask selector might be jammed, or might have lost its calibration.
Solution

Slitmask move fails - Galil controller needs reset

Symptom
Attempting a slitmask move fails and you have already checked that no mask is jammed
Attemps to recalibrate the mechanism fail.
Problem
The Galil controller may be failing and needs to be reset.
Solution
Please, follow the Galil controller reset procedure.

Electronics

Motor controller fault

Symptom
The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running.
Problem
The motor controller has faulted.
Solution
Attempt to reset the motor controller using the Galil controller reset procedure.

Motor controller hung

Symptom
The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running.
Problem
The motor controller has hung.
Solution
Attempt to reset the motor controller using the Galil controller reset procedure
If this fails, proceed to the next item.

Problem communicating with Galil controller

Symptom
Although the Lantronix terminal server is pingable, attempts to communicate with the Galil controller fail with this error:
  [1009] kics@keamano% telnet 192.168.6.4 3005
	Trying 192.168.6.4...
	telnet: Unable to connect to remote host: Connection
	refused 
Additionally, the command ctx (executed on polo) shows an error message in the dispatcher2 prcesses listed under "Current KEAMANO tasks". If there is a dispatcher2 process running, then the connection error message shown by the telnet command is normal, because the dispatcher is already communicating to the Galil controller through the same port you are trying to use.
Problem
The port on the Lantronix unit connected to the Galil controller is hung.
Solution
Follow the Lantronix port reset procedure.

Problem communicating with Lantronix unit

Symptom
Motor moves fail. Running testAll indicates a problem communicating to the Lantronix one of the Lantronix units (deits2 or deits3).
Problem
The most likely problems are:
Solution

Ion pump is off (power glitch)

Symptom
The testAll command reports that one or both of the ion pumps is off. This is very likely to happen after a power glitch or power outage at the summit.
Problem
The ion pumps need to be turned on.
Solution
State the solution to the problem here, numbering the steps to be taken:
  1. Go to the DEIMOS control GUI, click the Tricorder label followed by the Details... button to launch the Tricorder GUI.
  2. Verify that the status of the LN2 can ion pump #1 power (deipower.OUTLET_A1) and/or Detector vessel ion pump #2 power (deipower.OUTLET_A2) is Off. Also check the dewar temp; if the dewar temp is well above -100 C then you may not be able to start the ion pump.
  3. Click the Restart button under the appropriate ion pump to attempt to restart the ion pump. The status should change from Off to On.
  4. If the startup is successful then the status will eventually change to On. If the startup fails, the status will return to Off, and you should try again to start the ion pump. Note that ion pumps are restarted at the first attempt since we upgraded the controllers on March 28, 2022.
Alternatively you can also try to restart any of the ion pumps by modifying the keywords deipower.OUTLET_A1 (LN2 can) and/or deipower.OUTLET_A2 (detector vessel): Double check the pump status for a few minutes after restarting to make sure that both are on.

Rotator

Rotator won't go into DCS Control Mode

Symptom
Rotator will not go into DCS control mode when hitting the Enable DCS button on the Sky mode panel on the rotator control GUI.
Problem
DEIMOS is not the selected instrument on the TCS GUI and/on the telescope tertiary mirror is on at RNAS
Solution
Ask the observing assistant to reselect DEIMOS on the TCS GUI

Rotator reporting wrong angle

Symptom
Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.
Problem
The DEIMOS rotator control computer (roto) has lost its DCS connection, probably due to a communication issue with the DCS gateway.
Solution
Ask the OA to follow these steps:
  1. Check the updates on the DCS gateway terminal. Odds are that there are no DCS updates.
  2. Restart all gateways.
  3. Init the rotator on the TCS GUI. IMPORTANT: No need to Shutdown/Init, only Init is needed.

Rotator keywords arriving out of order

Symptom
The rotator stops following the pointing demands from DCS. FACSUM reports incorrect drive angle on the rotator.
Problem
The timestamp and demand keywords are not always in the correct order (rotdets-->rotpdsts-->rotdets2).
Solution
Ask the OA to:
  1. Restart the Linux DCS gateway.
  2. Initialize the rotator on the TCS GUI.

Can't find stars

Symptom
At the beginning of the night, pointing appears to be way off; stars not visible in guider.
Problem
Either the rotator has not been initialized and placed into DCS mode, or the TCS does not know the rotator position.
Solution
  1. Run testAll to verify that the instrument and telescope settings are correct (e.g., selected instrument is DEIMOS).
  2. If the rotator was not initialized at the start of the night, then follow these instructions to do so.
  3. If you have not yet commanded a rotator move, have the OA send the rotator to any angle (e.g., 0°).
  4. Verify on FACSUM that the rotator is in position angle mode.

Rotator spontaneously homes itself

Symptom
When sending a rotator move, the rotator unexpectedly executes a large slew to the home position. (Need to check if this would be the actual behavior with the new Galil controller.)
Problem
The rotator software determined that the discrepancy between the readings on its two absolute encoders was too large and decided to home itself in order to reset the encoders before proceeding.
Solution
Allow the move to complete, then proceed with observing. Note that properly initializing the rotator (Shutdown/Init on the TCS GUI) at the start of each night should generally prevent this from happening.

Can't do Shutdown/Init on rotator

Symptom
At the start of the night, the OA cannot perform the Shutdown/Init operation on the rotator.
Problem
The rotator software on roto is not communicating properly with DCS.
Solution
  1. Apply the same solution as above
  2. If the above does not fix it, restart the rotator dispatcher as follows:
    • Open a roto terminal (DEIMOS Engineering → xterm@roto)
    • On the roto terminal type:
      deimos restart deirotg

Rotator moves are all outside of limits

Symptom
When the OA attempts to send a rotator move, SKY complains that the requested angle is out of range for the DEIMOS rotator although it is clearly a legal request.
Problem
The rotator limits are not properly defined.
Solution
  1. Verify the problem by displaying the keyword values:
    show -s dcs rotccwlm rotcwlm
    The correct values are:
    rotccwlm = -330.00 deg
    rotcwlm = 402.00 deg
  2. If you get zero values back, have the OA re-load DEIMOS as the current instrument.

Discrepancy between the rotator status reported by DEIMOS and TCS.

Symptom
Problem
The rotator computer (roto) has lost communications with the DCS gateway.
Solution
Ask the OA to follow these steps:
  1. Check the updates on the DCS gateway terminal. Odds are that there are no DCS updates.
  2. Restart all gateways.
  3. Init the rotator on the TCS GUI. IMPORTANT: No need to Shutdown/Init, only Init is needed.
This will send the rotator back to where it was. It will be a good idea to do an alignment check if the telescope was aligned on a mask when this issue happened.

Rotator fails to move under DCS control.

Symptom
Problem
The DCS gateway has died. This is a different manifestation of the issue described on discrepancy between the rotator status reported by DEIMOS and TCS.
Solution
Apply the same solution as on the entry on discrepancy between the rotator status reported by DEIMOS and TCS.

Rotator disabled due to ESTOP or power outage

Symptom
Rotator will not move or initialize. The yellow message at the bottom of the DEIMOS rotator GUI indicates ESTOP from DCS.
Problem
The rotator is disabled because the Keck II facility ESTOP circuit was de-activated, either because someone pressed a facility ESTOP button or because of a power outage.
Solution
  1. Follow the DEIMOS rotator controller reset procedure.
  2. Verify that the ESTOP from DCS message on the DEIMOS rotator GUI is gone.

Rotator computer bypass activated

Symptom
The Rotator control GUI shows a yellow message indicating Computer/Manual bypass activated.
Problem
The rotator toggle bypass switch is in the BYPASSED position.
Solution
Have the summit crew change the toggle bypass switch to NORMAL (see Figure 2 in manual rotation procedure).

Rotator does not move after having restared deirotg

Symptom
Problem
The rotator controller is in a bad state, probably after having been power cycled.
Solution
The rotator controller needs to be reset. This can be done remotely as follows:
  1. As any DEIMOS numbered account on roto: deimos stop deirotg.
  2. As any DEIMOS numbered account on roto, keamano, or polo: telnet 192.168.6.20
  3. Hit Enter a couple times until the controller echoes back a : (colon) prompt.
  4. Type RS and hit Enter. The controller will echo back a : colon prompt after resetting.
  5. Terminate the telnet connection (ctrl + ] followed by quit).
  6. As any DEIMOS numbered account at roto: deimos start deirotg
  7. The rotator will require homing due to the controller reset.

Flexure Compensation System

No light on FCS images

Symptom
No light is visible in an FCS exposure
Problem
Probably one of the following:
  1. CCD shutter is not open
  2. I or Z-band filter is in use
  3. CuAr lamp burned out
  4. Grating cover left on
  5. Grating is tilted side-to-side
Solution
Follow these steps to recover:
  1. Verify that the CCD shutter is open.
  2. If the I or Z band filter is in use, ensure that the CuAr lamp is turned on; the LED does not emit at these wavelengths.
  3. If still no light, try switching to the LED lamp on the FCS GUI.
  4. If still no light, acquire a spectrum with the science array and check whether any light is received. If not, send someone up to check whether the grating cover has mistakenly been left in place.
  5. If still no light, check whether the science spectra appear shifted to the left or right relative to the expected positions based on the slit layout. If so, the grating may not be properly seated.

FCS cannot track (offset too large)

Symptom
FCS image shows light but software cannot lock on to image and complains that the offset is too large.
Problem
The grating tilt is not correct and the image is shifted.
Solution
  1. Verify that central wavelength and filter are set to exactly the correct values for FCS. If they are not, then the FCS GUI will show Lockout status.
  2. Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.

FCS cannot track (xcorr failed)

Symptom
FCS image shows light but software reports X-correlation failed. Image contrast too low or grating/slider misregistered errors.
Problem
Potential causes are:
  1. The grating tilt encoder has lost its position so the actual position does not match the reported position.
  2. Either the FCS reference image or the new FCS image contains a substantial cosmic ray.
  3. Grating became unclamped due to a loss in DEIMOS air pressure.
  4. Grating was not properly installed in its mount by summit technician.
Solution
  1. Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.
  2. Re-home the grating tilt mechanism by either clicking on the G3 tilt or G4 tilt button on the Init/Stow GUI, or by typing "mmot g#tltcal=1" at a polo prompt, and where "#" is either "3" or "4".
  3. Inspect the FCS reference image for cosmic rays and if needed, re-take the reference image.
  4. Unclamp and re-clamp the grating via the dashboard.
  5. Have summit staff remove and re-install the suspect grating to ensure that it is seated properly in the slider.

FCS won't permit "match previous" option

Symptom
You want to match against a previous observation on the FCS GUI Reference control panel, but no reference is taken.
Problem
There are no saved FCS setups which match the current setup.
Solution

FCS attempt to match previous fails

Symptom
An attempt to use the "Match Previous" feature of FCS fails with the error Dewar X translation stage hit a limit.
Problem
The grating is either improperly seated now, or was improperly seated when the original FCS reference exposure was taken. Alternatively, changes to the grating cell, grating system, or grating parameters (e.g., GRATOFFx) may cause the grating to clamp up in a slightly different position and thus shift the images. See also next item.
Solution
  1. Compare Y positions of the same reference spots on the original FCS reference image and the new reference image. Verify that they are separated significantly in Y. If not, then the problem is elsewhere.
  2. Have summit staff unload and re-install the grating.
  3. Re-try the "Match Previous" operation.
  4. If "Match Previous" continues to fail, then some change in the grating system may be the cause. In this case, matching previous may not be possible.

FCS cannot match previous

Symptom
When attempting to use the "Match Previous" function on the Reference control panel of the FCS GUI to match to the previous night's FCS reference image, the attempt fails with an error unable to match. Comparing the current FCS image to the previous reference image shows a large spatial offset.
Problem
The grating tilt encoder has slipped and the new image is not properly registered.
Solution
  1. Recalibrate the grating tilt by typing the following command on a polo or vm-deimos (local) terminal:
    modify -s deimot g3tltcal=1
    Note: if using slider 4, specify g4tltcal=1 instead.
  2. Re-try the match previous operation.
  3. If FCS still cannot match, use the fcsmov command to steer the current image so that it matches the previous reference image.

FCS ping-pongs in X

Symptom
FCS issues moves but does not converge on the target, which keeps bouncing back and forth in the X direction.
Problem
The piezo controller/actuator is not functioning properly.
Solution
  1. Inspect cabling at the FCS controller (located in DEIMOS barrel electronics bay #2), particularly the high voltage line coming out the center of the controller.
  2. If cabling looks fine, check that the FCS controller channel is working properly by using one of the other channels on the controller (consult Lick first).

FCSTRACK script dies on startup

Symptom
When attempting to run fcstrack, the script dies with an Subscript out of range. error.
Problem
The script attempted to read an undefined keyword.
Solution
If fcstrack does not restart automatically, then restart it from the VNC menu (DEIMOS Control Menu.. → Subcomponents... → Start fcstrack.)

FCS Lockout (cannot track due to focus mismatch)

Symptom
FCS refuses to lock in due to a lockout condition, which is claimed to be a focus mismatch although the focus is correct.
Problem
The focus tolerance keywords are not initialized properly.
Solution
  1. Check the keyword values via:
    	show -s deifcs fcsfoto1 fcsfoto2
  2. If they are zero, on any polo or vm-deimos (local) terminal, type
    	modify -s deifcs fcsfoto1=200 fcsfoto2=600

FCS Lockout (cannot track due to wavelength mismatch)

Symptom
FCS remains in lockout condition even when the grating, central wavelength, filter and focus values all look correct. It complains that there is no setup for the current configuration. Beware especially after a mask alignment.
Problem
Usually, the central wavelength of the grating is slightly off from reference value. There is finite resolution in the tilt mechanism and sometimes the "Go Back" function of the grating does not set the exact wavelength.
Solution
Move off the central wavelength, then back again.
  1. On the Mechanism control panel of the DEIMOS control, change the Wavelength field by 10 Angstrom.
  2. Return to your desired central wavelength.
This should solve the problem.

FCS CCD failure

Symptom
FCS cannot track. Looking at FCS images reveals that one of the two FCS CCDs is not reading out (i.e., half of the FCS image displayed on FIGDISP contains all zero-valued pixels), although the other CCD looks normal.
Problem
The signal chain for one of the FCS CCDs is interrupted.
Solution

WARNING Do not attempt to power cycle or reboot the FCS system. If you do, the FCS system can get caught in a catch 22. Without a proper cable connection you can't boot, but to test the connection you need to boot and take an image. Yikes. Trouble shoot only during the day, and an expert technician at the summit will need to check and repair the cabling to make a permanent repair (see [KD2-19923].)

As a temporary measure, as long as one of the CCDs is reading out, the Staff Astronomer can enable partial FCS compensation by telling the FCS software to use only the one working CCD. To do this:

  1. Edit the following file as dmosbld at polo:
    /kroot/rel/defult/bin/fcstrack
    Under normal operation (both CCDs working), line 63 of this script reads:
    set offline = 0
    If CCD 1 (left-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read:
    set offline = 1
    If CCD 2 (right-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read:
    set offline = 2
  2. Kill and restart the fcstrack script. You should get a warning message:
    *** FCS CCD #1 has been marked offline ***
    Verify that FCS is now able to track.

FCS dewar stage at negative (or positive) limit

Symptom
FCS complains of a lockout situation because the dewar stage is at a negative/positive limit. FCS cannot re-position the dewar stage to correct for differences between the FCS reference image and the current FCS image.
Problem
One problem may be that the grating is improperly clamped either currently or when the reference image was acquired in the afternoon. If the grating is improperly clamped, the result could be large shifts between FCS images.
Solution
  1. Try simply unclamping and reclamping the grating. If it does not work the first time, try it once more.
  2. If the problem was that the grating was inserted during the nigth either without using Select Slider or while the rotator was not in the correct orientation for the corresponding Select Slider command, the fastest solution is to:
    • Have the OA to record the current position and to stop guiding.
    • Run Select Slider 3|4. This should insert the grating using the correct rotator angle.
    • Wait for FCS to track. FCS might take a couple of minutes to track, because the dewar stage has to move away from its limit.
    • Once FCS is tracking, have the OA to start guiding again.
    • Check that your target is correctly aligned.
  3. If you are not certain of when the grating was improperly campled and the previous options did not work, rotate the instrument 90 degrees from the current position or to -30 deg (about the rotator value set by FCS setup in the afternoon). Then try unclamping and reclamping the grating. If this is done during the night while the instrument is guiding, it will require coordination with the OA, since guiding should be stopped.
  4. If none of the previous options work, the current position of the grating is likely not at fault. It may have been that the grating was improperly clamped when the FCS reference images were acquired. If this is indeed the problem, then an FCS image will need to be retaken as a reference. You will need to unlock the rotator and dis-allow DCS control of the rotator. Next, reacquire FCS images following the afternoon procedures. When finished acquiring new FCS images, you will need to put the rotator back into DCS control and have the OA do a standby-init on the rotator.

FCS fails to cross correlate images

Symptom
FCS images are flipped left/right or the spectrum appear shifted left/right relative to the FCS reference image. When FCS is tracking, large cross correlation errors are reported by the FCS error handler.
Problem
This problem is attributed to attenuated signal in the fiber optic digital communications link between the FCS CCD controller (inside DEIMOS) and FCS CCD VME crate (in the Keck-2 computer room).
Solution
This problem may not be solved at night. A technician will need to clean the FCS fibers and reseat the boards during the day.

FCS status constantly toggling between Emergency and Tracking

Symptom
One or more of the following conditions are given:
  1. FCS keeps toggling between Emergency and Tracking.
  2. The FCS log keeps issuing errors about a stage having reached a limit.
  3. The FCS cross correlation is stuck on a secondary maximum (see this cross-correlation plot as an example.)
Problem
One of the FCS stages has reached a limit, the spots on the current FCS image are offset by more than 25pix with respect to the reference image and FCS is tracking on a false broad maximum. This issue requires action as soon as possible.
Solution
Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.

FCS status constantly toggling between Warning and Tracking

Symptom
One or more of the following conditions are given:
  1. FCS keeps toggling between Warning and Tracking.
  2. The FCS log keeps issuing warnings such as:
    1. WARNING: dwxl8trg= is within 100 counts of its maximum range (-2300, 700).
    2. WARNING: gntltoff is larger than the recommended range (-100, 100).
    3. WARNING: tmirrtrg is within 490 counts of its maximum range (10, 9400).
  3. The FCS cross correlation looks good (see this cross-correlation plot as an example.)
Problem
One of the FCS stages is close to its limit, but it has not actually reached the limit yet.
Solution
This does not require immediate action as soon as the spots on the current FCS image are reasonably well aligned with the spots on the FCS reference image. You may want to silence the FcsAlarm sound on the EventSounds GUI.

FCS cannot save images. FCS CCD crate crashes

Symptom
When taking reference image or starting FCS tracking, the ccd reads out but no image is saved and a 30 seconds timeout warning appears. The FCS ccd crate become unresponsive.
Problem
Wrong directory permissions: the account running the fcstrack script does not have write permission to the current output directory.
Solution
Compare the group membership and the write permission of the current outdir with the other numbered (user) accounts. Have the SA check the proper group memberships, and if neither you nor the SA can change them, request the help of the software on call person. Unfortunately, this problem also crashes the FCS CCD crate which needs to be rebooted and restarted (see the FCS CCD powerup procedure).

FCS saves images with wrong name

Symptom
When taking reference image or starting FCS tracking, the CCD reads out and displays an image on the FCS GUI, but the image is saved under the wrong name (e.g., fcs0504_2057.fits.old.rNAkGG)
Problem
The FCS watch_ccd daemon is in a bad state.
Solution
Restart the watch_ccd program.

FCS power is disabled

Symptom
The testAll script indicates that power to the the FCS CCD system is disabled:
  Checking FCS CCD 15V Power.......ERROR! Current value 'disabled' should be 'enabled'
  Checking FCS CCD 30V Power.......ERROR! Current value 'disabled' should be 'enabled'
Problem
Although the power may be off, the more likely issue is that the FCS CCD system was reading out at the time you ran testAll
Solution
Re-run testAll and check whether the system still reports an error. If necessary, stop the FCS system by issuing the command
mfcs fcsmode=off
and then re-run testAll to check status.

TV Guider

Guider is not responsive

Symptom
Guider cannot be selected from MAGIQ GUI.
Problem
Guider is powered off or requires a power cycle.
Solution
Cycle power to the guider off and on by executing the following command on polo:
tvpower cycle
This should take about one minute to complete.

Bad guider images

Symptom
Guider can be selected on the MAGIQ GUI, but images from the guider do not show stars or only a portion of the field (top 1/3) show stars.
Problem
Either the guider is not properly configured or the shutter is broken.
Solution
Follow these steps to determine source of problem:
  1. Verify that the DEIMOS hatch is open from the main DEIMOS control GUI.
  2. On a polo terminal, type tvfilter to verify that the TV filter is set to a reasonable value (e.g., R). If it is set to Home, select R.
  3. On a polo terminal, type tvfocus to verify that the TV focus is set to a reasonable value for the selected filter (e.g., 970 for R). Note that 1 is not a reasonable value.
  4. Note that the default value for the guider filter and focus is R and 970. If you suspect that the defaults should be in use, reselect these values.
  5. If still no good image, close the hatch, turn on the Qz lamp, and see whether any light is seen on the image.
  6. If no light is seen, the guider shutter may be inoperative. Replacing this is an extensive job for the day crew.

Bad guider images

Symptom
Light is seen on the guider images, but the images appear "smeared" horizontally.
Problem
The serial clock card in the guider controller is malfunctioning.
Solution
Repairing this is a full day's work for a talented daycrew. Have daytime summit staff pull DEIMOS, remove access panels on front of instrument, and check serial card in the guider head.

Poor guiding in vertical angle mode

Symptom
When offset guiding in vertical angle mode, the guide star does not move, but the target drifts off the slit.
Problem
The wrong guiding mode has been selected, so the center of rotation is set to the guide star rather than the slit.
Solution
Please see table on guiding modes. This is a case of offset guiding. Center of rotation needed to be set and then start guiding with "On Star" button.

AUTOFOC and Mira

AUTOFOC returns abnormal telescope focus value

Symptom
After running AUTOFOC to focus the telescope, the secondary mirror position is more than a few tenths of a mm away from the nominal value of 0.8 mm.
Problem
The TV camera focus is not set correctly.
Solution
  1. Check the TV camera focus value by typing tvfocus on a polo terminal, which should be around 1000 units.
  2. Change this value as needed by typing tvfocus NEW_VALUE on a polo terminal.
  3. Re-run AUTOFOC and verify that the value has changed.

MIRA images are badly out of focus

Symptom
When running MIRA, the images are badly out of focus.
Problem
The telescope is badly out of focus, possibly because the DEIMOS TV camera focus is not set correctly.
Solution
See the description for AUTOFOC returns abnormal telescope focus value.

AUTOFOC crashes (OBSOLETE)

Symptom
Autofoc crashes with the following error message:
can't connect autofoc to aut_srv_task
Problem
The wrong version of AUTOFOC was run. There is a SunOS version for use with the SunOS guider, and a Solaris version for use with the Solaris guider.
Solution
Run the correct version of AUTOFOC.

MIRA fails to reset parameters

Symptom
Data directory, filename prefix, etc., are not properly reset after running MIRA.
Problem
The MIRA script failed to complete.
Solution
Have the OA or SA restore parameters as follows:
  1. Open a polo terminal
  2. Execute the command:
    restore_state -verify
    This should restore the pre-MIRA settings.

MIRA image shows gradient

Symptom
MIRA images show a steep top-to-bottom gradient in the brightness of the segments.
Problem
This is not generally a problem, just a benign symptom of using the 900-l/mm grating.
Solution
As long as all segments can be identified and MIRA completes without error, no additional action is required.

Slitmask Alignment

IRAF xbox Fails (Image not Written)

Symptom
When xbox is run, it fails before analyzing the first box, with an error akin to
extension BluSlits not found
Subsequent attempts to analyze this same image yield the same error.
Problem
The xbox program attempted to read the image before it was fully written.
Solution
  1. Type flpr and re-run xbox.
  2. If that fails, restart IRAF and re-run xbox.
  3. If that fails, inspect the image header (execute the command fitsheader -e BluSlits filename on polo) and verify that the specified extension exists in the image header.

IRAF xbox Fails (Missing FITS Chunks)

Symptom
When xbox is run, it fails before analyzing the first box, with an error akin to
extension DESIslits not found
Subsequent attempts to analyze this same image yield the same error.
Problem
The FITS header extension called DesiSlits is not being read by the software, either because it does not exist in the image header or because IRAF can't read it.
Solution
If you are trying to get a slitmask aligned and are pressed for time, proceed straight to the procedure for aligning slitmasks without FITS chunks. Otherwise, you can proceed as follows: If FITS chunks are missing, it is likely that something went wrong in the process of scanning the masks. Follow the procedure for dealing with a mask scan error, then take these additional steps:

IRAF xbox Fails (Bad box)

Symptom
When xbox is run, it crashes with a floating point error when it tries to analyze a certain box.
Problem
A truncated box (i.e., one that is at the edge of the CCD) may cause this to occur. The mask must be re-analyzed with the offending box excluded.
Solution
  1. Identfy the bad box by displaying it with the ds9_mosaic. Note the PANE corodinates of the bad box.
  2. In IRAF, run the command get_boxes image, where image is the name of an alignment image, to generate a "boxfile" (text file) listing the boxes (generally named box.maskname).
  3. Edit the boxfile. Locate the line corresponding to the bad box by comparing the PANE coordinates. Delete the line and save the file.
  4. In IRAF, run xbox from the command line using the box file as input:
    	xbox image input=boxfile

IRAF xbox reports STAR SKIPPED

Symptom
When xbox is run, it beeps and complains STAR SKIPPED when attempting to locate certain stars.
Problem
The alignment boxes were not found at the expected locations in the alignment image, probably due to either uncorrected flexure or changing to a different grating.
Solution
  1. In IRAF, execute the command
    	tune_qmodel last
    to adjust the pointing model so that the guide boxes are found. Note that after you mark the location of the alignment box, the program will invoke xbox in practice mode.
  2. Re-run do_xbox to compute and send telescope moves.

SAT fails when identifying the mask boxes

Symptom
During the afternoon checklist, SAT gets stuck when trying to display one of the user-defined slitmasks. The IDL window (slit_align) shows an error message and its prompt is active. Note that the IDL window (slit_align) is minimzed in the lower region of the VNC window, hidden behind the main SAT GUI. To bring up the slit_align window, double-click on it when it is minimized once you have shifted away the main SAT GUI.
Problem
There is some problem with the mask geometry, which does not allow boxes identification using the information in the image header.
Solution
Follow this procedure to identify the alignment boxes manually:

IMPORTANT NOTE: During the night, when doing the on-sky coarse alignment for this mask, make sure before starting the alignment, the Box file format option in the Options tab is set to maskname.box, but only for this mask. The Box file format option should be fits.extn for any other mask whose boxes were identified automatically rather than manually.

SAT fails during fine alignment

Symptom
Once the coarse alignment has been completed successfully, SAT crashes during the fine alignment procedure. The IDL window (slit_align) shows the following type of error message:
      % GET_BOX_OFFSETS_ITER: starting iteration 1
      input coords for box        0    3322    1176
      input section for box        0    3222    3422    1076    1276
      output section for box        0       0     200       0     200
      [...]
      output section for box        5       0     200       0     200
      input coords for box        6    7355    1023
      input section for box        6    7255    7455     923    1123
      output section for box        6       0     200       0     200
      % ABS: Variable is undefined: DX.
      % Execution halted at: GET_BOX_OFFSETS_ITER  124
      /home/lris/widgets/released/slit_align/get_box_offsets_iter.pro
      %                      LOADMASKIMAGE    4488
      /home/lris/widgets/released/slit_align/slit_align_event.pro
      %                      SLIT_ALIGN_EVENT 6578
      /home/lris/widgets/released/slit_align/slit_align_event.pro
      %                      WIDGET_PROCESS_EVENTS
      %                      $MAIN$          
      
Problem
There is a big shift in the mask alignment image, which caused the boxes not to be where the SAT software expects to find them. This might be related with instabilities in the grating tilt mechanism.
Solution
A possible way to solve the problem is to ensure that FCS is tracking before taking the alignment image.

SAT fails during fine alignment (ID boxes)

Symptom
The SAT cannot find the fiducial stars in most of the boxes, even though the stars can be clearly seen within the boxes in the fine alignment images.
Problem
This problem occurs even with FCS in tracking mode before the alignment image was taken and also having made sure that the afternoon mask images were taken with the grating selected using the select_slider script. Still, box identification using the information in the image header is not possible.
Solution
Follow this procedure to identify the alignment boxes manually:

SAT no stars on alignment boxes

Symptom
Coarse alignment is successful, but there are no stars on the alignment boxes on the first fine alignment image.
Problem
Alignment stars are slightly off the alignment boxes. This can be caused by a number or reasons, such as the slitmask center pointing origin being slightly off or a large relative flexure between the guider and the science detector.
Solution

Mask Milling

Mask cannot be milled , xaxis exceeded

Symptom
When the summit mask milling maestro attempts to mill a mask, the sequence fails with an error like: Block 43, error 401 x-axis software over travel. press esc to continue.
Problem
On occasion, the memory card in the Milltronics system may become corrupted and lose the stored parameters from its memory. This may cause the mill to drive the table into a limit or refuse to mill a mask because the coordinates are illegal. This results in an x-axis over travel condition.
Solution
Page 40 of the Slitmask Milling Manual describes the Parameter Recover Procedure. Please see: slitmask Milling System Manual of Operations.

DREMEL

Bad response when scanning barcodes

Symptom
When scanning a filter or grating barcode, the instrument responds with a "bad" signal (3 short honks).
Problem
The stage was not at a defined position. Unless the position is defined, DREMEL will not accept the barcode.
Solution
Verify that you have a green light from the stage's position indicator before you scan. If not, re-select the position using the local controls on the instrument.

No response when scanning barcodes

Symptom
When a barcode is scanned using the handheld barcode scanner, the scanner beeps an acknowledgement but DEIMOS does not give the honk response.
Problem
Either DREMEL or the bargun dispatcher is hung or otherwise unresponsive.
Solution
  1. Verify that the DREMEL and Hand Held Barcode Scanner daemons are alive:
    • On a polo terminal, run the script:
      testAll -s daemons
    • The DREMEL and barcode scanner daemons should appear as:
      Checking dispatcher.bargun.......OK
      Checking dremel..................OK
    • If not, then restart these processes as user kics at keamano:
      deimos restart dispatcher.bargun
      deimos restart dremel
  2. Verify that DREMEL is really by typing show -c -s deimot dremtick on any terminal and checking that the DREMEL tick counter increases. If not, restart DREMEL as user kics at keamano:
    deimos restart dremel
  3. Watch the dremel and bargun logs by issuing the following commands in a polo window:
    view_logfile -tail dremel
    view_logfile -tail bargn
    Also, issue the following command on polo to monitor broadcasts from the bar gun dispatcher:
    	cshow -s deimot bargnget
  4. Have a summit technician scan the barcode on a grating or filter, or scan one of the barcodes on the laminated sheets attached to the instrument.
  5. If the bar gun beeps but no entry appears in the bar gun log, restart the bargun dispatcher as follows:
    • Log into keamano as kics
    • Type the command:
      deimos restart dispatcher.bargun
  6. If the bar gun is functional but no entry appears in the dremel log, restart dremel as follows:
    • Log into keamano as kics.
    • Type the command:
      deimos restart dremel
  7. Repeat the above steps and verify that DREMEL now recognizes the bar gun signals.

Slitmask scanning error (bad barcodes)

Symptom
The automated Count/Scan operation to get the slitmask barcodes terminates with an error about mask scanning, indicating either that an incorrect number of barcodes were received or that invalid barcodes were scanned.
Problem
The barcode reader incorrectly scanned one or more barcodes.
Solution
Follow these steps to manually fix the barcode list. Note: unless you really know what you are doing, it is best to leave this to a DEIMOS expert.
  1. On the DEIMOS control GUI, click on the DREMEL Details... button. This will launch the DREMEL panel.
  2. On the DREMEL panel, click the Emergency/Diagnostics tools header to activate the buttons on emergency/diagnostics area.
  3. Click on the Get barcodes button. The current barcode list will be copied into the text box next to the button.
  4. In the text box, enter a comma-separated list of the correct slitmask barcodes starting from slot 2 and continuing through the last occupied slot. Be sure there is a one-to-one correspondence between barcodes and occupied slots (as indicated by the DREMPOPS keyword in the DREMEL information area). Do not press the Tell dremel button yet!
  5. On the DREMEL information are, verify that the DREMFITS keyword reads
    	Completed around...
    which indicates that the FITS-writing process has run to completion. If not, then the process may be hung.
  6. On the Emergency/Diagnostic tools area, click the Tell dremel button to force regeneration of FITS chunks.
  7. Click the COMMIT! button to save changes.
  8. Wait for the DREMFITS and DREMCMIT keywords to show
    	Completed around...
  9. Check the revised list of gratings, filters, and slitmasks on the Mechanism control area of the DEIMOS control GUI.

Review Config fails to launch

Symptom
Clicking the Review Config button on the DEIMOS does not bring up the tkremel window previewing the new configuration.
Problem
A possible cause is that the resulting configuration is invalid and cannot be displayed, perhaps because the same name appears in two different positions. As an example, the config below lists two Long1.0B masks with different barcodes. Because the names are the same, DREMEL fails.
	SLITMASK2: Long0.7 BC=8506
	SLITMASK3: Long1.0B BC=9090
	SLITMASK4: Long0.8 BC=7441
	SLITMASK5: LVMslitG BC=8850
	SLITMASK6: Long1.2 BC=7359
	SLITMASK7: Long1.5 BC=7360
	SLITMASK8: GOH_X BC=2126
	SLITMASK9: E0b BC=9024-no change
	SLITMASK10: uds_ft BC=8852-no change
	SLITMASK11: Long1.0B BC=7358-no change
	SLITMASK12: GOH_X BC=2124-no change
	SLITMASK13: Empty_13 (Bad port)-no change 
      
Solution
  1. Inspect the file /local/kroot/data/deimot/dyna/dremel.cfg. Check the listings for filters, gratings, and slitmasks looking for identically-named elements in two positions of the same sytem.
  2. If this fails to indicate the problem, then check the DREMEL log with the following command on a polo terminal:
    view_logfile -tail dremel
  3. Click the Review Config button again and check for messages like this:
    ERROR duplicates in stage position data!
    These seem to be duplicated
    GRATE.nam.600ZD GRATE.nam.600ZD GRSEL.nam.600ZD GRSEL.nam.600ZD
    FATAL, CANNOT PROCEED
    # found duplicates in stage data, cannot process
  4. If identical entries are found, fix the problem by either:
    • Removing the duplicate from the instrument if there are two masks for instance with the same name.
    • If duplicate names are not in the instrument, then try re-scanning the elements in the affected stage.
  5. Check tha DREMEL works as expected:
    • On the DEIMOS control GUI, click on the DREMEL Details... button. This will launch the DREMEL panel.
    • Click on Review Config and verify that it runs and displays the new configuration properly.

DREMEL can't access database

Symptom
DREMEL dies while attempting to initialize, or at the completion of a slitmask scan when attempting to read recods from slitmask database. The DREMEL panel on the DEIMOS control GUI indicates an unusual failure on the top status line.
Problem
The summit instrument Sybase server is not responding properly and requires a reboot.
Solution
  1. Use ping waiaha to determine whether waiaha is alive.
  2. Have summit staff determine whether waiaha is functioning properly, and reboot if needed.
  3. Restart DREMEL by logging into keamano as kics and running:
    deimos stop dremel
    deimos start dremel