DEIMOS Troubleshooting

Images
Computers & Software
Mechanism Moves
Electronics
Rotator
Flexure Compensation System
TV Guider
AUTOFOC and Mira
Slitmask Alignment
Mask Milling
- Mask cannot be milled, xaxis exceeded
DREMEL

Images

No light in images

Symptom

Images taken using a grating (spectral or zeroth order image) look like darks.

Problem

The grating is at the wrong tilt, or the grating cover was not removed when the grating was installed.

Solution

Re-home both grating tilts with the following commands on polo:
```
unstick_grating 3
```
```
unstick_grating 4
```
Have summit support personnel check for and, if necessary, remove the cover from the grating.

Images contaminated by red light

Symptom: Images show excess red light
Problem: A red LED inside or outside of the instrument has been left on.
Solution: Send someone up to locate the source of the illumination. One likely place is a red LED on the DEIMOS handpaddle which is lit when the paddle is left in MANUAL mode. Switch it to COMPUTER mode in order to disable the LED. Another thing to check is the plastic housing enclosing the optical encoders on the grating tilt stages, which can be deformed and thus leak light.

Direct images and spectra shifted vertically

Symptom

Spectra and/or direct images taken on the grating appear to be shifted vertically on the CCD images relative to their customary position.

Problem

The grating tilt encoder has lost its positional reference, most likely due to momentary loss of power to the tilt ensoder.

Solution

Re-home both tilt encoders by running the following commands on polo:

unstick_grating 3

unstick_grating 4

Bias jumps in images

Symptom

Images show a significant jump or drop in the bias level in one or more amplifier.

Problem

The CCD readout electronics may be too warm.

Solution

Check DEIMOS temperatures by issuing the temps command on polo.
If electronics temps are too high, then have summit staff remove covers on the electronics as needed to improve airflow.

NaN values in images

Symptom

DEIMOS images show data values as NaN.

Problem

The CCD readout electronics are malfunctioning, perhaps because they are too warm.

Solution

Check DEIMOS temperatures by issuing the temps command on polo.
If electronics temps are too high, then have summit staff remove covers on the electronics as needed to improve airflow.
If data values are bad in only one amplifier, it may be a malfunction on the Leach board. Try increasing the bias level by decreasing the value of the VIDOFFn keyword for the affected amplifier.

NaN or zero values in images only affecting one amplifier type

Symptom

DEIMOS images show data values as NaN or zeros (depending on which version of DS9 is used to open the images). The section of the image affected corresponds only to amplifier A or B.

Problem

The CCD readout electronics are malfunctioning.

Solution

Change the default readout mode. If, for instance, the problem appears in A amplifiers in Direct mode when the default amplifier mode was set to DUAL:A+B, then change the default amplifier mode to SINGLE:B by selecting the corresponding Amp mode on the Detector control panel of the DEIMOS instrument control GUI
Take an image in Direct mode by either hitting the Expose button on the DEIMOS instrument control GUI, or by using the command goi on a polo or deimos terminal to check that the amplifier mode has actually changed.

Stray light or light source detected

Symptom

Stray light detected in images

Problem

It may be that a light source was left on in DEIMOS. An example is when a grating system errors and the error is not cleared because it is not the primary grating in use.

Solution

use checkLamps to see if any of the lamps is currently on.
If a lamp is on, turn it off.
- For arclamps, you can simply use the Internal lamps → Details... button on the DEIMOS instrument control GUI and then turn off the corresponding lamps.
- If it is a motor lamp, you will first need to halt the stage and then turn the lamp off. We will use the grating 4 tilt mechanism as an example:
```
mmot g4tltmod=halt   # halts the stage
```
```
mmot g4tltfop=off    # turns off the lamp.
```
Note that the keywords in the output from checkLamps shows what keywords to modify to turn the lamps off.

Focus images with very low counts

Symptom: Focus images show very low counts during the focusloop (R+dome) script.
Problem: The tertiary mirror in not facing the correct Nasmyth focus.
Solution: Contact your Support Astronomer, who will take the proper actions to ensure the tertiary mirror is rotated to face DEIMOS Nasmyth focus.

Computers & Software

FACSUM and compass roses wrong

Symptom: Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.
Problem: The DEIMOS rotator control computer (roto) has lost its DCS connection.
Solution: See corrective actions described below.

Ginga will not load images

Symptom

Newly acquired images do not appear on the the Ginga display GUI

Problem

The Ginga display GUI failed to gather the information to create the DEIMOS mosaic. This can be caused by a temporary communication glitch with keyword services such as dcs.

Solution

Use the ds9_mosaic script to diplay the image:

Display the image on a polo or deimos terminal as follows:
```
cdata
```
```
ds9_mosaic image-name
```

If the problem persists over multiple images, then there might be an issue with the Ginga display GUI, in which case it is recommended to restart it:

Exit the Ginga display GUI by selecting File → Exit
Restart the Ginga display GUI on the VNC background menu: DEIMOS Control Menu → Subcomponents... → Start Ginga display GUI.

If there are missing dcs keywords in multiple images, then it is possible that there is a serious communication issue. In this case it is recommended to call SWOC, although observing can continue as soon as images displayed with the ds9_mosaic script look fine.

Images are not saved to proper directory

Symptom: An output directory is defined (outdir commands returns valid result) but images are not going there.
Problem: The output directory does not exist.
Solution: Create the output directory using the command newday -test.

Images are written incorrectly

Symptom

Images are saved with strange filenames (e.g., d0217_0001.fits.old.PdGWS5) and FRAMENO keyword does not increment between frames.

Problem

Either the ADDFRAME keyword is set incorrectly or the DEIMOS daemons are in a bad state.

Solution

Check the value of the ADDFRAME keyword with gshow -s deiccd addframe.
If the keyword value is false, then set it to true with modify -s deiccd addframe=1

DCS keywords missing in image headers

Symptom

Keywords such as RA, Dec, ROTPPOSN, etc., are not found in the image headers. These keywords can still be obtained from the polo command line via gshow -s dcs commands.

Problem

The watch_ccd process has lost its connection to DCS and needs to be restarted.

Solution

Wait for the exposure to end; do not begin another.

On a polo terminal:

deimos status watch_ccd

deimos restart watch_ccd

Newly-acquired images should now have these keywords in the image header. You can verify this by using the command:

fitsheader -e 0 -k rotposn image.fits

If a ROTPOSN value is shown, that's good news. If no output is received, then the keyword is missing from the header and the problem has not been fixed.

Keywords missing from image headers

Symptom

Keywords are found (or reported) to be missing from image headers.

Problem

Communication with a keyword service is interrupted.

Solution

View logfile. In general, if there is a problem with keywords missing from a DEIMOS FITS header, the best place to start is to look at the appropriate watch_ccd log file in /kroot/var/log on polo:
- If it is a FCS image that is missing one or more keywords in its FITS header, look at the logfile df_watch_ccdlog by selecting the VNC menu item DEIMOS Engineering → Log files... → Log view... → FCS watch_ccd log
- If it is a science CCD image, look at the logfile ds_watch_ccdlog by selecting the VNC menu item DEIMOS Engineering → Log files... → Log view... → Science watch_ccd log
Locate errors. Use the timestamp of the FITS file to key to the appropriate timestamped messages in the log file. In such cases where keywords are missing from a FITS file, the corresponding watch_ccdlog file should have error messages about either
- reading the missing keyword, or
- reading some other keyword from the same service.
If the former: then try to show the missing keyword; e.g., type the following at a polo prompt:
```
gshow -s deimot tmirrval
```
If that fails, then there is probably a problem either with the dispatcher that serves that keyword or the underlying hardware that that dispatcher controls.
In the latter case: when the FITS file is missing keyword X from service Y but the watch_ccdlog file instead reports an error reading keyword Y from service Y, then the problem may be more indirect. Recall that once watch_ccd gets an error reading any keyword from service Y, it will not attempt to read any more keywords from that service for that image. Thus, an error reading keyword Z will not only cause keyword Z (from service Y) to be missing from that FITS file but any other keywords from that same service which have not yet been collected by watch_ccd for this image will also be missing. In such cases, you should then check the dispatcher that serves keyword Z along with its underlying hardware.

CCD crate will not boot

Symptom

CCD crate is unresponsive. Attempts to reboot via RESET button or power cycling the unit do not succeed (see the CCD powerup procedure). Connecting via telnet to the crate shows the following message printed out every few seconds:

	Timeout waiting for ARP/RARP packet

Problem

The crate is asking for an IP address and is not being given one because the YP service does not know about it.

Solution

Ensure that the appropriate entry for the VME crate appears in the master YP database from which the 'ethers' database is pushed. For the FCS crate, that entry is:
```
	00:80:42:0b:4e:8c fcsvmep.keck.hawaii.edu
```
Change the nsswitch.conf file on keamano so that it resolves hardware ethernet addresses directly from the /etc/ethers file.

No Sounds from Speakers

Symptom

The eventsounds utility does not echo event sounds.

Problem

The soundplay/aplay utility may not be running on your machine, the eventsounds may not be up and running or the deisoundboard process running in deimos. Of course, it might also be the case that your computer's volume is too low.

Solution

Check if the deisoundboard process is running and restart it if needed. On the deimos machine:
```
deimos status deisoundboard
```
```
deimos restart deisoundboard
```
Kill the eventsounds GUI and restart it with the VNC background menu DEIMOS Control Menu → Subcomponents... → Start EventSounds GUI
If the process soundplay is not running in the local machine, open a terminal in the local machine and restart the process with the command soundplay.

Mechanism Moves

Move times out

Symptom: Mechanism move fails with error message "Error code -1: Someone needs to define an error message for this condition"
Problem: A motor move timed out or otherwise failed in a probably innocuous manner.
Solution: Re-try the move.

No light from grating

Symptom

Spectrum appears to be a dark image, although the shutter appeared to open.

Problem

Grating tilt encoder controller lost power thus sending grating to the wrong tilt for imaging, or technician forgot to remove the grating cover when inserting the grating into the instrument.

Solution

Re-home both grating tilts with the following commands on either polo or deimos:
```
unstick_grating 3
```
```
unstick_grating 4
```
Have summit support personnel check for and, if necessary, remove the cover from the grating.

Dewar filter wheel will not rotate

Symptom: Attempts to change the dewar filter wheel to another position fail. The filter wheel is stuck between two positions and cannot reach either of them.
Problem: A filter has not been loaded properly, and the filter holder is protruding from the wheel.
Solution: Have technician remove and inspect the offending filter, then re-install it properly.

Tent mirror not positioning reliably

Symptom

The tent mirror does not position reliably; the piezo controller may not appear to be functioning; FCS may not be tracking well.

Problem

The piezo dispatcher and/or controller operating the tent tent mirror may be in a bad state, possibly due to a power glitch.

Solution

First, restart the piezo dispatcher following the instructions obtained from the intrcon GUI:

Log into keamano as kics
Issue the command
```
	deimos restart dispatcher.piezo
```

If this fails to fix the problem, have summit technician cycle power on the piezo controller as follows:

The tent mirror actuator is driven by a Physik Instrument P-845 Piezo controller. This controller is located in DEIMOS barrel electronics bay #2.
The piezo controller is located in the outboard position (i.e., farthest from the center of the rotation axis) of this bay while the DEIMOS PXL camera electronics unit is located in the inboard position.
To access the piezo controller, rotate DEIMOS so that bay #2 is accessible, then remove the panel from the rear of the bay (i.e., the panel on the side of the electronics ring that faces away from the telescope).
The main power button is the square red pushbutton located in the lower right corner of the piezo controller's front panel. Switch the controller's power off for 30 seconds, then turn it back on.

Note that you can watch the output of the tent mirror control software (dispatcher.piezo) using the following command from the VNC background menu DEIMOS Engineering → Log files... → Log tail... &rarr tmirr.log

Filter move succeeds but filter is "Unknown"

Symptom

Following a filter move, the filter name turns to Unknown. Inspecting the filter wheel position on the DWFIL detail panel shows that the wheel appears to be at the proper position.

Problem

The filter wheel is unbalanced and thus when the filter rotation motor is depowered at the end of the move, the wheel turns enough that its raw encoder position no longer matches the defined position within the tolerance of 1000 units.

Solution

Possible solutions are:

Have summit staff exchange filters in an attempt to balance the wheel.
Have Lick software personnel modify the configuration file to increase the tolerance to an acceptable level.

Loss of air pressure

Symptom

The keyword deimot.AIRPRESS has the value failed. Note that the old Dashboard GUI would pop up the warning message "Air pressure has been lost!", but no such warning message has been implemented on the new DEIMOS control GUI.

The testAll script on polo shows an error on the following lines:

Checking Air pressure (barrel)...

Checking Air pressure (cradle)...

Problem

The pneumatic system on DEIMOS lost pressure, and pressure may still be out.

Solution

If an exposure is in progress, abort it immediately. The loss of air pressure has likely caused the grating to unclamp, and if the exposure continues it will be ruined. If air pressure remains off, the CCD shutter will not close and the exposure will be ruined anyway. But if you are near the end of a long exposure, it is worth trying to salvage it.
On the main Dashboard, open the Tricorder panel and check the barrel and cradle air pressure indicators at OKAY, then a problem exists. Have summit staff investigate the loss of air pressure.

Note that the air pressure is interrupted momentarily once per day around 8:30am by the air water trap purge system. Occasionally, this system may trigger one or more additional purges following at 30 minute intervals if the original purge did not clear the trap.

See also

Notes on air purge

Grating stage will not tilt (timeout)

Symptom

Attempts to change tilt of a grating stage fail. Trying to home the stage using the modify -s deimot G3TLTCAL=1 command yields a timeout error.

Problem

The motor control process for this stage is hung, or the drive belt has snapped.

Solution

Type the command unstick_grating n, where n is the number of the misbehaing grating (3 or 4), in a polo window to return the grating to normal operation.
If this times out, then suspect that the drive belt on the titl stage is broken.

Grating stage will not tilt - Galil controller needs reset

Symptom

Attempts to change tilt of a grating stage fail. Trying to home the stage using the modify -s deimot G3TLTCAL=1 command yields a timeout error.

Problem

The motor control process for this stage is hung.

Solution

Type the command unstick_grating n, where n is the number of the misbehaing grating (3 or 4), in a polo window to return the grating to normal operation.
If this fails several times, then you can try the Galil controller reset procedure.

Grating will not go To Zero

Symptom

Attempts to change tilt of a grating stage to zeroth order imaging position fail. Issuing the gozero command results in an error indicating that the move would exceed the allowable software range.

Problem

Accumulated tilt corrections are too large.

Solution

Check the value of the grating tilt offsets by issuing the command
```
smot gXtltoff
```
where X is the grating numger (3 or 4). Typically these parameters are 100 or less.
If the offset is large, (i.e., absolute value greater than 100), then reset the tilt offset and recalibrate the tilt on the stage by issuing the command;
```
gratecal
```
Re-try the To Zero operation.

Gratings will not move

Symptom: Attempts to move the grating fail. Both gratings will not tilt. A review of the grating detail panel on the DEIMOS control GUI reveals that there is an "ERROR: Controller CTRL2 is in MANUAL control mode." The rotator control GUI shows a message indicating that a panel is open.
Problem: The grating sub system is lock out because the grating hatch is open. It may have popped open when the instrument was rotated.
Solution: Have the summit crew rotate deimos to the grating load position and inspect the hatch. They may need to open and close the hatch again to get it to properly seat with the switches set. Then retry the grating moves.

Grating tilt in reverse limit

Symptom

Attempts to move the grating tilt stage fail. The deimot.G3TLTLIM or deimot.G4TLTLIM keywords indicate motor in secondary limit. The deimot.G3TLTMSG or deimot.G4TLTMSG keywords indicate that one of the grating tilt stages is at a limit.

Problem

The tilt stage has exceeded the allowed range of travel and has hit the secondary limit switch. This cuts power to the motor and prevents further motor moves.

Solution

It is necessary for summit personnel to assist in moving the stage out of a limit. This operation presents a physical hazard to the operator because the stage could move suddenly when it is backed off the limit switch. Accordingly, this task should only be performed by a qualified instrument technician!

Send the grating to the unload position, either by opening the grating access hatch and pressing the appropriate button, or by issuing the command
```
	unload_grating 3
```
to send slider 3 to the unload position, or
```
	unload_grating 4
```
to send slider 4 to the unload position. The technician can then access the stage through the grating changeout hatch.
Open the grating changeout hatch. De-power the motor by unplugging the black plastic connector carrying DC power to the motor.
Manually rotate the wheel on the friction drive mechanism to move the stage in the appropriate direction. An indication of the positive direction should be written on the wheel. If the grating tilt stage is in the negative secondary limit, you will need to move in the positive direction, and vice versa. You can monitor the current position using the G3TLTRAW or G4TLTRAW keywords to verify that the stage is moved in the proper direction. The technician may be able to detect the "click" of the mechanical limit switch when the stage moves off the limit.
With hands clear of the tilt stage, restore power the motor by plugging in the black plastic connector carrying DC power to the motor. BEWARE! The stage may begin a tilt move when power is restored.
When the stage is returned to a working state, re-home the tilt stage by clicking on either G3 tilt or G4 tilt on the Init/Stow panel on the DEIMOS control GUI.

Grating tilt in "bogus" secondary limit

Symptom

Grating stage(s) fails to move, reporting a secondary tilt limit. However, the grating detail panel on the DEIMOS control GUI indicates that the stage is not close to a tilt limit.

Problem

The secondary limit switch is falsely triggered.

Solution

Ensure via visual inspection that the stage is not in a limit. If needed, manually tilt the stage via the drive wheel to move the tilt for verification. This can be done by sending the affected grating to the unload position (e.g., modify -s deimot gratepos=-3 for the grating 3 unload position) and accessing it through the grating access hatch at the side of the instrument. If the limit switch is giving a bogus error, you can jumper out the secondary limit switch as follows:

Pull back DEIMOS from the science position.
Remove front cladding to access the grating slide area.
Locate the slider in question.
Locate the wiring interconnect panel on teh grating slider.
Disconnect the appropriate wires (detailed description TBD.

Grating will not unclamp (clamps won't release)

Symptom

Repeated attempts to unclamp the current DEIMOS grating fail.

Problem

The pneumatic clamps holding the grating will not release.

Solution

Try using the unclamp_grating command to unclamp the grating. By default, this will try 25 times to unclamp. If no success after 25 tries, go to next step.
Have summit personnel disconect the air supply line at the rear of the instrument via this procedure. This should remove the air pressure and allow the clamps to release.

Grating will not unclamp (pin still in)

Symptom

Repeated attempts to unclamp the current DEIMOS grating fail.

Problem

The grating pin remains in even after all clamps are released.

Solution

This is a tricky condition, since with the pin in you can't clamp up, nor can you unclamp. Here's one thing that has worked.

Rotate DEIMOS to ROTATVAL=-90°.
Follow the instructions on the grating clamps page to do the following:
- close clamps 1, 2, 4, and 5
- open clamps 5, 4, 2, and 1
then re-start the motor control software.

This should release the pin.

Grating will not move to unload position

Symptom: The grating will not move to the unload position, because the select stage hits a limit and the tilt move does not execute. The keyword deimot.GRSELORD=-3 as desired, but deimot.GRSELLIM shows a primary limit.
Problem: The move cannot complete due to the limit switch striker being out of position.
Solution: Use keyword-level commands to move the tilt to the unload position as described in this procedure.

Grating select move fails (pin not in)

Symptom

The grating fails to move into position. In the grating detail panel on the DEIMOS control GUI, the message reads:

 Can't set GRATEPOS=4, grating locating pin not in

And the pin is out on the grating clamps status panel.

Problem

We are at an unfavorable rotator postion where the gravity load prevents the grating from moving into position.

Solution

If you are trying to move to slider 4, execute the command

clampup_slider4

to attempt to clamp up. Otherwise, simply re-try the move from the dashboard GUI. If the pin will not engage, then rotate to a PA that is 90 degrees away from this location and retry the grating move. If this fails, rotate another 90 and try again. If it is still failing, there may be additional problems.

Grating select move fails (positioning error)

Symptom

Grating move fails with error:

Can't set GRATEPOS=3, motor shut off - positioning error.

Problem

The move failed, most likely due to excessive vibration in the lead screw generating an overtorque condition in the controller. This can happen in the vicinity of the slider 3 load position at certain DEIMOS position angles.

Solution

Rotate DEIMOS to the position at which the grating system is horizontal. This corresponds to a Rotator angle (deirotg.ROTATVAL) of -90 or +90 on the DEIMOS rotator GUI.
Home the grating select stage using the command
```
modify -d deimot GRSELCAL=1
```
or alternatively use the Initialize panel on the dashboard GUI and click the Grating selectbutton.
If the calibration succeeds, then re-try the original move. If it fails, suspect a mechanical problem.
If the calibration does not succed, try to reset the Galil controller.

Grating select move fails - Galil controller needs reset

Symptom: Grating move fails.
Problem: The Galil controller may need a reset.
Solution: Try the Galil controller reset procedure.

Timeout error after executing Recalibrate all

Symptom: The button Recalibrate All on the Init/Stow control GUI fails to home most of the mechanisms.
Problem: The mechanisms recalibration procedure failed most likely because too much air pressure was built in the DEIMOS pneumatic system.
Solution: Cycle the shutter mechanism several times to ensure that some of the air pressure is released. To do this bring up an xterm on polo as user and run the command cycle_shutter -100. This will keep trying to cycle the shutter until it finally opens and closes.

Troubleshooting motors

See Motor troubleshooting

Slitmask: select move fails with motor powered off error

Symptom

Slitmask move fails; SLMSKMSG and/or SLSELMSG keywords report that move failed and motor is powered off.

Problem

The motor move has failed due to a bad reading on a switch, usually either the one on the slitmask arm or on the door comb.

Solution

Based on one successful recovery from this error, the following procedure is suggested:

Rotate DEIMOS to the slitmask unload position (ROTATVAL=-90) and try the move again. If this works, then quit.
Attempt to calibrate (home) the stage (mmot slselcal=1). This may succeed, in which case you can quit.
Have summit support attempt to move the slitmask selector motor manually.
Attempt to calibrate the stage.

Slitmask: comb missaligned killed motor power

Symptom

The following are symptoms of a comb missalgnment:

slitmask moves fail
Slitmask sub-panel gui shows a red box for "missaligned"
When you attempt to move, the "comb" box on the grating sub-panel flashes red.
The slitmask status reads "UNKNOWN" after trying to select a slitmask
Slitmask detail panel on DEIMOS dashboard indicates comb error when attempting to move a mask.

Problem

Two possibilities exist.

A slitmask is sticking on the mask pusher mechanism, triggering the comb sensor. The slitmask system cannot be moved until the mask is freed from the pusher bar.
The comb was bumped in the process of moving a slitmask into beam. The comb needs to be re-aligned.

Solution

Automated recovery. Attempt to free the mask and/or reposition the comb by rotating DEIMOS as follows:
1. From the DEIMOS workspace menu, select DEIMOS Troubleshooting > DEIMOS Trouble Recovery > Fix Slitmask Comb Error. This will launch a script that will rotate DEIMOS in 45 degree increments until the mask is freed.
2. If this script fails to free the mask within one full revolution (8 iterations), then type CTRL-C in the xterm window to interrupt the script and proceed with the manual recovery process described below.
Manual recovery. If the automated recovery fails, then follow these steps to fix the problem manually:
1. Have summit staff send rotate the telescope to a position allowing access to the Keck II right Nasmyth platform.
2. On the Dashboard's rotator subpanel, set allow DCS control to no to permit local control.
3. On this same panel, click the Unlock button to allow the rotator to move.
4. Using the deimos rotator sub-dashboard, enter a rotator position of -80 (or 280) to allow summit staff to access the slitmask hatch, or have the OA rotate to stationary position 0 deg.
5. Have summit staff open the access hatch using the screwdriver at the instrument.
6. Have summit staff inspect the slitmasks for
  1. signs of a mask stuck on the comb.
  2. comb missalignment.
7. Check for a comb missalgnment.
  Gently push the brass comb left and right to see if the comb missalignment error will clear.
  
  Example of comb misalignment.
  
  Example of comb misalignment.
  
  Image of the comb properly aligned. The edges of the brass comb are flush with the bracket.
  
  Notes: During one error on May 13, 2015, the brass comb overlapped the black anodized support bracket on the left side of the comb by 1/32. A visual inspection suggested the comb was aligned but when the brass comb was flush with the bracket, the missalignment cleared.
8. If a mask is sticking, have summit staff remove this mask by undoing the 2 thumbscrews on the slitmask door, flipping down the door, and pulling out the mask. You can run without that mask installed the rest of the night.
9. Close the slitmask door and shut the access hatch.
10. Verify that the slitmask select stage will now operate freely by by selecting mask 2 and mask 12, thus running the system from end to end.
11. Allow DCS control of the rotator.
12. Have the OA run a stand-by init on the rotator.
13. Resume observing.

Slitmask move fails - slitmask jammed

Symptom: Attempting a slitmask move fails.
Problem: Slitmask may be jammed.
Solution: See procedure for dealing with jammed slitmask.

Slitmask move fails - fiducial hole blocked

Symptom

Attempting a slitmask move fails, but inspection of the masks shows that the masks are not jammed.

Problem

A slitmask may be missing its fiducial hole.

Solution

Remove slitmask and check to verify that all of them include a square hole on the bottom about 12 inches in from the end with the hot dog cutout.
If any such bad masks are found, remove them from the instrument.

Slitmask move fails - mask position is read both at beginning and end

Symptom

Attempting a slitmask move fails, error says that slitmask position is detected both at beginning and end)

Problem

Slitmask selector might be jammed, or might have lost its calibration.

Solution

Enable fiducial optical source to force re-read of slitmask position (mmot SLMSKFOP=1; mmot SLMSKFOP=0)
While the FOP is on, try to determine position of the slitmask arm, or slitmask.
Check that the arm is retracted, or retract it via
```
mmot SLARMPOS=retracted
```
Force a recalibration of the stage via
```
mmot slselcal=1
```

Slitmask move fails - Galil controller needs reset

Symptom: Attempting a slitmask move fails and you have already checked that no mask is jammed; Attemps to recalibrate the mechanism fail.
Problem: The Galil controller may be failing and needs to be reset.
Solution: Please, follow the Galil controller reset procedure.

Electronics

Motor controller fault

Symptom: The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running.
Problem: The motor controller has faulted.
Solution: Attempt to reset the motor controller using the Galil controller reset procedure.

Motor controller hung

Symptom: The testAll script reports that the clock on one of the motor controllers has failed, or that the Bekins/Uhaul/Monitor processes on one of the DEIMOS motor controllers are not running.
Problem: The motor controller has hung.
Solution: Attempt to reset the motor controller using the Galil controller reset procedure

Problem communicating with Galil controller

Symptom

Although the Lantronix terminal server is pingable, attempts to communicate with the Galil controller fail with this error:

  [1009] kics@keamano% telnet 192.168.6.4 3005
	Trying 192.168.6.4...
	telnet: Unable to connect to remote host: Connection
	refused

Additionally, the command ctx (executed on polo) shows an error message in the dispatcher2 prcesses listed under "Current KEAMANO tasks". If there is a dispatcher2 process running, then the connection error message shown by the telnet command is normal, because the dispatcher is already communicating to the Galil controller through the same port you are trying to use.

Problem

The port on the Lantronix unit connected to the Galil controller is hung.

Solution

Follow the Lantronix port reset procedure.

Problem communicating with Lantronix unit

Symptom

Motor moves fail. Running testAll indicates a problem communicating to the Lantronix one of the Lantronix units (deits2 or deits3).

Problem

The most likely problems are:

The Lantronix unit is hung and need to be reset.
The AUI (10Base-2 transceiver) which connects the Ethernet coax cable to the multi-pin port on the Lantronix unit is loose or broken.
The Ethernet coax cable is damaged.

Solution

Follow the procedure for power cycling the Lantronix unit.
Check cabling for loose conections or damaged cable.
Try swapping the AUIs on the two Lantronix terminal servers, then run TestAll to check whether the problem follows the AUI. If so, replace bad AUI.
Try running alternate Ethernet cable between the units to check for bad cable.
Try replacing the Lantronix unit with spare.

Ion pump is off (power glitch)

Symptom

The testAll command reports that one or both of the ion pumps is off. This is very likely to happen after a power glitch or power outage at the summit.

Problem

The ion pumps need to be turned on.

Solution

State the solution to the problem here, numbering the steps to be taken:

Go to the DEIMOS control GUI, click the Tricorder label followed by the Details... button to launch the Tricorder GUI.
Verify that the status of the LN2 can ion pump #1 power (deipower.OUTLET_A1) and/or Detector vessel ion pump #2 power (deipower.OUTLET_A2) is Off. Also check the dewar temp; if the dewar temp is well above -100 C then you may not be able to start the ion pump.
Click the Restart button under the appropriate ion pump to attempt to restart the ion pump. The status should change from Off to On.
If the startup is successful then the status will eventually change to On. If the startup fails, the status will return to Off, and you should try again to start the ion pump. Note that ion pumps are restarted at the first attempt since we upgraded the controllers on March 28, 2022.

Alternatively you can also try to restart any of the ion pumps by modifying the keywords deipower.OUTLET_A1 (LN2 can) and/or deipower.OUTLET_A2 (detector vessel):

Show the status of the ion pump on any DEIMOS Linux host terminal:
```
	    > show -s deipower outlet_a1 outlet_a2
	    outlet_a1 = Off
	    outlet_a2 = On
	  
```
Note that the LN2 can ion pump is Off.

Turn the LN2 can ion pump on:

	    > modify -s deipower outlet_a1=On
	    setting outlet_a1 = On (wait)

Check again the status of the ion pumps using either show -s deipower outlet_a1 outlet_a2 or testAll.

Double check the pump status for a few minutes after restarting to make sure that both are on.

Rotator

Rotator won't go into DCS Control Mode

Symptom: Rotator will not go into DCS control mode when hitting the Enable DCS button on the Sky mode panel on the rotator control GUI.
Problem: DEIMOS is not the selected instrument on the TCS GUI and/on the telescope tertiary mirror is on at RNAS
Solution: Ask the observing assistant to reselect DEIMOS on the TCS GUI

Rotator reporting wrong angle

Symptom

Compass roses display the wrong orientation. FACSUM reports incorrect drive angle on the rotator. The DCS keywords ROTPDEST (demanded position) and ROTPPOSN (actual position) differ by many degrees.

Problem

The DEIMOS rotator control computer (roto) has lost its DCS connection, probably due to a communication issue with the DCS gateway.

Solution

Ask the OA to follow these steps:

Check the updates on the DCS gateway terminal. Odds are that there are no DCS updates.
Restart all gateways.
Init the rotator on the TCS GUI. IMPORTANT: No need to Shutdown/Init, only Init is needed.

Rotator keywords arriving out of order

Symptom

The rotator stops following the pointing demands from DCS. FACSUM reports incorrect drive angle on the rotator.

Problem

The timestamp and demand keywords are not always in the correct order (rotdets-->rotpdsts-->rotdets2).

Solution

Ask the OA to:

Restart the Linux DCS gateway.
Initialize the rotator on the TCS GUI.

Can't find stars

Symptom

At the beginning of the night, pointing appears to be way off; stars not visible in guider.

Problem

Either the rotator has not been initialized and placed into DCS mode, or the TCS does not know the rotator position.

Solution

Run testAll to verify that the instrument and telescope settings are correct (e.g., selected instrument is DEIMOS).
If the rotator was not initialized at the start of the night, then follow these instructions to do so.
If you have not yet commanded a rotator move, have the OA send the rotator to any angle (e.g., 0°).
Verify on FACSUM that the rotator is in position angle mode.

Rotator spontaneously homes itself

Symptom: When sending a rotator move, the rotator unexpectedly executes a large slew to the home position. (Need to check if this would be the actual behavior with the new Galil controller.)
Problem: The rotator software determined that the discrepancy between the readings on its two absolute encoders was too large and decided to home itself in order to reset the encoders before proceeding.
Solution: Allow the move to complete, then proceed with observing. Note that properly initializing the rotator (Shutdown/Init on the TCS GUI) at the start of each night should generally prevent this from happening.

Can't do Shutdown/Init on rotator

Symptom

At the start of the night, the OA cannot perform the Shutdown/Init operation on the rotator.

Problem

The rotator software on roto is not communicating properly with DCS.

Solution

Apply the same solution as above
If the above does not fix it, restart the rotator dispatcher as follows:
- Open a roto terminal (DEIMOS Engineering → xterm@roto)
- On the roto terminal type:
```
deimos restart deirotg
```

Rotator moves are all outside of limits

Symptom

When the OA attempts to send a rotator move, SKY complains that the requested angle is out of range for the DEIMOS rotator although it is clearly a legal request.

Problem

The rotator limits are not properly defined.

Solution

Verify the problem by displaying the keyword values:
```
show -s dcs rotccwlm rotcwlm
```
The correct values are:
```
rotccwlm = -330.00 deg
```
```
rotcwlm = 402.00 deg
```
If you get zero values back, have the OA re-load DEIMOS as the current instrument.

Discrepancy between the rotator status reported by DEIMOS and TCS.

Symptom

There is a long-term (longer than ~1 min) discrepancy between the deirotg.ROTATSTA and dcs2.ROTSTAT keywords while DEIMOS is tracking on a target. For instance, one keyword may be tracking and the other on slewing. You can monitor these keywords by launching DEIMOS Utilities → Monitor ROT and DCS status.
There is an inconsistency on the Rotator status flag on FACSUM when you are tracking. For instance, you may be tracking on a target (at an elevation lower than 86°), and FACSUM keeps reporting acquiring on the Rotator status. Sometimes, the Rotator status on FACSUM toggles from tracking to slewing or acquiring even when the elevation is lower than 86°
The DEIMOS rotator log, which you can monitor by launching DEIMOS Engineering → Log files... → Log tail... → deirotg log, will start showing errors such as Have to restart keyword monitor for dcs2... can't read dcs2 keyword ROTXXXX... Virtual circuit disconnect.

Problem

The rotator computer (roto) has lost communications with the DCS gateway.

Solution

Ask the OA to follow these steps:

Check the updates on the DCS gateway terminal. Odds are that there are no DCS updates.
Restart all gateways.
Init the rotator on the TCS GUI. IMPORTANT: No need to Shutdown/Init, only Init is needed.

This will send the rotator back to where it was. It will be a good idea to do an alignment check if the telescope was aligned on a mask when this issue happened.

Rotator fails to move under DCS control.

Symptom

The rotator stops moving under DCS control. This may happen any time during a slew, or while acquiring or tracking.

When you type:

gshow -s dcs el

on any DEIMOS terminal, the output is:

keyword_read(el): ca_array_get failed 192
keyword_read(el): Virtual circuit disconnect
Can't blocking-read dcs.el: Can't get k2:dcs:axe:traj.EL ca_array_get failed 192 368 (Virtual circuit disconnect)
EL = deg

The DEIMOS rotator log, which you can monitor by launching DEIMOS Engineering → Log files... → Log tail... → deirotg log, will start showing errors such as Have to restart keyword monitor for dcs2... can't read dcs2 keyword ROTXXXX... Virtual circuit disconnect.

Problem

The DCS gateway has died. This is a different manifestation of the issue described on discrepancy between the rotator status reported by DEIMOS and TCS.

Solution

Apply the same solution as on the entry on discrepancy between the rotator status reported by DEIMOS and TCS.

Rotator disabled due to ESTOP or power outage

Symptom

Rotator will not move or initialize. The yellow message at the bottom of the DEIMOS rotator GUI indicates ESTOP from DCS.

Problem

The rotator is disabled because the Keck II facility ESTOP circuit was de-activated, either because someone pressed a facility ESTOP button or because of a power outage.

Solution

Follow the DEIMOS rotator controller reset procedure.
Verify that the ESTOP from DCS message on the DEIMOS rotator GUI is gone.

Rotator computer bypass activated

Symptom: The Rotator control GUI shows a yellow message indicating Computer/Manual bypass activated.
Problem: The rotator toggle bypass switch is in the BYPASSED position.
Solution: Have the summit crew change the toggle bypass switch to NORMAL (see Figure 2 in manual rotation procedure).

Rotator does not move after having restared deirotg

Symptom

The rotator does not move when commanded from the Rotator control GUI or using the hand paddle, even after having restarted the deirotg dispatcher.

The following errors appear on the /kroot/var/log/deirotgdebuglog at roto:

 Jun 11 18:37:08 roto deirotg[14091]: 8.998670 E: ERR_GALIL_ERR_RESP {1 - Unrecognized command}. Lidx={12}. Cmd=MG D_ISCAL.
 Jun 11 18:37:09 roto deirotg[14091]: 8.999678 D: notifyStdStage obo=-herror- ax=virtDispatcher/nowaitcv_value = {ERR_GALIL_ERR_RESP 1 - Unrecognized command} 
 Jun 11 18:37:09 roto deirotg[14091]: 8.999795 D: Not deleting SSID 1 because /nodelete is set

Problem

The rotator controller is in a bad state, probably after having been power cycled.

Solution

The rotator controller needs to be reset. This can be done remotely as follows:

As any DEIMOS numbered account on roto: deimos stop deirotg.
As any DEIMOS numbered account on roto, keamano, or polo: telnet 192.168.6.20
Hit Enter a couple times until the controller echoes back a : (colon) prompt.
Type RS and hit Enter. The controller will echo back a : colon prompt after resetting.
Terminate the telnet connection (ctrl + ] followed by quit).
As any DEIMOS numbered account at roto: deimos start deirotg
The rotator will require homing due to the controller reset.

Flexure Compensation System

No light on FCS images

Symptom

No light is visible in an FCS exposure

Problem

Probably one of the following:

CCD shutter is not open
I or Z-band filter is in use
CuAr lamp burned out
Grating cover left on
Grating is tilted side-to-side

Solution

Follow these steps to recover:

Verify that the CCD shutter is open.
If the I or Z band filter is in use, ensure that the CuAr lamp is turned on; the LED does not emit at these wavelengths.
If still no light, try switching to the LED lamp on the FCS GUI.
If still no light, acquire a spectrum with the science array and check whether any light is received. If not, send someone up to check whether the grating cover has mistakenly been left in place.
If still no light, check whether the science spectra appear shifted to the left or right relative to the expected positions based on the slit layout. If so, the grating may not be properly seated.

FCS cannot track (offset too large)

Symptom

FCS image shows light but software cannot lock on to image and complains that the offset is too large.

Problem

The grating tilt is not correct and the image is shifted.

Solution

Verify that central wavelength and filter are set to exactly the correct values for FCS. If they are not, then the FCS GUI will show Lockout status.
Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.

FCS cannot track (xcorr failed)

Symptom

FCS image shows light but software reports X-correlation failed. Image contrast too low or grating/slider misregistered errors.

Problem

Potential causes are:

The grating tilt encoder has lost its position so the actual position does not match the reported position.
Either the FCS reference image or the new FCS image contains a substantial cosmic ray.
Grating became unclamped due to a loss in DEIMOS air pressure.
Grating was not properly installed in its mount by summit technician.

Solution

Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.
Re-home the grating tilt mechanism by either clicking on the G3 tilt or G4 tilt button on the Init/Stow GUI, or by typing "mmot g#tltcal=1" at a polo prompt, and where "#" is either "3" or "4".
Inspect the FCS reference image for cosmic rays and if needed, re-take the reference image.
Unclamp and re-clamp the grating via the dashboard.
Have summit staff remove and re-install the suspect grating to ensure that it is seated properly in the slider.

FCS won't permit "match previous" option

Symptom

You want to match against a previous observation on the FCS GUI Reference control panel, but no reference is taken.

Problem

There are no saved FCS setups which match the current setup.

Solution

Go to the current FCS directory (via cdfcs) and verify that an FCS reference file exists with the grating, slider number, central wavelength, and blocking filter that exactly match the current settings.
If there is no matching reference image, grab it from the FCS reference archive by typing on a polo terminal:
```
fcsref_retrieve YYYYmmmDD
```
where the date, e.g. 2021mar18, must correspond to a previous date when an FCS reference was taken with the exact same configuration as the current one.

FCS attempt to match previous fails

Symptom

An attempt to use the "Match Previous" feature of FCS fails with the error Dewar X translation stage hit a limit.

Problem

The grating is either improperly seated now, or was improperly seated when the original FCS reference exposure was taken. Alternatively, changes to the grating cell, grating system, or grating parameters (e.g., GRATOFFx) may cause the grating to clamp up in a slightly different position and thus shift the images. See also next item.

Solution

Compare Y positions of the same reference spots on the original FCS reference image and the new reference image. Verify that they are separated significantly in Y. If not, then the problem is elsewhere.
Have summit staff unload and re-install the grating.
Re-try the "Match Previous" operation.
If "Match Previous" continues to fail, then some change in the grating system may be the cause. In this case, matching previous may not be possible.

FCS cannot match previous

Symptom

When attempting to use the "Match Previous" function on the Reference control panel of the FCS GUI to match to the previous night's FCS reference image, the attempt fails with an error unable to match. Comparing the current FCS image to the previous reference image shows a large spatial offset.

Problem

The grating tilt encoder has slipped and the new image is not properly registered.

Solution

Recalibrate the grating tilt by typing the following command on a polo or vm-deimos (local) terminal:
```
modify -s deimot g3tltcal=1
```
Note: if using slider 4, specify g4tltcal=1 instead.
Re-try the match previous operation.
If FCS still cannot match, use the fcsmov command to steer the current image so that it matches the previous reference image.

FCS ping-pongs in X

Symptom

FCS issues moves but does not converge on the target, which keeps bouncing back and forth in the X direction.

Problem

The piezo controller/actuator is not functioning properly.

Solution

Inspect cabling at the FCS controller (located in DEIMOS barrel electronics bay #2), particularly the high voltage line coming out the center of the controller.
If cabling looks fine, check that the FCS controller channel is working properly by using one of the other channels on the controller (consult Lick first).

FCSTRACK script dies on startup

Symptom: When attempting to run fcstrack, the script dies with an Subscript out of range. error.
Problem: The script attempted to read an undefined keyword.
Solution: If fcstrack does not restart automatically, then restart it from the VNC menu (DEIMOS Control Menu.. → Subcomponents... → Start fcstrack.)

FCS Lockout (cannot track due to focus mismatch)

Symptom

FCS refuses to lock in due to a lockout condition, which is claimed to be a focus mismatch although the focus is correct.

Problem

The focus tolerance keywords are not initialized properly.

Solution

Check the keyword values via:
```
	show -s deifcs fcsfoto1 fcsfoto2
```
If they are zero, on any polo or vm-deimos (local) terminal, type
```
	modify -s deifcs fcsfoto1=200 fcsfoto2=600
```

FCS Lockout (cannot track due to wavelength mismatch)

Symptom

FCS remains in lockout condition even when the grating, central wavelength, filter and focus values all look correct. It complains that there is no setup for the current configuration. Beware especially after a mask alignment.

Problem

Usually, the central wavelength of the grating is slightly off from reference value. There is finite resolution in the tilt mechanism and sometimes the "Go Back" function of the grating does not set the exact wavelength.

Solution

Move off the central wavelength, then back again.

On the Mechanism control panel of the DEIMOS control, change the Wavelength field by 10 Angstrom.

Return to your desired central wavelength.

This should solve the problem.

FCS CCD failure

Symptom

FCS cannot track. Looking at FCS images reveals that one of the two FCS CCDs is not reading out (i.e., half of the FCS image displayed on FIGDISP contains all zero-valued pixels), although the other CCD looks normal.

Problem

The signal chain for one of the FCS CCDs is interrupted.

Solution

WARNING Do not attempt to power cycle or reboot the FCS system. If you do, the FCS system can get caught in a catch 22. Without a proper cable connection you can't boot, but to test the connection you need to boot and take an image. Yikes. Trouble shoot only during the day, and an expert technician at the summit will need to check and repair the cabling to make a permanent repair (see [KD2-19923].)

As a temporary measure, as long as one of the CCDs is reading out, the Staff Astronomer can enable partial FCS compensation by telling the FCS software to use only the one working CCD. To do this:

Edit the following file as dmosbld at polo:
```
/kroot/rel/defult/bin/fcstrack
```
Under normal operation (both CCDs working), line 63 of this script reads:
```
set offline = 0
```
If CCD 1 (left-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read:
```
set offline = 1
```
If CCD 2 (right-hand CCD as viewed with FIGDISP) is malfunctioning, change this line to read:
```
set offline = 2
```
Kill and restart the fcstrack script. You should get a warning message:
```
*** FCS CCD #1 has been marked offline ***
```
Verify that FCS is now able to track.

FCS dewar stage at negative (or positive) limit

Symptom

FCS complains of a lockout situation because the dewar stage is at a negative/positive limit. FCS cannot re-position the dewar stage to correct for differences between the FCS reference image and the current FCS image.

Problem

One problem may be that the grating is improperly clamped either currently or when the reference image was acquired in the afternoon. If the grating is improperly clamped, the result could be large shifts between FCS images.

Solution

Try simply unclamping and reclamping the grating. If it does not work the first time, try it once more.
If the problem was that the grating was inserted during the nigth either without using Select Slider or while the rotator was not in the correct orientation for the corresponding Select Slider command, the fastest solution is to:
- Have the OA to record the current position and to stop guiding.
- Run Select Slider 3|4. This should insert the grating using the correct rotator angle.
- Wait for FCS to track. FCS might take a couple of minutes to track, because the dewar stage has to move away from its limit.
- Once FCS is tracking, have the OA to start guiding again.
- Check that your target is correctly aligned.
If you are not certain of when the grating was improperly campled and the previous options did not work, rotate the instrument 90 degrees from the current position or to -30 deg (about the rotator value set by FCS setup in the afternoon). Then try unclamping and reclamping the grating. If this is done during the night while the instrument is guiding, it will require coordination with the OA, since guiding should be stopped.
If none of the previous options work, the current position of the grating is likely not at fault. It may have been that the grating was improperly clamped when the FCS reference images were acquired. If this is indeed the problem, then an FCS image will need to be retaken as a reference. You will need to unlock the rotator and dis-allow DCS control of the rotator. Next, reacquire FCS images following the afternoon procedures. When finished acquiring new FCS images, you will need to put the rotator back into DCS control and have the OA do a standby-init on the rotator.

FCS fails to cross correlate images

Symptom: FCS images are flipped left/right or the spectrum appear shifted left/right relative to the FCS reference image. When FCS is tracking, large cross correlation errors are reported by the FCS error handler.
Problem: This problem is attributed to attenuated signal in the fiber optic digital communications link between the FCS CCD controller (inside DEIMOS) and FCS CCD VME crate (in the Keck-2 computer room).
Solution: This problem may not be solved at night. A technician will need to clean the FCS fibers and reseat the boards during the day.

FCS status constantly toggling between Emergency and Tracking

Symptom

One or more of the following conditions are given:

FCS keeps toggling between Emergency and Tracking.
The FCS log keeps issuing errors about a stage having reached a limit.
The FCS cross correlation is stuck on a secondary maximum (see this cross-correlation plot as an example.)

Problem

One of the FCS stages has reached a limit, the spots on the current FCS image are offset by more than 25pix with respect to the reference image and FCS is tracking on a false broad maximum. This issue requires action as soon as possible.

Solution

Execute the FCS fix procedure. The FCS fix GUI can be launched from the Actions menu on the DEIMOS control GUI.

FCS status constantly toggling between Warning and Tracking

Symptom

One or more of the following conditions are given:

FCS keeps toggling between Warning and Tracking.
The FCS log keeps issuing warnings such as:
1. WARNING: dwxl8trg= is within 100 counts of its maximum range (-2300, 700).
2. WARNING: gntltoff is larger than the recommended range (-100, 100).
3. WARNING: tmirrtrg is within 490 counts of its maximum range (10, 9400).
The FCS cross correlation looks good (see this cross-correlation plot as an example.)

Problem

One of the FCS stages is close to its limit, but it has not actually reached the limit yet.

Solution

This does not require immediate action as soon as the spots on the current FCS image are reasonably well aligned with the spots on the FCS reference image. You may want to silence the FcsAlarm sound on the EventSounds GUI.

FCS cannot save images. FCS CCD crate crashes

Symptom: When taking reference image or starting FCS tracking, the ccd reads out but no image is saved and a 30 seconds timeout warning appears. The FCS ccd crate become unresponsive.
Problem: Wrong directory permissions: the account running the fcstrack script does not have write permission to the current output directory.
Solution: Compare the group membership and the write permission of the current outdir with the other numbered (user) accounts. Have the SA check the proper group memberships, and if neither you nor the SA can change them, request the help of the software on call person. Unfortunately, this problem also crashes the FCS CCD crate which needs to be rebooted and restarted (see the FCS CCD powerup procedure).

FCS saves images with wrong name

Symptom: When taking reference image or starting FCS tracking, the CCD reads out and displays an image on the FCS GUI, but the image is saved under the wrong name (e.g., fcs0504_2057.fits.old.rNAkGG)
Problem: The FCS watch_ccd daemon is in a bad state.
Solution: Restart the watch_ccd program.

FCS power is disabled

Symptom

The testAll script indicates that power to the the FCS CCD system is disabled:

  Checking FCS CCD 15V Power.......ERROR! Current value 'disabled' should be 'enabled'
  Checking FCS CCD 30V Power.......ERROR! Current value 'disabled' should be 'enabled'

Problem

Although the power may be off, the more likely issue is that the FCS CCD system was reading out at the time you ran testAll

Solution

Re-run testAll and check whether the system still reports an error. If necessary, stop the FCS system by issuing the command

mfcs fcsmode=off

and then re-run testAll to check status.

TV Guider

Guider is not responsive

Symptom

Guider cannot be selected from MAGIQ GUI.

Problem

Guider is powered off or requires a power cycle.

Solution

Cycle power to the guider off and on by executing the following command on polo:

tvpower cycle

This should take about one minute to complete.

Bad guider images

Symptom

Guider can be selected on the MAGIQ GUI, but images from the guider do not show stars or only a portion of the field (top 1/3) show stars.

Problem

Either the guider is not properly configured or the shutter is broken.

Solution

Follow these steps to determine source of problem:

Verify that the DEIMOS hatch is open from the main DEIMOS control GUI.
On a polo terminal, type tvfilter to verify that the TV filter is set to a reasonable value (e.g., R). If it is set to Home, select R.
On a polo terminal, type tvfocus to verify that the TV focus is set to a reasonable value for the selected filter (e.g., 970 for R). Note that 1 is not a reasonable value.
Note that the default value for the guider filter and focus is R and 970. If you suspect that the defaults should be in use, reselect these values.
If still no good image, close the hatch, turn on the Qz lamp, and see whether any light is seen on the image.
If no light is seen, the guider shutter may be inoperative. Replacing this is an extensive job for the day crew.

Bad guider images

Symptom: Light is seen on the guider images, but the images appear "smeared" horizontally.
Problem: The serial clock card in the guider controller is malfunctioning.
Solution: Repairing this is a full day's work for a talented daycrew. Have daytime summit staff pull DEIMOS, remove access panels on front of instrument, and check serial card in the guider head.

Poor guiding in vertical angle mode

Symptom: When offset guiding in vertical angle mode, the guide star does not move, but the target drifts off the slit.
Problem: The wrong guiding mode has been selected, so the center of rotation is set to the guide star rather than the slit.
Solution: Please see table on guiding modes. This is a case of offset guiding. Center of rotation needed to be set and then start guiding with "On Star" button.

AUTOFOC and Mira

AUTOFOC returns abnormal telescope focus value

Symptom

After running AUTOFOC to focus the telescope, the secondary mirror position is more than a few tenths of a mm away from the nominal value of 0.8 mm.

Problem

The TV camera focus is not set correctly.

Solution

Check the TV camera focus value by typing tvfocus on a polo terminal, which should be around 1000 units.
Change this value as needed by typing tvfocus NEW_VALUE on a polo terminal.
Re-run AUTOFOC and verify that the value has changed.

MIRA images are badly out of focus

Symptom: When running MIRA, the images are badly out of focus.
Problem: The telescope is badly out of focus, possibly because the DEIMOS TV camera focus is not set correctly.
Solution: See the description for AUTOFOC returns abnormal telescope focus value.

AUTOFOC crashes (OBSOLETE)

Symptom

Autofoc crashes with the following error message:

can't connect autofoc to aut_srv_task

Problem

The wrong version of AUTOFOC was run. There is a SunOS version for use with the SunOS guider, and a Solaris version for use with the Solaris guider.

Solution

Run the correct version of AUTOFOC.

MIRA fails to reset parameters

Symptom

Data directory, filename prefix, etc., are not properly reset after running MIRA.

Problem

The MIRA script failed to complete.

Solution

Have the OA or SA restore parameters as follows:

Open a polo terminal
Execute the command:
```
restore_state -verify
```
This should restore the pre-MIRA settings.

MIRA image shows gradient

Symptom: MIRA images show a steep top-to-bottom gradient in the brightness of the segments.
Problem: This is not generally a problem, just a benign symptom of using the 900-l/mm grating.
Solution: As long as all segments can be identified and MIRA completes without error, no additional action is required.

Slitmask Alignment

IRAF xbox Fails (Image not Written)

Symptom

When xbox is run, it fails before analyzing the first box, with an error akin to

extension BluSlits not found

Subsequent attempts to analyze this same image yield the same error.

Problem

The xbox program attempted to read the image before it was fully written.

Solution

Type flpr and re-run xbox.
If that fails, restart IRAF and re-run xbox.
If that fails, inspect the image header (execute the command fitsheader -e BluSlits filename on polo) and verify that the specified extension exists in the image header.

IRAF xbox Fails (Missing FITS Chunks)

Symptom

When xbox is run, it fails before analyzing the first box, with an error akin to

extension DESIslits not found

Subsequent attempts to analyze this same image yield the same error.

Problem

The FITS header extension called DesiSlits is not being read by the software, either because it does not exist in the image header or because IRAF can't read it.

Solution

If you are trying to get a slitmask aligned and are pressed for time, proceed straight to the procedure for aligning slitmasks without FITS chunks. Otherwise, you can proceed as follows:

Verify that a slitmask is in place (the DesiSlits extension is not written when slitmask is set to None or Unknown). If not, insert mask and re-run xbox.
Use fitsheader -e DesiSlits filename on polo to check if the extension exists.
If DesiSlits is found, then try restarting IRAF and re-reading the file.
If DesiSlits is not found, then run the command check_fits_chunks to determine whether the FITS chunks are on disk.
If all indications are OK, then restart the write_ccd process as the current observer on polo:
```
	deimos restart write_ccd
```

If FITS chunks are missing, it is likely that something went wrong in the process of scanning the masks. Follow the procedure for dealing with a mask scan error, then take these additional steps:

Verify that slitmask list on the DEIMOS control GUI agrees with expectation.
Run the command check_fits_chunks to determine whether the FITS chunks are on disk. If not, then seek help.
Then try taking a new alignment image and running the alignment software.

IRAF xbox Fails (Bad box)

Symptom

When xbox is run, it crashes with a floating point error when it tries to analyze a certain box.

Problem

A truncated box (i.e., one that is at the edge of the CCD) may cause this to occur. The mask must be re-analyzed with the offending box excluded.

Solution

Identfy the bad box by displaying it with the ds9_mosaic. Note the PANE corodinates of the bad box.
In IRAF, run the command get_boxes image, where image is the name of an alignment image, to generate a "boxfile" (text file) listing the boxes (generally named box.maskname).
Edit the boxfile. Locate the line corresponding to the bad box by comparing the PANE coordinates. Delete the line and save the file.
In IRAF, run xbox from the command line using the box file as input:
```
	xbox image input=boxfile
```

IRAF xbox reports STAR SKIPPED

Symptom

When xbox is run, it beeps and complains STAR SKIPPED when attempting to locate certain stars.

Problem

The alignment boxes were not found at the expected locations in the alignment image, probably due to either uncorrected flexure or changing to a different grating.

Solution

In IRAF, execute the command
```
	tune_qmodel last
```
to adjust the pointing model so that the guide boxes are found. Note that after you mark the location of the alignment box, the program will invoke xbox in practice mode.
Re-run do_xbox to compute and send telescope moves.

SAT fails when identifying the mask boxes

Symptom

During the afternoon checklist, SAT gets stuck when trying to display one of the user-defined slitmasks. The IDL window (slit_align) shows an error message and its prompt is active. Note that the IDL window (slit_align) is minimzed in the lower region of the VNC window, hidden behind the main SAT GUI. To bring up the slit_align window, double-click on it when it is minimized once you have shifted away the main SAT GUI.

Problem

There is some problem with the mask geometry, which does not allow boxes identification using the information in the image header.

Solution

Follow this procedure to identify the alignment boxes manually:

At the slit_align IDL window prompt, execute the command retall to restore all IDL internal variables.
On the SAT GUI, select the Options tab.
Change the Box file format to maskname.box.
Select the tab ID boxes.
Load the desired image on the SAT display. This may take a few seconds. Once the image is loaded, NONE of alignment boxes will appear enclosed in a square box.
Place the cursor on each of the alignment boxes and press m to mark them. A red square should enclose each of the identified boxes.
Once you are done selecting manually all alignment boxes for the mask, open a terminal and type cdata to change to the directory where DEIMOS images are being saved.
List the files in the data directory to check a file named box.YourMaskName exists. The file should contain a list with the coordinates of all identified boxes for that particular mask.
On the SAT GUI, select the tab Options.
Change the Box file format to fits.extn.

IMPORTANT NOTE: During the night, when doing the on-sky coarse alignment for this mask, make sure before starting the alignment, the Box file format option in the Options tab is set to maskname.box, but only for this mask. The Box file format option should be fits.extn for any other mask whose boxes were identified automatically rather than manually.

SAT fails during fine alignment

Symptom

Once the coarse alignment has been completed successfully, SAT crashes during the fine alignment procedure. The IDL window (slit_align) shows the following type of error message:

      % GET_BOX_OFFSETS_ITER: starting iteration 1
      input coords for box        0    3322    1176
      input section for box        0    3222    3422    1076    1276
      output section for box        0       0     200       0     200
      [...]
      output section for box        5       0     200       0     200
      input coords for box        6    7355    1023
      input section for box        6    7255    7455     923    1123
      output section for box        6       0     200       0     200
      % ABS: Variable is undefined: DX.
      % Execution halted at: GET_BOX_OFFSETS_ITER  124
      /home/lris/widgets/released/slit_align/get_box_offsets_iter.pro
      %                      LOADMASKIMAGE    4488
      /home/lris/widgets/released/slit_align/slit_align_event.pro
      %                      SLIT_ALIGN_EVENT 6578
      /home/lris/widgets/released/slit_align/slit_align_event.pro
      %                      WIDGET_PROCESS_EVENTS
      %                      $MAIN$

Problem

There is a big shift in the mask alignment image, which caused the boxes not to be where the SAT software expects to find them. This might be related with instabilities in the grating tilt mechanism.

Solution

A possible way to solve the problem is to ensure that FCS is tracking before taking the alignment image.

Make sure that SAT is recovered, either by restarting it or by typing retall and .reset at the IDL prompt in the slit_align terminal.
On the DEIMOS control GUI, select the science wavelength in the GRATINGS panel.
Check if FCS is tracking. In case FCS is not tracking use the Fix FCS GUI from the Actions menu on the DEIMOS control GUI.
Once the FCS is tracking, go back to SAT and Start Fine Alignment.

SAT fails during fine alignment (ID boxes)

Symptom

The SAT cannot find the fiducial stars in most of the boxes, even though the stars can be clearly seen within the boxes in the fine alignment images.

Problem

This problem occurs even with FCS in tracking mode before the alignment image was taken and also having made sure that the afternoon mask images were taken with the grating selected using the select_slider script. Still, box identification using the information in the image header is not possible.

Solution

Follow this procedure to identify the alignment boxes manually:

In the main SAT GUI, select the tab Options.
Change the Box file format to maskname.box.
Select the tab ID boxes.
Load the desired image in the SAT display (either an image from the afternoon mask identification procedure, or the latest fine alignment image). This may take a few seconds.
Once the image is loaded in the SAT display window, there are two options: (1) All/sufficient alignment boxes are detected automatically (boxes are enclosed in a red square), (2) NONE of the alignment boxes appear enclosed in a red square.
If none of the alignment boxes are automatically marked, place the cursor on each of the alignment boxes and press m to mark them. A red square should enclose each of the identified boxes. Use d to delete unwanted/mistaken identifications.
Once you are done selecting manually all alignment boxes for the mask (or in the case the boxes were identified automatically), open a terminal and type cdata to change to the directory where DEIMOS images are being saved.
List the files in the data directory to check if a file named box.YourMaskName exists. The file should contain a list with the coordinates of all identified boxes in that particular mask.
In the main SAT GUI, select Actions → Load Fine Alignment Image and select the last fine alignment image.

SAT no stars on alignment boxes

Symptom

Coarse alignment is successful, but there are no stars on the alignment boxes on the first fine alignment image.

Problem

Alignment stars are slightly off the alignment boxes. This can be caused by a number or reasons, such as the slitmask center pointing origin being slightly off or a large relative flexure between the guider and the science detector.

Solution

Option 1. Use the Imaging Coarse Align procedure to move the alignment stars inside the alignment boxes.
Option 2. This is a manual version of the previous option, in case the previous option fails.
1. Select slitmask None on the DEIMOS control GUI.
2. Hit Expose on the DEIMOS control GUI. This will take an image of the FOV without any mask on the focal plane.
3. Open the fine alignment image with no stars in the boxes using ds9_mosaic image_name.
4. Open the image of the FOV without slitmask on the same DS9 using the option File → Open as → Mosaic WCS...
5. On the DS9, select Frame → Match → Frame → WCS.
6. Blink the images with and without the slitmask.
7. Visually identify the alignment stars on the image of the FOV without the slitmask.
8. Apply the offset from one of the alignment stars to the center of its corresponding alignment box as follows:
  - Calculate the approximate offet in X (pix) from the star to the box center: delta_x = x_box - x_star.
  - Calculate the approximate offet in Y (pix) from the star to the box center: delta_y = y_box - y_star.
  - On a polo terminal type: pxy delta_x delta_y
9. Insert back the slitmask using the DEIMOS control GUI.
10. Start fine alignment on the SAT.

Mask Milling

Mask cannot be milled , xaxis exceeded

Symptom: When the summit mask milling maestro attempts to mill a mask, the sequence fails with an error like: Block 43, error 401 x-axis software over travel. press esc to continue.
Problem: On occasion, the memory card in the Milltronics system may become corrupted and lose the stored parameters from its memory. This may cause the mill to drive the table into a limit or refuse to mill a mask because the coordinates are illegal. This results in an x-axis over travel condition.
Solution: Page 40 of the Slitmask Milling Manual describes the Parameter Recover Procedure. Please see: slitmask Milling System Manual of Operations.

DREMEL

Bad response when scanning barcodes

Symptom: When scanning a filter or grating barcode, the instrument responds with a "bad" signal (3 short honks).
Problem: The stage was not at a defined position. Unless the position is defined, DREMEL will not accept the barcode.
Solution: Verify that you have a green light from the stage's position indicator before you scan. If not, re-select the position using the local controls on the instrument.

No response when scanning barcodes

Symptom

When a barcode is scanned using the handheld barcode scanner, the scanner beeps an acknowledgement but DEIMOS does not give the honk response.

Problem

Either DREMEL or the bargun dispatcher is hung or otherwise unresponsive.

Solution

Verify that the DREMEL and Hand Held Barcode Scanner daemons are alive:
- On a polo terminal, run the script:
```
testAll -s daemons
```
- The DREMEL and barcode scanner daemons should appear as:
```
Checking dispatcher.bargun.......OK
```
```
Checking dremel..................OK
```
- If not, then restart these processes as user kics at keamano:
```
deimos restart dispatcher.bargun
```
```
deimos restart dremel
```
Verify that DREMEL is really by typing show -c -s deimot dremtick on any terminal and checking that the DREMEL tick counter increases. If not, restart DREMEL as user kics at keamano:
```
deimos restart dremel
```
Watch the dremel and bargun logs by issuing the following commands in a polo window:
```
view_logfile -tail dremel
```
```
view_logfile -tail bargn
```
Also, issue the following command on polo to monitor broadcasts from the bar gun dispatcher:
```
	cshow -s deimot bargnget
```
Have a summit technician scan the barcode on a grating or filter, or scan one of the barcodes on the laminated sheets attached to the instrument.
If the bar gun beeps but no entry appears in the bar gun log, restart the bargun dispatcher as follows:
- Log into keamano as kics
- Type the command:
```
deimos restart dispatcher.bargun
```
If the bar gun is functional but no entry appears in the dremel log, restart dremel as follows:
- Log into keamano as kics.
- Type the command:
```
deimos restart dremel
```
Repeat the above steps and verify that DREMEL now recognizes the bar gun signals.

Slitmask scanning error (bad barcodes)

Symptom

The automated Count/Scan operation to get the slitmask barcodes terminates with an error about mask scanning, indicating either that an incorrect number of barcodes were received or that invalid barcodes were scanned.

Problem

The barcode reader incorrectly scanned one or more barcodes.

Solution

Follow these steps to manually fix the barcode list. Note: unless you really know what you are doing, it is best to leave this to a DEIMOS expert.

On the DEIMOS control GUI, click on the DREMEL Details... button. This will launch the DREMEL panel.
On the DREMEL panel, click the Emergency/Diagnostics tools header to activate the buttons on emergency/diagnostics area.
Click on the Get barcodes button. The current barcode list will be copied into the text box next to the button.
In the text box, enter a comma-separated list of the correct slitmask barcodes starting from slot 2 and continuing through the last occupied slot. Be sure there is a one-to-one correspondence between barcodes and occupied slots (as indicated by the DREMPOPS keyword in the DREMEL information area). Do not press the Tell dremel button yet!
On the DREMEL information are, verify that the DREMFITS keyword reads
```
	Completed around...
```
which indicates that the FITS-writing process has run to completion. If not, then the process may be hung.
On the Emergency/Diagnostic tools area, click the Tell dremel button to force regeneration of FITS chunks.
Click the COMMIT! button to save changes.
Wait for the DREMFITS and DREMCMIT keywords to show
```
	Completed around...
```
Check the revised list of gratings, filters, and slitmasks on the Mechanism control area of the DEIMOS control GUI.

Review Config fails to launch

Symptom

Clicking the Review Config button on the DEIMOS does not bring up the tkremel window previewing the new configuration.

Problem

A possible cause is that the resulting configuration is invalid and cannot be displayed, perhaps because the same name appears in two different positions. As an example, the config below lists two Long1.0B masks with different barcodes. Because the names are the same, DREMEL fails.

	SLITMASK2: Long0.7 BC=8506
	SLITMASK3: Long1.0B BC=9090
	SLITMASK4: Long0.8 BC=7441
	SLITMASK5: LVMslitG BC=8850
	SLITMASK6: Long1.2 BC=7359
	SLITMASK7: Long1.5 BC=7360
	SLITMASK8: GOH_X BC=2126
	SLITMASK9: E0b BC=9024-no change
	SLITMASK10: uds_ft BC=8852-no change
	SLITMASK11: Long1.0B BC=7358-no change
	SLITMASK12: GOH_X BC=2124-no change
	SLITMASK13: Empty_13 (Bad port)-no change

Solution

Inspect the file /local/kroot/data/deimot/dyna/dremel.cfg. Check the listings for filters, gratings, and slitmasks looking for identically-named elements in two positions of the same sytem.
If this fails to indicate the problem, then check the DREMEL log with the following command on a polo terminal:
```
view_logfile -tail dremel
```

Click the Review Config button again and check for messages like this:

ERROR duplicates in stage position data!
These seem to be duplicated
GRATE.nam.600ZD GRATE.nam.600ZD GRSEL.nam.600ZD GRSEL.nam.600ZD
FATAL, CANNOT PROCEED
# found duplicates in stage data, cannot process

If identical entries are found, fix the problem by either:
- Removing the duplicate from the instrument if there are two masks for instance with the same name.
- If duplicate names are not in the instrument, then try re-scanning the elements in the affected stage.
Check tha DREMEL works as expected:
- On the DEIMOS control GUI, click on the DREMEL Details... button. This will launch the DREMEL panel.
- Click on Review Config and verify that it runs and displays the new configuration properly.

DREMEL can't access database

Symptom

DREMEL dies while attempting to initialize, or at the completion of a slitmask scan when attempting to read recods from slitmask database. The DREMEL panel on the DEIMOS control GUI indicates an unusual failure on the top status line.

Problem

The summit instrument Sybase server is not responding properly and requires a reboot.

Solution

Use ping waiaha to determine whether waiaha is alive.
Have summit staff determine whether waiaha is functioning properly, and reboot if needed.
Restart DREMEL by logging into keamano as kics and running:
```
deimos stop dremel
```
```
deimos start dremel
```