Images

Charge Persistence on Detectors

Symptom

Remnant images appear in subsequent images.

Problem

Charge persistence in detectors (both SPEC and SCAM can get persistence, but this is more of a problem with SPEC because it is the main science detector).

Solution

Avoid it by not exposing above 6000 counts (DN aka ADU) in a single SPEC coadd. Remove by flushing and waiting

1. If exposures are kept below 6000 counts per coadd, there is little, if any, charge persistence.
2. Remove by flushing: type flush from a waimea prompt.
3. There is a component of the persistence that only goes away with time, even after repeated flushes.
4. For more info, see the charge persistence webpage.

One Quadrant appears Dead

Symptom

One quadrant of the SPEC detector looks dead

Problem

NIRSPEC server started in a strange state

Solution

Restart the server:

1. Right-click in background and run: NIRSPEC Control Menu -> End of Night Shutdown
2. Right-click in background and run: NIRSPEC Control Menu -> Start NIRSPEC Control Software

No light in calibration images

Symptom

Calibration spectra have no light.

Problem

Calibration pinhole is in the beam.

Solution

Remove the pinhole

1. Type pinholeout from a waimea prompt
2. Wait 15 seconds.
3. Take a short (0.1 second) SCAM image to see if light is falling down the slit.
4. For more info and images, please see the "no light" webpage.

No light behind AO

Symptom

There is no light from the AO fiber in images

Problem

AO pupil (in filter wheel 1) may have moved

Solution

Return the AO pupil to the beam:

1. Type set_ao_pupil from a waimea prompt
2. When complete, take a short (0.1 second) SCAM image to ensure light from the AO fiber is visible.
3. For more info and images, please see the "no light" webpage.

Weird-looking spectra

Symptom

Your spectra do not look right. See examples here.

Problem

A low-res slit is in beam with the echelle in high-res mode or vice versa

Solution

Re-select the slit or echelle position.

1. If you have doubts, use the echelle format simulator (EFS).

Computers

waimea prompt

Symptom

Troubleshooting requires access to the NIRSPEC host computer

Problem

Need to open a window on waimea

Solution

Open a waimea window:

1. Right-click in background --> Login Windows --> xgterm 'waimea' (NIRSPEC)

no more processes

Symptom

waimea becomes unresponsive, cannot take images, move motors or even type simple commands such as ls.

Problem

The process table for user "nirspec" has filled on waimea

Solution

Reboot waimea -- although that may be hard!

1. Log into waimea under a different account like teloper
2. Perform a soft OA Reboot
3. If you cannot log in as another user, perform the following until you do get a working waimea prompt:
   1. Quit NIRSPEC GUI's (e.g. XNIRSPEC, Echelle Format Simulator, Quicklook, Image Rotator GUI, Slitnod Widget)
   2. Kill the window that says "DO NOT KILL THIS WINDOW"
4. Perform a soft OA Reboot

Soft reboot of waimea

Symptom

NIRSPEC and waimea are unresponsive and other troubleshooting steps have told you to perform an OA Reboot.

Problem

waimea, the NIRSPEC instrument host, is sick inside

Solution

Reboot (NOT to be done lightly!)

1. Log into waimea as teloper
2. Execute the command /etc/oareboot
3. Wait a couple of minutes
4. ping waimea or ping waimea.keck.hawaii.edu from another machine until you see the message: waimea is alive
5. Before you can observe, there are a few things you must do.

After a reboot

Symptom

You have just rebooted waimea

Problem

NIRSPEC is not yet ready to observe

Solution

Check the communications chain and health of instrument

1. Log into waimea as nirspec
2. Execute the command iboot from a waimea prompt (this tests the communication chain)
3. If the output looks like this:

                         setting nirspec_mbox_query = 1 (wait)
                         nirspec_mbox_status = ON
                         setting nirspec_oibb_query = 1 (wait)
                         nirspec_oibb_status = ON
                         setting nirspec_crbb_query = 1 (wait)
                         nirspec_crbb_status = ON
                         

   Then start up the NIRSPEC software and continue observing.
4. If not, run iboot again.
5. If you still don't see the proper output, execute the command iboot all. This should clear most communication problems. The output should show you the "ON" status from all 3 iboots.
6. Start up NIRSPEC control software
7. Click here if you have problems starting the server, or you see error messages in the startup scripts.
8. Once the NIRSPEC software is up and running, issue the following commands from a waimea prompt:
   1. m test2=1
   2. cp /tmp/TESTFITS2 /tmp/TESTSKY2

iBoots do not respond

Symptom

After running iboot from a waimea prompt, you do not receive an "ON" response from one or more of the iBoots

Problem

The network router connected to the non-responsive iBoot needs a power cycle. This most often happens to the iBoot connected to the computer room blackbox.

Solution

Ask a summit tech to power cycle the router for the iBoot (it is NOT necessary to power cycle the iBoot itself).

• If the tech is unfamiliar with this procedure, photos and more details are located on the iBoot webpage.

Software

XNIRSPEC Crashes

Symptom

XNIRSPEC disappears or stops updating.

Problem

Unknown.

Solution

Restart XNIRSPEC

1. Right-click in background --> NIRSPEC Control Menu --> Restart XNIRSPEC
2. If a SPEC exposure is in progress, wait for it to read out.
3. While waiting for the SPEC exposure, try a SCAM exposure just to see that XNIRSPEC is functional
4. If this fails, the NIRSPEC server may have crashed. Follow the server recovery instructions here.

Quicklook Crash

Symptom

Quicklook does not respond, its IDL window indicates a crash

Problem

Too many popup functions open at once, unknown causes

Solution

Restart Quicklook

1. Exit both Quicklook tools
2. Right-click in background --> NIRSPEC Control Menu --> Restart Quicklook

NIRSPEC Server Crash

Symptom

1. tklogger popup indicating server crash
2. Cannot take images
3. Cannot show keywords

Problem

The NIRSPEC server (nirspec_server_bin) has crashed

Solution

Run the recover script.

1. Right-click in background --> NIRSPEC Engineering --> Recover from server crash
2. Follow prompts of recover script
3. Run whenever you think there may be a problem with the NIRSPEC server.
4. N.B. take any calibrations you need before reinitializing motor stages.

Image never displays

Symptom

User takes an image but Quicklook does not display it.

Problem

Usually one of two reasons:

1. itime x coadds less than 1 second
2. Quicklook has crashed

Solution

Solution to Problem #1

1. Increase the itime or coadds so that their product exceeds 1 second.
2. N.B. that the image is not lost. If a "test" image, it will be in the /tmp directory. If a "go" image, it will be in outdir or outdir2.

Solution to Problem #2: Restart Quicklook

1. Exit both Quicklook tools
2. Right-click in background --> NIRSPEC Control Menu --> Restart Quicklook

Quicklook popups do not respond

Symptom

Cannot access Quicklook's popup features (Gaussian Fit, Diagonal Cut, etc.

Problem

More than one popup feature open (known Quicklook bug) or Quicklook has crashed

Solution

Close all popup features, try again

1. Click on DONE buttons of all open popups
2. Try to access the desired features one at a time.
3. If this doesn't work, restart Quicklook.

Quicklook "Move Tel" not working

Symptom

"Move Tel" gives the first prompt ("Click Mouse on Start Point") but never returns with the second prompt ("Click Mouse on End Point").

Problem

Conflict with "Diagonal Cut"

Solution

Dismiss the Diagonal Cut popup

1. Click on the DONE button within the diagonal cut popup.
2. If this doesn't work, restart Quicklook

Server does not start properly

Symptom

Error messages in startup script windows such as "RPC timeout". They may look like this:

                  rpc_clientOpen: RPC: Program not registered
                  Sorry, the show command was not able to contact the
                  control system: rpc_clientOpen() for waimea.keck.hawaii.edu.
		  

Problem

The server did not start correctly, communications chain may be interrupted

Solution

Restart the server

1. From a waimea prompt: kill_all (tries to kill all server processes)
2. Once the prompt is returned: iboot all (power cycles elements of the communication chain)
3. Once that prompt is returned: check (talks to the transputers)
4. Type at the prompt: runserver (starts the NIRSPEC keyword server). The output should look like this:

                  Hunting caRepeat processes on waimea...
                  nirspec 25012
                  Kill -9 sent to pid = 25012.
                  Checking for existing servers (real and simulated)...
                  Running server program.
                  [1] 25191 25192
                  Setting Configuration file values.
                  ==================================
                  setting q1offsetspec = 3240 (wait)
                  setting q2offsetspec = 3240 (wait)
                  setting q3offsetspec = 3240 (wait)
                  setting q4offsetspec = 3240 (wait)
                  setting q1offsetscam = 729 (wait)
                  setting q2offsetscam = 725 (wait)
                  setting q3offsetscam = 724 (wait)
                  setting q4offsetscam = 722 (wait)
                  setting tsptrace = 0 (wait)
                  setting detbias = 450 (wait)
                  setting getcryotemp = 1 (wait)
                  setting getdetectortemp = 30 (wait)
                  setting sensor.read = 30 (wait)
                  setting outdir = /sdata600/nirspec (wait)
                  setting outdir2 = /sdata600/nirspec (wait)
                  

5. N.B. that the "setting ..." lines should start streaming out within 1 second of the line "Setting Configuration file values.". If not, execute a kill_all and then restart the server.
6. Start the NIRSPEC control software via a right click in the background --> NIRSPEC Control Menu --> Start NIRSPEC Control Software

Mechanism Moves

Motors will not initialize or move

Symptoms

1. NIRSPEC never acknowledges that initialization is complete
2. Motors do not move
3. Some or all cryotemps are not reporting cryotemps from a waimea prompt

Problem

The Lakeshore 330 temperature controller has been set incorrectly. The correct communication rate is 300 bps. Logic in the motor control software prevents motor moves if the temperatures are unknown or outside acceptable ranges.

Solution

Reset the Lakeshore 330 to the correct communication rate

1. Kill the NIRSPEC server.
2. Ask a summit tech to power cycle the Lakeshore 330.
3. The communication rate can be set from the Lakeshore unit.
4. Verify that the temperatures are reporting (temps should cycle on the front display of the Lakeshore 330).
5. Restart the NIRSPEC server.

Motor never stops slewing

Symptom

A motor move has been commanded, and it starts, but it never finishes. There is no message that the move failed or timed out, but the XNIRSPEC GUI reports that the mechanism is "slewing" forever.

Problem

A glitch in the communications between the server and the transputers. This problem occurs very infrequently.

Solution

Abort and re-init the offending mechanism

1. Abort: type m [stage]abort=1 from a waimea prompt.
2. Re-init: type m [stage]init=1 from a waimea prompt.
3. Re-try: Use XNIRSPEC to send the move again to the offending mechanism.
4. N.B. If the above procedure fails, a full restart of the server is necessary to fix this problem.

The keywords covered by [stage]abort are:

• irotabort abort image rotator movement
• fil1abort abort filter 1 wheel movement
• fil2abort abort filter 2 wheel movement
• slitabort abort current slit wheel movement
• echlabort abort echelle mechanism movement
• dispabort abort cross disperser movement
• calmabort abort calibration mirror movement
• calpabort abort calibration pinhole movement
• calcabort abort calibration cover movement

So, for example, to halt the Echelle grating, the command would be:

m echlabort=1

The keywords covered by [stage]init are:

• irotinit init image rotator movement
• fil1init init filter 1 wheel movement
• fil2init init filter 2 wheel movement
• slitinit init current slit wheel movement
• echlinit init echelle mechanism movement
• dispinit init cross disperser movement
• calminit init calibration mirror movement
• calpinit init calibration pinhole movement
• calcinit init calibration cover movement

So, for example, to halt the Echelle grating, the command would be:

m echlinit=1

Rotator

Poor slit nodding #1 (rotator server restart)

Symptom

The position angle is incorrect, the rotator will not move, or TkLogger pops up with repeated rotator warnings.

Problem

The rotator server has died.

Solution

Follow the TkLogger directions:

1. Right-click in background and run: NIRSPEC Control Menu -> Restart Image Rotator
2. Wait 5 seconds or so
3. Turn rotator tracking on from XNIRSPEC: IROT -> Tracking On
4. Reset your desired rotator mode and angle from the Rotator GUI.

The object should still be in the slit, but you will want to check it.

Poor slit nodding #2 (rotator needs initializing)

Symptom

You see a message in XNIRSPEC suggesting you reinitialize the rotator or restarting the rotator server did not fix the problem.

Problem

Rotator has lost motor steps or tracked into a limit

Solution

Reinitialize the rotator

1. Tell OA you need to reinitialize the rotator and ask them to stop guiding
2. Within the XNIRSPEC GUI: Click Engineering --> Motors --> Init --> Image Rotator --> Init --> Dismiss
3. The status box beneath the rotator icon in XNIRSPEC should read INIT. Once this status box reads OK, reset your desired Physical or Position Angle from the rotator GUI

Red box on GUI never turns white

Symptom

You sent a rotator command, but the indicator box on the rotator GUI never turned from red (off target) to white (on target).

Problem

Many possible problems including:

1. IROT tracking not started
2. Rotator software crash
3. Rotator slewed into a limit

Solution

Solutions given in order:

1. (Re)Start IROT tracking
   1. Within the XNIRSPEC GUI: Click IROT --> Tracking Off
   2. Within the XNIRSPEC GUI: Click IROT --> Tracking On
   3. Within Rotator GUI: Type in the Physical angle set box: 0.0 and click SET.
   4. The red box should turn white within 15 seconds.
   5. Set your desired Physical or Position Angle.
2. If the above doesn't work, restart rotator software
   1. Right click in background --> NIRSPEC Control Menu --> Restart Image Rotator
   2. Reset your desired Physical or Position Angle from the rotator GUI.
3. If that doesn't fix it, reinitialize the rotator mechanism

Position Angle is Incorrect

Symptom

The desired position angle is incorrect after a telescope slew

Problem

The rotator does not move during telescope slews

Solution

Reset the position angle

1. If guiding, ask the OA to stop guiding.
2. Type in your desired position angle into the rotator GUI.
3. Click SET
4. If this does not fix your problem, you may need to restarting the image rotator server or reinitializing the rotator mechanism may help.

SCAM GUIDER

CAM failure: can't open FITS file.

Symptom

XGuide gives the error: "CAM failure: can't open FITS file."

Problem

The file /tmp/TEMPSKY2 has been deleted on waimea

Solution

Create a new TEMPSKY2 file (OA and Observer cooperation)

1. Stop guiding
2. Take a short test SCAM exposure
3. Issue the following command from a waimea prompt: cp /tmp/TEMPFITS2 /tmp/TEMPSKY2
4. Start guiding

Other "CAM failure" messages

Symptom

XGuide reports a CAM failure other than the above case

Problem

The cam server task didn't start correctly

Solution

Restart XGuide

Cannot get centroid on SCAM

Symptom

XGuide complains that it cannot centroid on a star, even if the star is relatively bright and unsaturated.

Problem

The sky background image may be old yielding an uneven background within the guide box

Solution

Make a new sky background

1. Offset the telesccope ~30 arcseconds to move the star off SCAM (FOV 46 arcseconds)
2. Click Make Sky on the NIRSPEC Eavesdrop GUI
3. Undo the offset
4. Try guiding again

PXL/Annular/NIRSPEC Guider

Electronics PXL Troubleshooting pages

Symptom

PXL guider problems

Problem

Need the Electronics Department's PXL troubleshooting page

Solution

Click the links:

1. PXL Troubleshooting Start Page
2. PXL Troubleshooting Index Page
3. PXL Troubleshooting Imaging Page
4. PXL Troubleshooting Communications Page

PXL Camera too warm

Symptom

OA sees warnings from cam_server task running on waimanu aka k2camserver1:

waimanu cam_server [pid]: pxl current temp too warm [T]

where [T] is the temperature in degrees C multiplied by 100. N.B. that the normal operating temp is -25 C.

Problem

Communications glitch between camera head and controller

Solution

Have the OA reset the controller:

1. Kill the XGuide or kcamtest software.
2. Go up to the instrument, and go around to the right side of the instrument as viewed from the back (i.e., the right as you stand at the back of the instrument looking in towards the tertiary mirror).
3. Remove the short middle insulation cover to gain access to the PXL CEU2 controller box.
4. On the CEU2 controller box, there is a square button that should normally be illuminated green, and an LCD display that gives the temperature.
5. If the button is red, note what the LCD display says then hit the button to reset the camera.
6. If the button returns to green, wait and watch the LCD screen. After a few moments, the temperature should gradually begin to drop towards the setpoint of -25C.
7. Watch it for a minute or so to verify that it is dropping, then replace the insulation cover and go back to a warmer place.
8. NOTE: If at any point in the procedure the controller is not behaving as described above, call Rich Matsuda for further directions.

Mira

No light in focus (Mira) images

Symptom

There is no light in focus (Mira) images

Problem

Either the cal unit is incorrectly configured or the wrong filter is in the beam.

Solution

Mira was designed for K-band or K-like filters

1. Ensure the Cal Cover is open, the Cal Pinhole is out, and the Cal Mirror is out.
2. Ensure the selected filter is one of the following: NIRSPEC-7, K, K-PRIME, NIRSPEC-6, NIRSPEC-5 (H); each with the Thin Blocker also in place.