SPEC detector not reading out

Symptoms
  1. A commanded exposure does not read out (e.g. goi)
  2. The detector dialog window (close to the SPEC Ginga Display) is frozen, or never shows "ready" in the output text
  3. NIRSPEC instrument control window shows an error or warning status below "science Camera" instead "Idling"
Problem
The NSPEC detector service needs to be re-started
Solution
Stop and Restart the NSPEC service from a local (observer's) xterm
  1. stop_spec
  2. start_spec
  3. Change the frame number to the next available frame number with "modify -s nspec framenum=[next available frame number]"
If you see errors when running the start_spec script, try the following sequence of command lines:
  1. stop_spec
  2. modify -s nsrestart nspec=start [wait ~5 sec]
  3. modify -s nspec init=1, wait for STATUS=Idling state, takes ~60sec
Problem
Detector is still not reading out properly (or at all)
Solution
Stop and Restart the NSPEC detector service, including a power cycle
  1. powerdown_spec
  2. start_spec
  3. Change the frame number to the next available frame number with "modify -s nspec framenum=[next available frame number]"

SCAM detector not reading out

Symptoms
  1. A commanded exposure does not read out (e.g. goi2)
  2. The detector dialog window (close to the SCAM Ginga Display) is frozen, or never shows "ready" in the output text
  3. NIRSPEC instrument control window shows an error or warning status below "Slit Camera" instead "Idling"
Problem
The NSCAM detector service needs to be re-started
Solution
Stop and Restart the NSCAM service from a local (observer's) xterm
  1. stop_scam
  2. start_scam
  3. Change the frame number to the next available frame number with "modify -s nscam framenum=[next available frame number]"
If you see errors when running the start_scam script, try the following sequence of command lines:
  1. stop_scam
  2. modify -s nsrestart nscam=start
  3. [wait for STATUS=Idle, takes ~60 sec]
  4. If no joy, then try modify -s nscam init=1, wait for Idle state
Problem
Detector is still not reading out properly (or at all)
Solution
Stop and Restart the NSCAM detector service, including a power cycle
  1. powerdown_scam
  2. start_scam
  3. Change the frame number to the next available frame number with "modify -s nscam framenum=[next available frame number]"

Missing Header Info in Fits files

Symptoms
Header information for SPEC or SCAM frames is missing or corrupted
Problem
The nsheader service is in a bad state
Solution
Stop and Restart the NSHEADERS service from a local (observer's) xterm
  1. modify -s nsrestart nsheaders=stop
  2. modify -s nsrestart nsheaders=start

A Red "x" shown in NIRSPEC GUI

Symptoms
A red "x" appears to the right of one or more given mechanisms in the NIRSPEC instrument GUI, example here
Problem
The mechanism has lost its known position.
Solution
Recalibrate the mechanism(s)
  1. Close the NIRSPEC hatch
  2. Recalibrate the affected mechansism. From a nirspec xterm: "init_all_motors". It will not re-calibrate mechanisms that do not need it.

Large shift on the spectra

Symptoms
  1. The spectral orders are shifted by more than half order with respect to their nominal location, and/or
  2. The arc lines on each order are shifted with respect to their nominal location.
See arc lamp plots for reference of the nominal location of the orders and arc lines in some of the most widely used filters.
Problem
Either the echelle or the cross-disperser or both did not reach the desired angle.
Solution
First try to determine if a Galil power cycle is needed. The fastest way to determine this is to recalibrate the SCAM filter wheel. Click on the slit camera filter status flag on the NIRSPEC mechanism GUI and hit the Recalibrate button. The recalibration should be fast.
  • If it does not find the switches right away, we are in a mode where a power cycle is requred.
  • If the recalibration is fast and successful, then proceed to recalibrate the mechanism that can be causing the shift on the spectra (typically either the echelle, the cross disperser or both).
If a power cycle is required, then use the following commands on the observer's xterm on vm-nirspec:
  1. modify -s nsrestart nsmotor1=stop
  2. modify -s nspower outlet_d3=off
  3. modify -s nspower outlet_d3=on
  4. modify -s nsrestart nsmotor1=start
Bring up the detail panel popup corresponding to each of the mechanisms that need to be calibrated by hitting the mechanism status flag on the NIRSPEC mechanism GUI. Re-calibrate each mechanism connected to the controller by hitting the Recalibrate button on the corresponding detail panel popup.
To see which mechanisms are connected to the Galil controller use the command: gshow -s nsmotor disp1_motor
To see the NIRSPEC instrument status, use the command: showpower

Rotator Error

Symptoms
  1. Rotator Status Indicator on Control GUI shows red
  2. Attempts to Halt or Init the rotator from the GUI do not fix it
Problem
The rotator mechanism needs to be controller needs to be re-started
Solution
Power cycle and restart the Galil controller using commands issued to the observer's xterm on vm-nirspec
  1. modify -s nsrestart nsmotor1=stop
  2. modify -s nspower outlet_d3=off
  3. modify -s nspower outlet_d3=on
  4. modify -s nsrestart nsmotor1=start
  5. Re-initialize the motor: from an xterm on vm-nirspec, "init_all_motors" (it will skip ones that are already initialized)

Motors will not initialize or move

Symptoms
  1. NIRSPEC mechanism GUI shows a red state for some or all of the motor, example here
  2. Attempts to calibrate a single mechanism does nothing (e.g. Clicking on the mechanism detail from the GUI brings up a sub-screen, then click on "re-calibrate" - no activity
  3. Attempts to calibrate (i.e. initialize) the affected motors from the command line does not succeed: 1) close the NIRSPEC hatch, 2)"init_all_motors"
Problem
One of the NIRSPEC calibration lamps could be on (Arcs or Flatlamp). There is a safeguard to prevent motion of some mechanisms while any lamps remain on. Turn off the lamps via the NIRSPEC mechanism GUI and try the move again.

If still trouble, then one or both of the servers that run the motor mechanisms need to be restarted. There are 2 mechanism servers: nsmotor1 (internal mehcanisms) & nsmotor2 (cal unit mechanisms). This may likely happen immediatly following a reconfig of the instrument in or out of AO, where the gets dicsonnected and reconnected.
Solution
Bring up the Event Monitor GUI to see where the problem is: from a local xterm on vm-nirspec: "vis_emir nsmon"
  1. If needed, restart nsmotor1 and/or nsmotor2
  2. ssh nspeceng@nirspecserver
  3. from an xterm type: "nirspec restart nsmotor1"
  4. from an xterm type: "nirspec restart nsmotor2"
  5. Re-initialize all motors: from an xterm on vm-nirspec, "init_all_motors" (it will skip ones that are already initialized)

Motor never stops slewing

Symptom
A motor move has been commanded, and it starts, but it never finishes. There is no message that the move failed or timed out, but the NIRSPEC Mechanism GUI shows the gear icon (moving) forever and never shows a ready state.
Problem
The mechanism is lost and needs to be re-inialized.
Solution
From the NIRSPEC Mechanism GUI, open the sub-page details by clicking to the right of the mechanism name that has the issue.
  1. Click "stop motion"
  2. Click "re-calibrate"
  3. Re-try: From the top level NIRSPEC Mechanism GUI, click the mechanism setting, and select a new value from the pulldown, and click apply.
  4. If the move fails again. You will need to restart one or both of the motor servers (see above).

The keywords covered by [stage]abort are:

  • rotatorstp abort image rotator movement
  • scifilt1stp abort SPEC filter 1 wheel movement
  • scifilt2stp abort SPEC filter 2 wheel movement
  • scamfiltstp abort SCAM filter wheel movement
  • slitstp abort current slit wheel movement
  • echellestp abort echelle mechanism movement
  • xdisperserstp abort cross disperser movement
  • calfoldstp abort calibration mirror movement
  • pinholestp abort calibration pinhole movement
  • hatchstp abort hatch movement

So, for example, to halt the Echelle grating, the command would be:

modify -s nsmotor echellestp=1

The keywords covered by [stage]init are:

  • rotatorcal init image rotator movement
  • scifilt1cal init SPEC filter 1 wheel movement
  • scifilt2cal init SPEC filter 2 wheel movement
  • scamfiltcal init SCAM filter wheel movement
  • slitcal init current slit wheel movement
  • echellecal init echelle mechanism movement
  • xdispersercal init cross disperser movement
  • calfoldcal init calibration mirror movement
  • pinholecal init calibration pinhole movement
  • hatchcal init hatch movement

So, for example, to init (i.e. calibrate) the Echelle grating, the command would be:

modify -s nsmotor echellecal=1

Tracking Directions in Normal Operation

In normal operations, the image rotator tracks in different directions depending on the location of the object in the sky:

The Rotator GUI software is smart enough to be able to find the correct slit position on the sky and have the rotator track accordingly. Once you input the Rotator Mode (i.e. position angle) and the Slit PA destination value, you can see where the physical limits are for the rotator as it slews to the tracking position (left panel on the GUI). If the positon is close to a limit (i.e. 90 or 270 physical) and tracking towards it (shown by the yellow directional arrows), you may want to consider the 180-degree complement of your desired PA to ensure you will have a safe amount of time to observe before reaching the rotator limit.

No light in focus (Mira) images

Symptom
There is no light in focus (Mira) images
Problem
Either the cal unit is incorrectly configured or the wrong filter is in the beam.
Solution
MIRA was designed for Near-IR filters. All the numbered SPEC filters (e.g. NIRSPEC-1 through -7) are parfocal.
  1. Ensure the Nirspec Hatch is open, the Cal Pinhole is out, and the Cal Mirror is out. If necessary, ask the observer to re-send an instrument "Setup Only" script from the EFS.
  2. Ensure the SPEC selected filter is one of the following: NIRSPEC-7, NIRSPEC-6, NIRSPEC-5, NIRSPEC-4, NIRSPEC-3, NIRSPEC-2, or NIRSPEC-1, with either the "thin" or "open" filter selected with it.
  3. Ensure the SCAM selected filter is "Shortpass".