Table of contents

Image Quality

Double peaked images

Symptom
Double peaked images.
Problem
Possibly due to Pupil Mask Rotator (PMR) not operating.
Solution
Check the NIRC2 status gui. If you are using one of the Hex patterns, and either tracking is false, or, the angle is fixed at 0.0, then likely the problem is due to misalignment of the hex mask. If tracking is false, start tracking with "pmrrot on". If tracking is true, but the angle is stuck at 0.0, restart the daemon with nirc2 start pmr on a waikoko terminal.

Elongated images

Symptom
Elongated images in off axis acquisitions. On axis acquisitions appear normal.
Problem
Rotator mode is not set to Position angle. This can happen if you realign the laser at zenith using ACAM after releasing NIRC2 for observing.
Solution
If the rotator position angle is set to stationary or vertical angle, we need to reset it. In a vm-nirc2 terminal, issue the command rotate 0 posang to set the rotator to 0 in position angle mode. Check that DCS inidcates the rotator is now in position angle mode. Acquire a test image to check image quality.

Chevron noise pattern

Symptom
Chevron noise pattern as seen in left hand side of the two images shown below (displayed with ds9 using histeq).

Problem
Possibly caused by the filter animatics motor (most likely motor 5, the outer filter wheel) staying in an energized state because it did not quite reach its destination. This seems to happen more frequently when selecting narrow band filters (e.g., Hcont, Kcont).
Solution
At a vm-nirc2 prompt, type user eng, followed by exxon. This moves the outer filter wheel 1 count relative to its current position, forcing it into a motor off state. A dark should look more like the nominal noise pattern that is shown on the right-hand side of the above ds9 figure.

Vignetted image

Symptom
Vignetting over 1/3 of the array is apparent in wide camera images. Much lower vignetting may also be present in the narrow camera, but the narrow camera effects are difficult to detect.

Above: Wide camera flat image with vignetting present, and a diagonal cut across the image from upper left to lower right.

Problem
Slit aperture is partially in the beam.
Solution
Issue a sltclr in a vm-nirc2 terminal.

Crop circle in flat

Symptom
See a dark annulus that looks like a crop circle in a flat image like the one shown below.

Problem
The AO bench has not been configured for flats, and the AO hatch is closed.
Solution
At a vm-nirc2 terminal prompt, type configAOforFlats to set up the bench.

Server Problems

Exposure times are negative numbers or cannot access alad keywords

Symptom
Cannot show or modify alad keywords and the exposure meter is showing negative numbers or there are negative numbers in a waikoko xterm where the exposure command was issued.
Problem
There is something wrong with the alad server. Probably due to less than 3 alad servers running.
Solution
Restart the alad server (as nirc2eng on waikoko)
nirc2 stop alad
Wait 5 seconds
nirc2 start alad
Check servers using ctx or testAll on a vm-nirc2 terminal.
If this fails because motor and IO daemons not running, restart all three (io,motor,alad) with:
nirc2 start deamons
See below for possible need to re-home and reset cold head speeds.

RPC: Program not registered motor error

Symptom
"RPC: Program not registered" errors appear in response to motor commands.
Problem
Lost communication with the Animatics motor controller.
Solution
Procedure:
  1. Type ctx on a vm-nirc2 terminal to see if the motor and i/o daemons are already running. If they are, stop daemons, otherwise go to step 2. To stop daemons type on a waikoko
    nirc2 stop motors
  2. Start the motor and i/o daemons. To do that type:
    nirc2 start motors
    The motor daemon takes 30-60 seconds to start if all motors are connected and longer if one or more motors are not connected.
  3. Type ctx on a vm-nirc2 terminal to confirm that the motor and I/O daemons are now running, if not, call software support.
  4. Whenever the motor power is cycled, you have to use the motinit command on a vm-nirc2 terminal to execute the Animatics init subroutine, which setups up motion dynamics, before any commands will work.
  5. Use the following script on a waikoko terminal:
    tlog
    to watch the motor messages come scrolling out of the nirc2 log file.

Cannot show or modify motor keywords

Symptom
Cannot show or modify motor keywords (Unable to read NIRC2 THEAD2LO email alarm)
Problem
Likely due to power failure.
Solution
Restart the daemons and, if necessary reset cold speed and re-home motors.
  1. nirc2 start motors on a waikoko terminal (starts both IO and motor daemons)
  2. Check motors with showmot on a vm-nirc2 terminal (false values under home imply need to re-home)
  3. If motors are homed skip down to cold steps.
  4. Rehome the motors as follows:
    1. For all motors except camera and slit slides (i.e., FWI, FWO, GRS, SHR, PMR, PMS, PSI, PSO), on a vm-nirc2 terminal:
      1. mm xxxRESET=1
      2. mm xxxINIT=1
      3. mm xxxHOME=1
    2. For SLS, SLM and CAM, use the poshome command on a waikoko terminal after the reset and init. Check homelog.070220 to be sure behavior is consistent. Try not to home SLM and SLS to negative limits (see RC for details). Update the log if big changes.
    3. Reset cold head speeds (procedure)
  5. Some power failures have resulted in chiller-failure-and-then-CCR-off. If CCR off for approx. 2 hours or more, follow this procedure
  6. If the problem is specific to motor and io daemons (showmot and snaptemps hang); the problem may be with the emulex terminal server. Try telnet nirc2ts1. During troubleshooting 20aug2010 this connection attempt alone possibly resolved a comms problem with the io/motor daemons.
  7. A problem that has occurred about 8 times as of 01nov2010: Cannot show or modify keywords (or snaptemps or showmot). AND any new attempt to log into waikoko is very slow and takes about 5 minutes. Solution seems to be either wait it out (suspect some external NIS problem or other network service problem); or, reboot waikoko. To reboot waikoko follow: waikoko reboot procedure

Lakeshore communication problems

Symptom
Cannot read Lakeshore keywords (e.g., tempdet). First indication is often the email alarm: "Unable to read NIRC2 Lakeshore 340 keywords".
Problem
Serial communication occasionally locks up.
Solution
See procedure

Alad server crashed

Symptoms
Exposures stall or a detector command fails with information written to the terminal showing a combination of the following messages:
Unable to modify any of the detector parameters such as integration time, frame size or number of coadds.
Problem
The alad server has crashed.
Solution
  1. From the pulldown menu select NIRC2 Control → Server Subcomponents → Launch recover script . Alternatively, execute the command recover from a vm-nirc2 terminal.
  2. Answer Y at the prompt.
  3. The crash recovery script does the following steps:
    1. Kill Status Display, if it was already running
    2. Kill Quicklook, if it was already running
    3. Kill Program Identification GUI (PIG), if it was already running
    4. Stop nirc2plus service on vm-nirc2
    5. Stop nirc2history service on moolelo
    6. SSH to waikoko as nirc2eng to run restart_alad -noacs
    7. Run restore_alad script on vm-nirc2
    8. Restart nirc2history service on moolelo
    9. Restart nirc2plus on service vm-nirc2
    10. Launch PIG, if it was running before alad crash
    11. Launch Quicklook, if it was running before alad crash
    12. Launch Status Display, if it was running before alad crash
    13. Arrange NIRC2 GUIs
  4. The recovery script is completed once the message "Alad recovery script completed" on the terminal.
  5. If you launched the recover script from the VNC menu, then hit enter on the terminal to end the xterm session.
  6. If the NIRC2 Status GUI or the Quicklook did not restart automatically, then you can launch them from the VNC menu:
    • NIRC2 Control → Interface Subcomponents → NIRC2 Status Display
    • NIRC2 Control → Interface Subcomponents → Start QuickLook
  7. IMPORTANT: Once the recovery script is completed, Verify that the new Status GUI displays the last known detector parameters. In particular, make sure that:
    • The data directory is the same as before the crash.
    • The frame number is the next frame number on the data directory.
  8. If the data directory is not the same as before the sever crashed, then change the keywords to the correct values on a waikoko terminal :
    • modify -s alad outdir=/sdata904/nirc4/2020nov26/
    • modify -s alad frameno=146
    (The example above is ssuming that the alad server crashed when the data directory was /sdata904/nirc4/2020nov26/ and the last frame on the directory was 145.)

Cannot recover from alad server crash

Symptom
The recover script fails after an alad server crash
Problem
There is an alad_server_bin process that cannot be killed.
Solution
Procedure:
  1. Run the recover script on a vm-nirc2 terminal or from the VNC pulldown menu.
  2. If the above fails, on a waikoko terminal:
    nirc2 stop daemons
    nirc2 start daemons
    The output when starting the daemons fails is:
    Hunting alad_server_bin processes on waikoko...
    nirc2eng 1349 0:04 /kroot/rel/default/bin/alad_server_bin -noacs
    Kill -9 sent to pid = 1349.
    Testing transputer communication, try 1 of 3...
    Error : can't access resource [c1t4l0]
    
    Error - check - cannot access root transputer (result = -1).
    Transputer communication test failed
    Testing transputer communication, try 2 of 3...
    Error : can't access resource [c1t4l0]
    
    Error - check - cannot access root transputer (result = -1).
    Transputer communication test failed
    Testing transputer communication, try 3 of 3...
    Error : can't access resource [c1t4l0]
    
    Error - check - cannot access root transputer (result = -1).
    Transputer communication test failed
    
    Could not connect to transputers after 3 attempts.
  3. If the above fails, on a waikoko terminal:
    kill -15 [PID of stuck alad_server_bin process]
  4. If the above fails, on a waikoko terminal:
    kill -9 [PID of stuck alad_server_bin process]
  5. Check communication with the transputers using the check tspbits commands. This is how the output looks like when communication fails:
    waikoko{nirc2eng}128: check
    Error : can't access resource [c1t4l0]
    
    Error - check - cannot access root transputer (result = -1).
    
    waikoko{nirc2eng}129: tspbits 1 1
    Opening TSP link...
    Bypassing Transputer boot!
    Sending loopback command to transputer 1 for 1 iterations.
    Command failed at iteration 1
    There were 1 failures in 1 interations
    tsp test completed
  6. If staff is available on the summit, call for assistance to power cycle the NIRC2 computer room black box (BlackBox branded SCSI fiber optic extender).

    Run the recover script on a vm-nirc2 terminal or from the VNC pulldown menu.
  7. If the above fails, contact computer support to help rebooting waikoko followig this procedure.

AO Rotator Issues

Unstable NIRC2 compass roses and poiting issues

Symptoms
Problem
If this happens on a split night, it is possible the previous instrument interfeering with the AO rotator.
Solution
The Observing Assistant and/or Staff Astronomer will ensure that the previous instrument's rotator does not talk to the telescope control system, and will re-run the night-time setup for the AO system.

Issues observing vith the vortex cornagraph

Star off the vortex after QACITS sequence abort in PyWFS mode

Symptoms
Problem
QACITS uses the the PyWFS modulator to make the small corrections that are required to center the star on the vortex mask. If for some reason (e.g. faint object observation or variable background) the star is sent off the vortex and a QACITS sequequence is aborted, starting a new QACITS sequence without centering the PyWFS will fail.
Solution
Ask the Observing Assistant to rerun acquire star on the PyWFS acquisition GUI.

QACITS hungs due to aodmstat error

Symptoms
A QACITS sequence hungs with an error reading aodmstat.
Problem
A communications glitch between the machine where QACITS runs and the AO keywords server.
Solution
Stop and restart the QACITS sequence:
  1. Use CTRL-C on the terminal where QACITS is running to stop QACITS. Sometimes you need to issue several CTRL-C followed by Enter for the QACITS sequence to stop. Ultimately, you can close the vm-nirc2 terminal where QACITS is running.
  2. Once the QACITS sequence has been stopped, take a test image on a vm-nirc2 terminal with the goi or itest command to check if the star is still centered on the vortex.
  3. Center the star on the vortex, if needed.
  4. If you closed the vm-nirc2 terminal:
    1. Open a new one from the VNC background menu Login Windows → Xterm (local - nirc2).
    2. Type go_qacits at the terminal prompt.
    3. Start a new QACITS sequence.
  5. If you did not close the nirc2 terminal:
    1. Type retall at the IDL propmt.
    2. Start a new QACITS sequence.

NIRC2 Aladdin Quicklook freezes

Symptoms
The NIRC2 Aladdin Quicklook stops updating.
Problem
It is not clear why this occurs sometimes. It might be a timing issue if the quicklook tool tries to display an image before it has been written.
Solution
Restart the quicklook tool:
  1. Double-click on the iconified terminal QuickLook at the bottom of the VNC desktop. This will expand the terminal that runs the quicklook IDL software.
  2. Type exit at the IDL prompt.
  3. Restart the quicklook tool from the VNC background menu NIRC2 Control → Interface Subcomponents... → Start QuickLook.