AO Troubleshooting

WWW Optics pages

AO Performance

User Guide for Optimization Tool (also named bandwidth widget)

The optimization tool has its dedicated web page.

Checklist for bad AO performance

Symptom
The image quality is bad
Possible Problems
The list of possible problems includes:
  1. Bad or fast seeing
  2. Telescope windshake or telescope vibrations, low elevation
  3. Problem with the WFS background
  4. Problem with the reconstructor
  5. Vignetted pupil
  6. Tip-tilt loop unstable
  7. Tip-tilt offloading problem
  8. Bad calibrations
  9. ACS snap
Solution
Note that proper troubleshooting should be performed on a 10th magnitude B-A type star (NGS) or one of the LGS checkout stars.
  1. Measure the seeing using the atmospheric characterization tool, acamtool, open loop images on NIRC2 or using the LBWFS (read the aolbfwhm keyword).
  2. Use the bandwidth widget to optimize the TT and DM gains and check for any problems with vibrations.
  3. Record new background
  4. Trigger a new reconstructor using MAORI→WFS with both loops closed and check that reconstructor updates after a few seconds. If not, then restart the reconstructor from MAORI→Menu. Check the pupil on the WFS intensity display with the reconstructor mask on and off (the pupil should look the same).
  5. Check windshake, check tracking errors for el and az
  6. Check for possible vibration produced by the secondary vacuum valve, the telescope drives, etc
  7. Check that tip-tilt mirror is powered up.
    • N.B. if the TT graph shows the mirror position changing, it is VERY likely powered up.
  8. Check for problem with the tip-tilt mirror (broken actuator?)
  9. Check for problem with the DM using the WYKO interferometer. Open the WYKO hatch and look at the WYKO when the DM loop is opened or closed. You should see the DM move constantly in closed loop. You should also see that it is not producing strange shapes, such as waffle or a lot of focus.
  10. Check for calibrations: DM to lenslet, COG, SFP home problem, WLS centering, FSS vignetting, etc.
    COG file
    If you see poor image quality, you can try an old COG file (this is quick and can be done with loops closed)
    Current COGOld COG
    In this case, we decided the old COG was better than the current one.
    We made the old one the default by:
      as kNobsao@kNaoserver
    1. cdcog
    2. cp -p 24NIRC2-N2x2.cog31Aug2016 24NIRC2-N2x2.cog
    3. The change takes effect when one next runs "Setup Bench" from AOAcq
  11. Check ACS snap
    If you suspect the cals are fine or you see non-diffraction limited performance, check the snap that ACS is using (OA's are very helpful here)
    Bad (DEIMOS) SnapGood (PCS) Snap

How do I measure the seeing

Symptom
Need to measure seeing
Problem
How?
Solution
  1. Using the atmospheric characterization tool
  2. Using the science camera: pick a fairly bright star (7 < V < 10), open DM loop, close TT loop (or open both loops).
    tint 1; coadd 20; filt kp; wait4ao off; goi
    Measure FWHM. In IDL, type ttcomp,0.45 or whatever the FWHM is.
  3. Using ACAM: you may use the IDL acamtool and record a 1-3sec exposure. The tool will display the FWHM for star which gives an estimate for the seeing in the V band. The real star is at the bottom, the ghost image is always on top (the opposite than the guider).
    In NGS AO mode:
    Make sure that xguide is IDLE. Set the integration, and record image. The tool will automatically display the star in the sub-frame to the left, and a model to the right. The value for the FWHM is displayed on the tool. More values for the photometry is displayed in the xterm window.
    In LGS AO mode:
    The tool shows up in this mode if the AO system is configured for LGS AO mode. Same use than in NGS-AO, make sure that xguide is IDLE. Set the integration, and record image. The tool will display more option to move the stages to be focused either on the NGS or for the LGS. It also gives the possibility to acquire manually the laser.
    Or estimate the FWHM from the guider display (less accurate?).
Seeing sizes smaller than 0.8'' and 0.5'' in V are considered respectively fair and excellent. The AO correction gets worse as the seeing increases and the image quality really suffers for seeing larger than 1'' in V.
The important information for seeing is not only the spatial scale but also the temporal scale (correlation time). The temporal scale for seeing can only be measured using dedicated tools that we lack at Keck Observatory.
You can also check for seeing estimate from the Mauna Kea weather web page. They display seeing forecast as well as distribution of wind layers and their velocity. The stronger the wind associated with a turbulent atmospheric layer, the worse the shorter the atmospheric decorrelation time. This is especially important when the WFS is run at a low frame rate. If the jet streams are shown in proximity to the Island, the seeing is very likely to be fast and large.

Seeing is either large or very fast, what should I do?

Symptom
In case of bad (fast or large) seeing, there are various setup tricks that may help.
Problem
AO performance may not be optimal, how can I make sure it is?
Here are some tips that may help when bad seeing is resulting in bad image quality.
Solution
  1. Adjust the centroid gain by changing the seeing setting in the AO Acquisition widget.
  2. Check AO performance using the bandwidth widget. If seeing is faster, it may be that our lookup table is way off... increasing the gains or the frame rate may help.
  3. Adjust the tip-tilt offloading parameters (see below)
  4. Trigger a new reconstructor from the MAORI→WFS tab with the loops closed and check that the reconstructor updates. If not, restart the reconstructor from MAORI→Menu.

The seeing is OK but LGSAO Performance is NOT, TT gain is very low (less than 0.1)

Symptom
Elongated images or generally poor performance.
Problem
TT gain maybe set too low due to worsening seeing as compared to the start of the night , Use Optimization Tool (also named bandwidth widget)
In some cases the auto gain setting logic fails and sets the TT gain too low resulting in bad image quality. This can happen when the seeing is exceptionally good at a the star of the night but gets worse later in the night. The auto gain setting uses the Tubulent power spectrum, Turb PS , sample from the AO Checkout, if the seeing (turbulence) changes, the Turb PS sample and the gain may need updated.
Solution
  1. Select STRAP in the Optimizer widget
  2. Select DIAGNOSTIC in the Optimizer widget
  3. If current TT Gain (see MAORI) is different than the suggested gain (see Xterm associated with Optimizer tool) Adjust the TT loop gain manually in MAORI or button on the Optimizer widget
  4. Repeat DIAGNOSTIC and gain ajust until no change is suggested.
  5. Enter DIAG number into Turb PS box (turbulent spectrum sample)
  6. Current gain should be good and auto gain setting for subsequent targets should be improved.
  7. For more info, The optimization tool has its dedicated web page.

Checking the WFS centroid origin calibrations

Symptom
Bad AO performance, even for bright star (Vmag<10). Artifacts in the image.
Problem
Bad calibrations or corrupted file for the centroid origins?
Solution
  1. From an xterm on k2aoserver as k2obsao, type cdcog
  2. Then dir will list all centroid origins calibration files sorted by date.
  3. The name reflects the instrument and the date. For a NIRC2 night in NGS AO mode with the 2.4 arcsec plate scale and 2x2 binning the default file would be 24NIRC2-N2x2.cog. Find a file with the same name but a different date appended (e.g., 24NIRC2-N2x2.cog16Feb2007) that you trust. In this example,
  4. \cp -p 24NIRC2-N2x2.cog16Feb2007 24NIRC2-N2x2.cog
  5. In IDL, type LOADCOG,'24NIRC2-N2x2.cog'
  6. The new calibration file should be loaded on the system.
  7. try other calibration files if necessary or use different binning modes or plate scale.

Stars appear elongated, what should I do?

Symptom
Stars appear elongated on science detector.
The TT mirror and TT residual graph could show a lot of jitter.
Telescope servo-error tracking values could be large.
Problem
Many possible reasons, but the most likely are:
  1. Tip-tilt mirror oscillations.
  2. Problem with TT offloading: could happen when star goes through meridian.
  3. Wind shake: do you see the same effect when you point into the wind vs other directions?
  4. Lots of vibrations (produced by the secondary-mirror vacuum valve?): experience has shown it could be worse at lower elevation
  5. Reference star is a binary or has a peculiar morphology: 2-5 arcseconds binary used as a reference star may produce some elongation. If it is a close binary (less than 100 mas separattion) then the image may appear to be elongated.

The Guide Star is a 2"-5" binary (NGS only)

Symptom
We are guiding on a 2 to 5 arcsec binary of comparable magnitude and the image quality is poor.
Problem
Both stars are in the wavefront sensor, so the star is not centered on the cross hairs of the wavefront sensor.
Solution
A possible solution is to introduce a smaller field stop in front of the wavefront sensor.
  1. Open the loops using MAORI
  2. From MAORI → WFS, left click on the entry by the FSS row and select "0.6". FSS is the field stop stage.
  3. Once FSS reads back "0.6", you may reclose the loops and continue.
  4. Next time a star is acquired, the field stop will automatically go back to the "2.4" named position.

Telescope/AO interactions

AO/DCS communications: general

The AO/DCS communication is crucial to AO observing for:

Dither failure: light lost on WFS:

Symptom
Observers reports a dither failure: after/during a dither with NIRC2 or OSIRIS, the star is lost: the light is not seen on the WFS or STRAP anymore.
AO loops may be open or close at that point.
Problem
Dither failure may happen rarely. It is AO/DCS problem that is quite complex to investigate on the spot.
Solution
Open AO loops immediately!
We recommend to follow this procedure when a dither failure happens as
  1. Write down UT time and include the time stamp in the ticket.
  2. Re-acquire the target
    1. by re-sending the "move" command from sky.
      We believe this is the only way for now to reset the TT offloading control point to (0,0).
    2. By adjusting pointing on REF if necessary
    3. By sending object to the default DCS pointing origins (nirc2 for NIRC2, opt-axis for IF)
  3. Restart "AO/DCS communication"
  4. Once you see the target on the WFS, no need for a new background, just close TT and DM loop, and you should be ready to go.
  5. Notes:
    • If you had been on this target for a while, record a new background WFS and close loops.
    • You could re-run the AO setup script again for this target, but it is unlikely that would fix the dither failure problem.

Dither failure 2: light lost on WFS:

Symptom
On the first offset/dither of the night with NIRC2 or OSIRIS, the star is lost: the light is not seen on the WFS or STRAP anymore.
AO loops may be open or close at that point.
Problem
The SC needs to be rebooted.
Solution
Open AO loops.
Please follow this procedure when a dither failure happens at the start of the night:
  1. reboot SC: type reboot in the xterm SC terminal server window avialable on the "Telnets" VNC session.
  2. Wait for reboot sequence to finish. Prompt will return. It takes a couple of minutes.
  3. Start the Calibration tools GUI
  4. Select from the calibration tools gui -> LBWFS Calibrations
  5. On the LBWFS Calibration GUI -> run "record image"
  6. From the MAORI-> run nightime-setup
  7. Restart the AOacq gui
  8. Run setup bench for observing setup.
  9. Lock on TT star and complete a 5 arcsec offset to confirm the problem is resovled.

NEEDS UPDATE: Generalized offset/dither problem in NGS and LGS

Symptom
Observers reports a dither failure and AO cannot offset either. The starlight may or not be seen on the WFS or STRAP and AO loops may stay open or close.
In addition, some of the IDL tools may not be responsive anymore.
Problem
Likely, the problem originates from one real-time crate (OBS/SC/WFC2/STRAP or TDC/AUX) that is hung and does not talk to the other crates for AO/DCS interaction.
This problem is rare and hard to troubleshoot. It may take a lot of time to recover from it.
Solution
Open AO loops if you can.
A few questions to help diagnose/document and follow up with the problem:

Dither failure: timeouts in AO/DCS handshake

Symptom
The observer reports that loops are not closing after a dither and NIRC2 and OSIRIS are sitting there waiting for the closed loops.
Problem
The problem is the following: during the AO/DCS handshake, AO opens the loops, moves the stages that needs to be moved and return its status to DCS. If the FSMs (or the TSS) are slow to move, and take longer than 10-15 sec, there is a timeout low in the EPICS code that would prevent the AO loops to close again.
We are trying to tune the TSS and FSM stages so that their moves are fast enough, but that may still happen.. Also, we are trying to understand where that timeout is!
Also, sometime, there a problem with the aoloop kwd.This high level keyword is used to open/close/restore the AO loops during dither. if for any reason (FSMs too slow, communication problem), a time-out occurs during the dither on NIRC2, this kwd could be set to a wrong value and the loops would not close automatically for the next dither.
Restart "AO/DCS comms" would reset that kwd to its default value. One can also monitor this kwd by typing: xshow -s ao aoloop: values are open (0), close (1), restore (2). It should be set to 1 or 2 during a dither.
Solution
Close the loops manually and estimate the time lost waiting for it.
If this happens too often, try a "restart AO/DCS coms"

About "Restart AO/DCS communications":

How to "Restart AO/DCS communications":
Click "re-start AO/DCS coms" from MAORI→Menu
What does "Restart AO/DCS communications" do?
  1. Opens all AO loops (TT, DM, TT and focus offloading)
  2. Resets the aoloop keyword (see above)
  3. Resets dcs.autactiv guiding kwd to 0
  4. Checks Rotator:
    • Puts rotator in manual
    • Offsets rotator by 3 degrees
    • Checks if it is healthy
    • Sets Rotator back to TRACKING
    • Checks if it's healthy there too
  5. Resets automation units used to record a WFS sky background.
  6. Moves FSMs where they are to offload any possible DAR components
  7. Zeros-out X and Y values of the DT control offsets. Inputs for these values come from DAR (differential atmospheric refraction software) and KAT offload (during IF operations).
  8. Reloads the last used centroid origin file for the wavefront controller software.
  9. Restarts the DCS script that sets the TT offloading ouput as guider inputs
  10. Closes TT and focus offloading loops.
  11. The "Restart AO/DCS communications" script does not close automatically the TT and DM loops; thus requiring the operator to do it manually.

How do I check the TT offloading parameters

Symptom
The TT graph are showing very large amplitude moves and the TT residual graphs show a similar trend.
Problem
The problem could come from bad telescope tracking performance, poor seeing, windshake, etc.
It may help to try adjusting the TT offloading parameters: gain and "frequency" (always > 2 seconds)
Solution

AO Operations

To learn about the new AO Acquisition tool, please refer to the Observing with the NGWFC webpage.

How to set the WFS parameters manually?

Symptom
Need to adjust manually frame rate, binning or attenuation.
Problem
How?
Solution
Warning: changing the frame rate, binning or attenuation will cause the loops to open! Do not attempt this in the middle of a science integration! If you only need to change the frame rate, EXPERT ONLY: If you want to change the binning or the attenuation,

How to insert/remove a neutral density filter (WND)

Symptom
The wavefront sensor is saturated or the WND is in place but the star is not bright enough.
Problem
How?
Solution
From MAORI, click on the WFS tab. Then click on the WND button and make a selection (usually 15% or open). The stage will begin to move straight away.

Why is the PSF observation so important to AO observers?

Symptom
Need more information on PSF
Problem
Solution
Most of Adaptive Optics observations have in common to get the best possible image quality and study some unknown or poorly known structures, companions, surface features... It is very important to many astronomers to observe a calibration star after the science target observations to measure this image quality. It allows the astronomer to measure/compare any structure seen in the science target from the calibration star. The measure of the Point Spread Function would provide a reference for further image analysis that could include PSF subtraction, deconvolution or PSF fitting.
For many observer, the absence of a good observation of the PSF calibrator may prejudice a lot the observations.
Particularly, it is crucial to observe the PSF with the same frame rate and number of counts on the WFS. Don't ever run the auto AO setting tool on PSF calibrator star and don't change the frame rate! Just go to the star and close the loops manually from MAORI.

The AO operator may have to manually set WFS Neutral density filters to get the intensity on the PSF star to match the one from the science target.
To set these WNDs, go to MAORI → WFS, click on the WND status display button and select a level of attenuation. There is no need to take a new background since the frame rate has not changed.

How much should you care about the B-V magnitude

Symptom
The observer doesn't know the B-V magnitude of their target.
Problem
Is this really important so that we should insist they provide it?
Solution
B - V is used to derive and set the effective wavelength used by the software that compensates for the Differential Atmospheric Refraction (DAR). This software is important when astronomer do long integration time (>5min) on target for a single exposure. the B-V value is also used by the auto AO setup script to check whether the object is very red and will appear brighter for the WFS.
The astronomer should know the value for the B-V of their target if compensating for DAR is important to their science. You may want to remind him/her about the use of this value if there are long integration exposures involved.

So, is it important??? the answer is:
Yes if the observer is doing long integration on the same object (NIRC2: spectroscopy, coronagraphy, looking for faint galaxies, etc. OSIRIS: in most cases since it is a spectrograph. IF: yes because their objects are a lot of time very red)
Not really in most cases. The observer would be doing short integration imaging. in those most common cases, make sure the B - V value is set within 0.3 - 0.5 and not set to an extreme value like -1 (very blue stars) or 2 (very red stars).

How do I get B-V for a given star?
Go to Simbad data base http://simbad.u-strasbg.fr/sim-fid.pl .
Enter the target name and click on "submit"
The B magn and V mag. get returned together with the star data. We need the difference : B-V

No sounds???

Symptom
You never hear "Keck II AO loops are closed" or "The wavefront controller has crashed".
Problem
The IDL tools are sending sounds to the wrong computers or the volume is too low.
Solution
From a summit computer (e.g. k2aoserver, k2vncserver, etc.) as k2obsao:
  1. Type sounds to list to where IDL is sending sounds.
  2. If your computer is not listed, type sounds your_computer1 your_computer2 to define the sound hosts. N.B. you may define 1 or more sound hosts
  3. Type volume volume1 volume2 to define how loud you want the sounds (must be an integer between 0 and 100).
  4. Make sure your speakers are turned up.

Poor Correction on Extended Objects

Symptom
AO correction on planets or moons is bad, but good on stars
Problem
Easy: Centroid gain is set incorrectly
Hard: Comprehensive calibrations for the wavefront sensor may be out of date.
Solution
Easy: Select object and estimate seeing from AOAcq Tool (angular extent):
Additional Info
Please see the Centroid Gain web page for more detail.

Acquisition tool (AOAcq) pointing origin (PO) mismatch

Symptom
The AO Acquisition tool (AOAcq) gives you the following error: "please ensure the DCS PO is the same as AO PO", possibly accompanied by "DCS PO problems. Fix and redo setup and acquisition".

Nightlog reference: K1-26258
Problem
This is generally the result of a miscommunication between the SA and OA about which pointing origin to use. It's also possible that the AOAcq is misbehaving, especially if the telescope is set to the correct pointing origin.
Solution
The correct pointing origin needs to be selected in AOAcq.
  1. Ensure that the correct pointing origin is selected in the telescope.
  2. Select a different (incorrect) pointing origin in AOAcq.
  3. Select the corret pointing origin in AOAcq.

LGS AO Operations

What are the main differences between LGS AO and NGS AO

The differences are summarized in this power point (PPT) presentation.

Where are the LTCS pages?

Symptom
LTCS pages needed!
Problem
Can't find pages!
Solution
Laser Traffic Control System (LTCS):

NGS and LGS operations procedures

The KeckShare directory is here, it includes the support schedule, many documents as well as the LGS AO: checkout and calibrations

Laser Manual Acquisition

Symptom
LGS AO acquisition has failed.
Problem
You can't get the laser to fall on the WFS.
Solution
  1. ABORT the acquisition
  2. Stop Acam update in Magiq
  3. Open the TT, DM and UTT loops (use maori so that laser keeps propogating)
  4. Confirm LASER PROPAGATION.
  5. Check that the UTT integrator, "Int_Out X & Y" (lower right:M3 Pointing Model), is less than ~2
      if not zero the integrator
      • Select "Pointing Ctrl" from the TOP GUI
      • Select "M3 Pointing Model" (bottom left red btn) from "Pointing Ctr GUI"
      • Toggle Zero from "No" to "Yes" (located bottom right corner).
      • Wait for Int_Out X & Y values to zero. Should take less than 5 seconds.
      • Toggle Zero from "Yes" to "No"
  6. Select Opt Axis on the AO Acq widget and hit FSM to ref.
  7. Using the acamtool and the WFS intensity display
    • Click on Light → Acam
    • Click on Acam lgs
    • Click on Reset (next to Manual acquisition)
    • Take a 10 second exposure
    • Once the tool displays the image and the spot location has been automatically found,
      click on Manual Acquisition
        (If the laser doesn't fall on ACAM, this page may help).
    • If the light is not on WFS, repeat from 7 to here
  8. Once the light is pretty much centered on WFS,
    • Select the desired pointing origin
    • hit FSM to ref.
    • de-select taking a STRAP backround
  9. Click on "Acquire star"

Acquisition when background is uneven (e.g., M31)

Symptom
Want to acquire a tip-tilt star in a region where the background is uneven.
Problem
Difficulty in acquiring the tip-tilt star or getting good tip-tilt correction.
Solution
    The acquisition is performed as follows:

  1. Ensure that the observation is carried out in position angle (PA) mode. If not, the background will rotate and you will eventually lose the star.
  2. AOACQ: Set WFS background to NO
  3. AOACQ: Set STRAP background to NO
  4. AOACQ: Run SETUP BENCH
  5. MAORI → TEL: Set tip-tilt offloading to OPEN
  6. Acquire the star
  7. AOACQ: Run ACQUIRE GUIDE STAR
  8. AOACQ: OPEN LOOPS
  9. Now you need to offset the telescope. The offset should be about 5 arcsec to a region where the background is expected to have a similar gradient. If you have to choose between a region with a lower background or a higher background, choose a lower background. For the core of M31, the best tip-tilt star is NB95. So we are going to offset away from the core 5 arcsec:
    NIRC2: en 4 -3
  10. WFSSTATUS: RECORD BKGD ONLY (say YES when it queries whether you want a STRAP background)
  11. NIRC2: en -4 3 (return to the tip-tilt star)
  12. AOACQ: RECLOSE LOOPS
  13. MAORI → TEL: Set tip-tilt offloading to CLOSED
  14. AOACQ: OFFSET TO TARGET (or check the image quality on the tip-tilt star first)
  15. If the LBWFS performance is not very good, you may want to use 11 modes or the low order mode of the LBWFS.

    16. Remember to set the WFS and STRAP background to AUTO when you move to the next target

Noise on the LBWFS

Symptom
The noise builds up on the LBWFS. It is seen on the individual LBWFS images, on the LBWFS manager tool, and may be even seen on the WFS intensity display.
Problem
The problem could be related to LBWFS calibrations, bad TT corrections (wind-shake) and/or lots of TT residuals.
Solution
There a different ways to mitigate the problem:
  1. Check the seeing (use the methods described here) in the direction of propagation.
  2. Check for wind shake or secondary mirror vibration as well (from the optimization tool).
  3. Check the LBS stage or TSS stage have not faulted.
  4. Try switching to 11 modes from the LBWFS Manager (recommended option).
  5. Try lowering the Cog Control Gain using the LBWFS Manager.
  6. You could also try to increase the integration time on the LBWFS. This has to be done between observing sequences though, as the LBWFS operations are sync. with the observing sequences.
  7. Last, you can try to adjust the integrator leak factor. This is a parameter that determines how much of the current COG file is used in the next COG file. A factor of 0.9 means that the current COG file is multiplied by 0.9 and this is added to what the LBWFS sees to determine the next COG file.
    Usually, the leak is set to 0.95. Try setting the leak factor to a lower value, such as 0.9 (but no lower). show -s ao aolbleak
    modify -s ao aolbleak=0.9
  8. If this problem persists throughout the night, even on axis, it is almost certainly due to a bad DM-to-LBWFS registration. Make sure that the DM-to-LBWFS registration and the LBWFS calibrations are performed again before the next observing night!

Cannot measure the unstacked LGS spot size on ACAM

Symptom
We are unable to measure the unstacked LGS spot size on ACAM, either because the spot is too big or you are having problems with ACAM.
Problem
The spot might be too elongated
Solution
The spot size is needed to set the minimum and maximum spot size that the reconstructor uses. You can set these keywords manually. Estimate what the smallest and largest spot sizes are.
  1. modify -s ao recmnlgs=smallest spot size.
  2. modify -s ao recmxlgs=largest spot size.

Express LGS AO Checkout

Symptom
We are late to get on sky and we want to observe in LGS AO ASAP.
Problem
We normally perfom the full LGS AO check-out, but we want to bypass many of the steps.
Solution
The Express LGS AO Checkout is a flexible, yet minimal procedure to align the laser, perform the LGS AO check-out on a bright star and then go to the first target.
While you are reading this, send laser spotters outside ASAP!-)
See on KeckShare the procedure

Checking STRAP

Symptom
The TT control works well in NGS AO mode but does not in LGS AO mode. Or STRAP does not behave as expected.
Problem
STRAP rarely has a problem, but that could be the day...
Solution

TSS: DAR and TSS FOCUS control in LGS mode

Symptom
You are suspicious something in wrong and would like to check manually TSS tracking for DAR and the focus correction.
Problem
Unkown.
Solution
The DAR software functions are used to compensate for the differential atmospheric refraction while the "focus function" on TSS is used to compensate for the field curvature when TSS moves off axis (do not get confused with the LBWFS focus here!). Both DAR and TSS focus functions are implemented in the SC and the OBS real time software with a lot of their key-parameters that are set during the night time script and from the LGS AO acquisition widget depending on the observing setup.
Forget about doing anything manually! The layers of software around TSS control are really complicated. r.g., the IDL LGS AO acquisition widget is used for setting the TSS gold numbers depending on the SOD stage (sodiumDichroic or beamSplitter) and the location of the optical-axis on ACAM.
Check the offset to the TSS gold numbers: the keywords are aotsdx, aotsdy and aotsdz. The first two should be zero unless we are doing non-siderial tracking. The third should be set to whatever focus value we measured on the LBWFS during the LGS AO checkout.
There are scripts that can be operated to check the TSS and its functions:

LGS AO VNC session Xserver seems dead

Symptom
Pulldown menus appear but do not launch applications and/or starting applications from the command line gives this type of error:
Xlib: connection to "polo:17.0" refused by server
Xlib: Client is not authorized to connect to Server
xrdb: Can't open display 'polo:17.0'
Problem
xauth is confused and does not believe you are authorized to start applications on that display
Solution

LGS AO VNC Norse Viking Attack

Symptom
Typing in an xterm does not produce English, but only Umlauts.
Problem
The Norse Vikings have executed a sneak attack
Solution

Cannot take LBWFS images

Symptom
Typically in the afternoon, attempts to take images from the LBWFS calibration tool result in messages like this:
ls: no match
Problem
Low level LBWFS software is not running
Solution
Restart the LBWFS Camera Server by 1 of 2 easy ways and 1 hard way:
  1. Pull-down:
    1. Right-click in background -> "KN AO Tools" -> "Other AO" -> "LBWFS Server STOP"
    2. Right-click in background -> "KN AO Tools" -> "Other AO" -> "LBWFS Server Start"
  2. Command Line:
    • As kNobsao@kNaoserver, run 
      LBWFSCamServer restart
  3. Still not fixed?:
    • As kNobsao@kNaoserver, run 
      LBWFSCamServer stop
    • As kNobsao@kNaoserver, run 
      LBWFSCamServer restart
    • You should now be able to take images

Transponder-Based Aircraft Detection

Full details: TBAD Twiki Page

TBAD Basics

TBAD has replaced human spotters
Human spotters are a backup option
Spotter packs are required
Even though they are unused, the laser safety system still requires the presense of the spotter packs on K2
What triggers a TBAD shutter?
  1. More than 3 "in-beams" within any 10 second time frame -- an aircraft is approaching the beam.
  2. oSaturation -- omni-directional antenna saturation, may indicate a "Top Gun" pilot buzzing the dome
  3. dSaturation -- directional antenna saturation, may indicate a "Top Gun" pilot buzzing the dome
TBAD permissive triggers are not latched
If TBAD goes to deny (RED), it will go to grant (GREEN) as soon as the condition clears
TBAD permissive triggers do latch in the AO alarm handler
One must acknowledge the permissive change to clear. If not, it may cause confusion later.

TBAD xshow Timestamp Incorrect

Symptom
Timestamp in TBAD Status xshow is not within 2 minutes of the current time
Problem
TBAD is not healthy -- Cease laser operations until fixed!
Solution
If propagating:
  1. Run "tbad_log_tail" as kNobsao on kNaoserver.
  2. Laser operations my continue if the log file timestamp is current and updating and the following keywords have these values:
    • LSTBDSHUT = grant
    • LSTBDSTATE = grant
    • LSTBDBYPAS = notBypassed
  3. At your earliest convenience, stop propagating and follow the "not propagating" steps
If not propagating:
  1. Do not propagate
  2. As kNlaser@kNaoserver-new
  3. Good Status (exactly one process running): 
    12 k2aoserver-new{/home/makaha/k2laser}: /kroot/rel/ao/default/bin/linux-x86_64/tbadAdmin.sh status
198823 python3
  4. If your status is not good:
  5. /kroot/rel/ao/default/bin/linux-x86_64/tbadAdmin.sh restart
More details on the TBAD Twiki Page

Wavefront Controller

Wavefront Controller Crash

Symptom
WFS intensity does not update
Fault detector may trigger an audio alarm
TT graphs do not update
Problem
The wavefront controller may have crashed.
Solution
To determine whether there are problems with the WFC, please refer to the Observing with the NGWFC webpage. There are several failure modes associated with the WFC; many of them are not crashes but can be fixed by rebooting the WFC.
To reboot the WFC, click on MAORI → Menu → Reboot WFC. Then reacquire the star using the AO acquisition widget.

Powering up the tip-tilt mirror

Symptom
The tip-tilt mirror is being commanded but does not appear to be moving
Problem
The tip-tilt mirror is not powered up
Solution
Turn on the TT mirror power supply by either -or-
    Browser (requires Java)
  1. Go to the K1 or K2 web-based interface.
  2. Select "Browser Web Control".
  3. Turn all three power supplies on.

Loading new WFS centroid origin calibrations

Symptom
Need to load new COG file
Problem
None, just dunno how to do it!
Solution
  1. From an xterm on k2aoserver as k2obsao, type cdcog
  2. Then dir will list all centroid origins calibration files sorted by date.
  3. The name reflects the instrument and the date. For a NIRC2 night in NGS AO mode for 2x2 binning with the 2.4 arcsec plate scale, the default file would be 24NIRC2-N2x2.cog.
  4. In IDL, type LOADCOG,'the-one-you-want.cog'
  5. The new calibration file should be loaded on the system.

Querying TRS

Symptom
Want to query TRS to extract data or parameters
Problem
None, just dunno how to do it!
Solution
Refer to the TRS queries webpage.

AO Rotator

Who controls the AO rotator?

Symptom
The AO rotator needs to be moved to set the sky PA and sometimes fixed if it has FAULTED.
Problem
Who should command the rotator moves?
Who should fix it when it faults?
Solution

How to detect a ROTATOR FAULT in NGS or LGS AO

Symptom
At acquistion on ACAM, the OA does not manage to send the reference star to the desired pointing origin from Xguidee
TT offload does not work well; The TT graphs display the TT mirror position off-center, although the WFS average centroids (X, Y) are close to 0.0.
Problem
The rotator may not be tracking well and not read yet a FAULT status.
Solution
From the AO acquisition widget, run the "reset AO/DCS coms".

Recover from a ROTATOR FAULT in NGS or LGS AO

Symptom
Rotator status could read FAULT on telescope FACSUM.
An audio message informs the AO operator that the rotator has FAULTED.
Problem
Need to fix the rotator and set it back to TRACKING
Solution
EXPERT ONLY: Drastic Measures

OBS device FAULTs

Why do these stages fault?

Symptom
The Optics Bench Subsystem (OBS) hosts as many as 21 stages on K2. Most stages have more than 1 axis; their moves need to be fast and accurate (~< 10 microns).
PMAC boards control the way each stages move for each mode (manual, tracking, simul). The OBS works very well most of the times, but the system is not 100% reliable and there are problems from time to time,
Also, there is a basic design flaw in the OBS stages to my mind - and some of it is due to the fact that some stages have coordinated moves - : each motor tries to SLEW to any demanded position, w/o checking whether the demanded position is within the range limits.
Problem
Some FAULTs happen because the stage hit its positive or negative SOFTWARE LIMIT.
Other times, it could be a TRACKING problem.
Other times, the stage could be slightly detuned stage and the FAULT is a "max following error"
Solution
Let the automated recovery script run for the FSM, the ROTATOR, the FCS [and the TSS?]
The philosophy to recover from a FAULT is:

Repeated FSM FAULTs

Symptom
The FSM FAULTs are happening for moves that are within the range of the FSM field-of-regard.
The FSM error message displayed in the IDL Observing Tools window reads"Max following error"
Problem
One of the FSM motors (FSM1x, FSM1y, FSM2x, FSM2y) could be detuned and/or experiencing problems.
Solution
First, one needs to find which FSM motor experiences the problem.
  1. From AO SC GUI, bring an OBS DM screen, click on the FSM button to bring up the FSM dm screen.
  2. Dismiss the AO acquisition widget (to avoid triggering the automated recovery)
  3. Reproduce the FAULT and leave the stage in the FAULT state.
  4. From the FSM dm screen (ecPDev4.dl), note which motor displays a red indicator for the following error.
  5. All FSMS are on PMAC3, FSM1x is on motor5, FSM1y → motor6, FSM2x → motor7, FSM2y → motor8.
  6. From the OBS dm screen, click on PMC3, click on "MOTOR-X IVARs" where X is 5, 6, 7 or 8.
  7. Enter new value for Ix11 in yellow field, increase value by 10 to 20% in a first attempt to solve the problem (300 or 350k). Hit return and click update.
  8. Init the FSM from scgui and try to reproduce the fault.
  9. If the fault persists, call an AO support person.
  10. Don't forget to document the changes to the I variables in the nightlog
  11. Bring up AO acquisition widget and resume operations.

FCS FAULT in tracking mode (LGS AO)

Symptom
FCS reads fault, or gets into a FAULT state for every new target. Once faulted, the stage does not track.
Problem
In LGS AO mode, the FCS stage needs to TRACK as a function of telescope elevation and sodium altitude. If it is not tracking, observing must stop and the problem needs to be fixed.
In NGS AO, the stage does not track so it should be less a problem and have a lesser impact on the observations; yet the fault needs to be cleared as soon as possible.
Solution
The FCS fault should be automatically detected by the AO acquisition widget and be reported to the operator. Another script will try to recover automatically from the fault and restore the FCS to its initial state.

TSS FAULT in LGS AO mode

Symptom
TSS faults would occur either:
  1. if the requested offset is out of the TSS range,
  2. if the stage does not transition well to or from servo-in mode
  3. when the stage experiences problem to track in servo-in mode.
Problem
It will likely impact the observations and could require to reboot OBS if the TSS tracking mode is required for the observations.
Solution
We have developed an automated recovery script for the TSS so hopefully it will work well.. If not:
  1. Turn DAR and TRACKING OFF from LGS AO acq widget
  2. Put the TSS in MANUAL mode from MAORI->Track
  3. from SC GUI, try to send TSS to optsodstrap (yes, it won't go there...)
  4. then click HALT and INIT for the TSS stage
  5. Return the TSS to SERVO mode from MAORI->Track

Other resources and help pages

Where can I learn about all these acronyms??!!

Cindy has put a page together here.
There is also some information posted here.

Observing without ACAM

Symptom
ACAM dies, its shutter locks in the closed position, or some other problem prevents you from using ACAM. This technique was successfully used in 2003 April 21 to continue observing with a locked shutter.
Problem
How do you center stars on the wavefront sensor?
Solution
NIRC2 or OSIRIS (imager)
NIRSPAO and other small field instruments
SCAM has such a small field of view that you might easily miss the star on both SCAM and the wavefront sensor. If so, follow this procedure:
  1. Point to the star
  2. Assuming you don't see it on the wavefront sensor, take a WFS sky image here.
  3. Put in PMFM (primary mirror focus mode) of about 500 nm. This will create 36 segment images and increase your chances of finding one.
  4. If you still don't see a segment, either use the handpaddle to search around until the WFS sees a bright segment, or take images with SCAM and look for a segment. The latter was used most successfully in April 2003, but selecting SCAM from xguide and looking for segments in "video mode" might be equally successful.
  5. Once you have a segment on the WFS or on SCAM, try to estimate where the center of the 36-segment PMFM pattern is. Roughly center the pattern (i.e. get several segments on the image, not just one.)
  6. Put in PMFM of 300, and immediately start a SCAM exposure that will last for a few seconds. (You can increase the number of coadds to get the total exposure up to 5 seconds or so.) The moving segments will create streaks that point roughly back towards the pattern center.
  7. Again estimate the pattern center from the radial streaks and recenter.
  8. You can repeat with a PMFM of 150, but once you have the pattern center on SCAM you should be able to go straight to zero PMFM (normal images) and center the star on SCAM.
  9. The star should now be on the WFS. (If it's not, check your FSM -(If it's not, check your FSM position.)
  10. Once the WFS sees the target, you can lock on and proceed with the observations.

I Can't See Light on the WFS!

I Can't See Light on the WFS!

Problems with User Interfaces (IDL, etc)

Problems with User Interfaces