AO Troubleshooting

WWW Optics pages

AO Performance

User Guide for Optimization Tool (also named bandwidth widget)

The optimization tool has its dedicated web page.

Checklist for bad AO performance

The image quality is bad
Possible Problems
The list of possible problems includes:
  1. Bad or fast seeing
  2. Telescope windshake or telescope vibrations, low elevation
  3. Problem with the WFS background
  4. Problem with the reconstructor
  5. Vignetted pupil
  6. Tip-tilt loop unstable
  7. Tip-tilt offloading problem
  8. Bad calibrations
  9. Etc
Note that proper troubleshooting should be performed on an 8th magnitude B-A type star (NGS) or one of the LGS checkout stars.
  1. Measure the seeing using the atmospheric characterization tool, acamtool, open loop images on NIRC2 or using the LBWFS (read the aolbfwhm keyword).
  2. Use the bandwidth widget to optimize the TT and DM gains and check for any problems with vibrations.
  3. Record new background
  4. Trigger a new reconstructor using MAORI→WFS with both loops closed and check that reconstructor updates after a few seconds. If not, then restart the reconstructor from MAORI→Menu. Check the pupil on the WFS intensity display with the reconstructor mask on and off (the pupil should look the same).
  5. Check windshake, check tracking errors for el and az
  6. Check for possible vibration produced by the secondary vacuum valve, the telescope drives, etc
  7. Check that tip-tilt mirror is powered up.
  8. Check for problem with the tip-tilt mirror (broken actuator?)
  9. Check for problem with the DM using the WYKO interferometer. Open the WYKO hatch and look at the WYKO when the DM loop is opened or closed. You should see the DM move constantly in closed loop. You should also see that it is not producing strange shapes, such as waffle or a lot of focus.
  10. Check for calibrations: DM to lenslet, COG, SFP home problem, WLS centering, FSS vignetting, etc.

How do I measure the seeing

Need to measure seeing
  1. Using the atmospheric characterization tool
  2. Using the science camera: pick a fairly bright star (7 < V < 10), open DM loop, close TT loop (or open both loops).
    tint 1; coadd 20; filt kp; wait4ao off; goi
    Measure FWHM. In IDL, type ttcomp,0.45 or whatever the FWHM is.
  3. Using ACAM: you may use the IDL acamtool and record a 1-3sec exposure. The tool will display the FWHM for star which gives an estimate for the seeing in the V band. The real star is at the bottom, the ghost image is always on top (the opposite than the guider).
    In NGS AO mode:
    Make sure that xguide is IDLE. Set the integration, and record image. The tool will automatically display the star in the sub-frame to the left, and a model to the right. The value for the FWHM is displayed on the tool. More values for the photometry is displayed in the xterm window.
    In LGS AO mode:
    The tool shows up in this mode if the AO system is configured for LGS AO mode. Same use than in NGS-AO, make sure that xguide is IDLE. Set the integration, and record image. The tool will display more option to move the stages to be focused either on the NGS or for the LGS. It also gives the possibility to acquire manually the laser.
    Or estimate the FWHM from the guider display (less accurate?).
Seeing sizes smaller than 0.8'' and 0.5'' in V are considered respectively fair and excellent. The AO correction gets worse as the seeing increases and the image quality really suffers for seeing larger than 1'' in V.
The important information for seeing is not only the spatial scale but also the temporal scale (correlation time). The temporal scale for seeing can only be measured using dedicated tools that we lack at Keck Observatory.
You can also check for seeing estimate from the Mauna Kea weather web page. They display seeing forecast as well as distribution of wind layers and their velocity. The stronger the wind associated with a turbulent atmospheric layer, the worse the shorter the atmospheric decorrelation time. This is especially important when the WFS is run at a low frame rate. If the jet streams are shown in proximity to the Island, the seeing is very likely to be fast and large.

Seeing is either large or very fast, what should I do?

In case of bad (fast or large) seeing, there are various setup tricks that may help.
AO performance may not be optimal, how can I make sure it is?
Here are some tips that may help when bad seeing is resulting in bad image quality.
  1. Adjust the centroid gain by changing the seeing setting in the AO Acquisition widget.
  2. Check AO performance using the bandwidth widget. If seeing is faster, it may be that our lookup table is way off... increasing the gains or the frame rate may help.
  3. Adjust the tip-tilt offloading parameters (see below)
  4. Trigger a new reconstructor from the MAORI→WFS tab with the loops closed and check that the reconstructor updates. If not, restart the reconstructor from MAORI→Menu.

Checking the WFS centroid origin calibrations

Bad AO performance, even for bright star (Vmag<10). Artifacts in the image.
Bad calibrations or corrupted file for the centroid origins?
  1. From an xterm on k2aoserver as k2obsao, type cdcog
  2. Then dir will list all centroid origins calibration files sorted by date.
  3. The name reflects the instrument and the date. For a NIRC2 night in NGS AO mode with the 2.4 arcsec plate scale and 2x2 binning the default file would be 24NIRC2-N2x2.cog. Find a file with the same name but a different date appended (e.g., 24NIRC2-N2x2.cog16Feb2007) that you trust. In this example,
  4. \cp -p 24NIRC2-N2x2.cog16Feb2007 24NIRC2-N2x2.cog
  5. In IDL, type LOADCOG,'24NIRC2-N2x2.cog'
  6. The new calibration file should be loaded on the system.
  7. try other calibration files if necessary or use different binning modes or plate scale.

Stars appear elongated, what should I do?

Stars appear elongated on science detector.
The TT mirror and TT residual graph could show a lot of jitter.
Telescope servo-error tracking values could be large.
Many possible reasons, but the most likely are:
  1. Tip-tilt mirror oscillations.
  2. Problem with TT offloading: could happen when star goes through meridian.
  3. Wind shake: do you see the same effect when you point into the wind vs other directions?
  4. Lots of vibrations (produced by the secondary-mirror vacuum valve?): experience has shown it could be worse at lower elevation
  5. Reference star is a binary or has a peculiar morphology: 2-5 arcseconds binary used as a reference star may produce some elongation. If it is a close binary (less than 100 mas separattion) then the image may appear to be elongated.

The Guide Star is a 2"-5" binary (NGS only)

We are guiding on a 2 to 5 arcsec binary of comparable magnitude and the image quality is poor.
Both stars are in the wavefront sensor, so the star is not centered on the cross hairs of the wavefront sensor.
A possible solution is to introduce a smaller field stop in front of the wavefront sensor.
  1. Open the loops using MAORI
  2. From MAORI → WFS, left click on the entry by the FSS row and select "0.6". FSS is the field stop stage.
  3. Once FSS reads back "0.6", you may reclose the loops and continue.
  4. Next time a star is acquired, the field stop will automatically go back to the "2.4" named position.

Telescope/AO interactions

AO/DCS communications: general

The AO/DCS communication is crucial to AO observing for:

Dither failure: light lost on WFS:

Observers reports a dither failure: after/during a dither with NIRC2 or OSIRIS, the star is lost: the light is not seen on the WFS or STRAP anymore.
AO loops may be open or close at that point.
Dither failure may happen rarely. It is AO/DCS problem that is quite complex to investigate on the spot.
Open AO loops immediately!
We recommend to follow this procedure when a dither failure happens as
  1. Write down UT time and include the time stamp in the ticket.
  2. Re-acquire the target
    1. by re-sending the "move" command from sky.
      We believe this is the only way for now to reset the TT offloading control point to (0,0).
    2. By adjusting pointing on REF if necessary
    3. By sending object to the default DCS pointing origins (nirc2 for NIRC2, opt-axis for IF)
  3. Restart "AO/DCS communication"
  4. Once you see the target on the WFS, no need for a new background, just close TT and DM loop, and you should be ready to go.
  5. Notes:
    • If you had been on this target for a while, record a new background WFS and close loops.
    • You could re-run the AO setup script again for this target, but it is unlikely that would fix the dither failure problem.

Dither failure 2: light lost on WFS:

On the first offset/dither of the night with NIRC2 or OSIRIS, the star is lost: the light is not seen on the WFS or STRAP anymore.
AO loops may be open or close at that point.
The SC needs to be rebooted.
Open AO loops.
Please follow this procedure when a dither failure happens at the start of the night:
  1. reboot SC: type reboot in the xterm SC terminal server window avialable on the "Telnets" VNC session. Wait for reboot sequence to finish. Prompt will return. It takes a couple of minutes.
  2. Start the Calibration tools GUI
  3. Select from the calibration tools gui -> LBWFS Calibrations
  4. On the LBWFS Calibration GUI -> run "Setup for Cals"
  5. On the LBWFS Calibration GUI -> run "record image"
  6. Confirm on LBWFS image that all sub-apertures are illuminated. If not check TSS position.
  7. From the MAORI-> run nightime-setup
  8. Restart the AOacq gui
  9. Run setup bench for observing setup.
  10. Lock on TT star and complete a 5 arcsec offset to confirm the problem is resovled.

Generalized offset/dither problem in NGS and LGS

Observers reports a dither failure and AO cannot offset either. The starlight may or not be seen on the WFS or STRAP and AO loops may stay open or close.
In addition, some of the IDL tools may not be responsive anymore.
Likely, the problem originates from one real-time crate (OBS/SC/WFC2/STRAP or TDC/AUX) that is hung and does not talk to the other crates for AO/DCS interaction.
This problem is rare and hard to troubleshoot. It may take a lot of time to recover from it.
Open AO loops if you can.
A few questions to help diagnose/document and follow up with the problem:

Dither failure: timeouts in AO/DCS handshake

The observer reports that loops are not closing after a dither and NIRC2 and OSIRIS are sitting there waiting for the closed loops.
The problem is the following: during the AO/DCS handshake, AO opens the loops, moves the stages that needs to be moved and return its status to DCS. If the FSMs (or the TSS) are slow to move, and take longer than 10-15 sec, there is a timeout low in the EPICS code that would prevent the AO loops to close again.
We are trying to tune the TSS and FSM stages so that their moves are fast enough, but that may still happen.. Also, we are trying to understand where that timeout is!
Also, sometime, there a problem with the aoloop kwd.This high level keyword is used to open/close/restore the AO loops during dither. if for any reason (FSMs too slow, communication problem), a time-out occurs during the dither on NIRC2, this kwd could be set to a wrong value and the loops would not close automatically for the next dither.
Restart "AO/DCS comms" would reset that kwd to its default value. One can also monitor this kwd by typing: xshow -s ao aoloop: values are open (0), close (1), restore (2). It should be set to 1 or 2 during a dither.
Close the loops manually and estimate the time lost waiting for it.
If this happens too often, try a "restart AO/DCS coms"

About "Restart AO/DCS communications":

How to "Restart AO/DCS communications":
Click "re-start AO/DCS coms" from MAORI→Menu
What does "Restart AO/DCS communications" do?
  1. Opens all AO loops (TT, DM, TT and focus offloading)
  2. Resets the aoloop keyword (see above)
  3. Resets dcs.autactiv guiding kwd to 0
  4. Checks Rotator:
    • Puts rotator in manual
    • Offsets rotator by 3 degrees
    • Checks if it is healthy
    • Sets Rotator back to TRACKING
    • Checks if it's healthy there too
  5. Resets automation units used to record a WFS sky background.
  6. Moves FSMs where they are to offload any possible DAR components
  7. Zeros-out X and Y values of the DT control offsets. Inputs for these values come from DAR (differential atmospheric refraction software) and KAT offload (during IF operations).
  8. Reloads the last used centroid origin file for the wavefront controller software.
  9. Restarts the DCS script that sets the TT offloading ouput as guider inputs
  10. Closes TT and focus offloading loops.
  11. The "Restart AO/DCS communications" script does not close automatically the TT and DM loops; thus requiring the operator to do it manually.

How do I check the TT offloading parameters

The TT graph are showing very large amplitude moves and the TT residual graphs show a similar trend.
The problem could come from bad telescope tracking performance, poor seeing, windshake, etc.
It may help to try adjusting the TT offloading parameters: gain and "frequency" (always > 2 seconds)

SKY/AO conflict for non-sidereal (planetary) targets

You have selected a planetary object to observe from SKY, Xguide crashed and target can't be acquired.
There is some history of problems with SKY on non-sidereal targets. Avoid selecting planetary targets directly from SKY, use a starlist instead.
There is a DCS keywords DOFFGE that needs to be set to 0. for AO guiding.
Use a starlist, don't select directly from SKY.
As of July 2006, dcs.doffge is being set to 0 everytime we begin acquire a target in NGS or LGS mode.
You should be able to use non-sidereal tracking rate from the starlist just make sure that DCS.DOFFGE is set to 0 at all times.

AO Operations

To learn about the new AO Acquisition tool, please refer to the Observing with the NGWFC webpage.

How to set the WFS parameters manually?

Need to adjust manually frame rate, binning or attenuation.
Warning: changing the frame rate, binning or attenuation will cause the loops to open! Do not attempt this in the middle of a science integration! If you only need to change the frame rate, If you want to change the binning or the attenuation,

How to insert/remove a neutral density filter (WND)

The wavefront sensor is saturated or the WND is in place but the star is not bright enough.
From MAORI, click on the WFS tab. Then click on the WND button and make a selection (usually 15% or open). The stage will begin to move straight away.

Why is the PSF observation so important to AO observers?

Need more information on PSF
Most of Adaptive Optics observations have in common to get the best possible image quality and study some unknown or poorly known structures, companions, surface features... It is very important to many astronomers to observe a calibration star after the science target observations to measure this image quality. It allows the astronomer to measure/compare any structure seen in the science target from the calibration star. The measure of the Point Spread Function would provide a reference for further image analysis that could include PSF subtraction, deconvolution or PSF fitting.
For many observer, the absence of a good observation of the PSF calibrator may prejudice a lot the observations.
Particularly, it is crucial to observe the PSF with the same frame rate and number of counts on the WFS. Don't ever run the auto AO setting tool on PSF calibrator star and don't change the frame rate! Just go to the star and close the loops manually from MAORI.

The AO operator may have to manually set WFS Neutral density filters to get the intensity on the PSF star to match the one from the science target.
To set these WNDs, go to MAORI → WFS, click on the WND status display button and select a level of attenuation. There is no need to take a new background since the frame rate has not changed.

How much should you care about the B-V magnitude

The observer doesn't know the B-V magnitude of their target.
Is this really important so that we should insist they provide it?
B - V is used to derive and set the effective wavelength used by the software that compensates for the Differential Atmospheric Refraction (DAR). This software is important when astronomer do long integration time (>5min) on target for a single exposure. the B-V value is also used by the auto AO setup script to check whether the object is very red and will appear brighter for the WFS.
The astronomer should know the value for the B-V of their target if compensating for DAR is important to their science. You may want to remind him/her about the use of this value if there are long integration exposures involved.

So, is it important??? the answer is:
Yes if the observer is doing long integration on the same object (NIRC2: spectroscopy, coronagraphy, looking for faint galaxies, etc. OSIRIS: in most cases since it is a spectrograph. IF: yes because their objects are a lot of time very red)
Not really in most cases. The observer would be doing short integration imaging. in those most common cases, make sure the B - V value is set within 0.3 - 0.5 and not set to an extreme value like -1 (very blue stars) or 2 (very red stars).

How do I get B-V for a given star?
Go to Simbad data base .
Enter the target name and click on "submit"
The B magn and V mag. get returned together with the star data. We need the difference : B-V

No sounds???

You never hear "Keck II AO loops are closed" or "The wavefront controller has crashed".
The IDL tools are sending sounds to the wrong computers or the volume is too low.
From a summit computer (e.g. k2aoserver, k2vncserver, etc.) as k2obsao:
  1. Type sounds to list to where IDL is sending sounds.
  2. If your computer is not listed, type sounds your_computer1 your_computer2 to define the sound hosts. N.B. you may define 1 or more sound hosts
  3. Type volume volume1 volume2 to define how loud you want the sounds (must be an integer between 0 and 100).
  4. Make sure your speakers are turned up.

Poor Correction on Extended Objects

AO correction on planets or moons is bad, but good on stars
Easy: Centroid gain is set incorrectly
Hard: Comprehensive calibrations for the wavefront sensor may be out of date.
Easy: Select object and estimate seeing from AOAcq Tool (angular extent):
Additional Info
Please see the Centroid Gain web page for more detail.

LGS AO Operations

What are the main differences between LGS AO and NGS AO

The differences are summarized in this power point (PPT) presentation.

Where are the LTCS pages?

LTCS pages needed!
Can't find pages!
Laser Traffic Control System (LTCS):

NGS and LGS operations procedures

The KeckShare directory is here, it includes the support schedule, many documents as well as the LGS AO: checkout and calibrations

Laser Manual Acquisition

LGS AO acquisition has failed.
You can't get the laser to fall on the WFS.
  1. ABORT the acquisition
  2. Stop Acam update in Magiq
  3. Open the TT, DM and UTT loops (use maori so that laser keeps propogating)
  5. Check that the UTT integrator, "Int_Out X & Y" (lower right:M3 Pointing Model), is less than ~2
      if not zero the integrator
      • Select "Pointing Ctrl" from the TOP GUI
      • Select "M3 Pointing Model" (bottom left red btn) from "Pointing Ctr GUI"
      • Toggle Zero from "No" to "Yes" (located bottom right corner).
      • Wait for Int_Out X & Y values to zero. Should take less than 5 seconds.
      • Toggle Zero from "Yes" to "No"
  6. Select Opt Axis on the AO Acq widget and hit FSM to ref.
  7. Using the acamtool and the WFS intensity display
    • Click on Light → Acam
    • Click on Acam lgs
    • Click on Reset (next to Manual acquisition)
    • Take a 10 second exposure
    • Once the tool displays the image and the spot location has been automatically found,
      click on Manual Acquisition
        (If the laser doesn't fall on ACAM, this page may help).
    • If the light is not on WFS, repeat from 7 to here
  8. Once the light is pretty much centered on WFS,
    • Select the desired pointing origin
    • hit FSM to ref.
    • de-select taking a STRAP backround
  9. Click on "Acquire star"

Acquisition when background is uneven (e.g., M31)

Want to acquire a tip-tilt star in a region where the background is uneven.
Difficulty in acquiring the tip-tilt star or getting good tip-tilt correction.
    The acquisition is performed as follows:

  1. Ensure that the observation is carried out in position angle (PA) mode. If not, the background will rotate and you will eventually lose the star.
  2. AOACQ: Set WFS background to NO
  3. AOACQ: Set STRAP background to NO
  5. MAORI → TEL: Set tip-tilt offloading to OPEN
  6. Acquire the star
  9. Now you need to offset the telescope. The offset should be about 5 arcsec to a region where the background is expected to have a similar gradient. If you have to choose between a region with a lower background or a higher background, choose a lower background. For the core of M31, the best tip-tilt star is NB95. So we are going to offset away from the core 5 arcsec:
    NIRC2: en 4 -3
  10. WFSSTATUS: RECORD BKGD ONLY (say YES when it queries whether you want a STRAP background)
  11. NIRC2: en -4 3 (return to the tip-tilt star)
  13. MAORI → TEL: Set tip-tilt offloading to CLOSED
  14. AOACQ: OFFSET TO TARGET (or check the image quality on the tip-tilt star first)
  15. If the LBWFS performance is not very good, you may want to use 11 modes or the low order mode of the LBWFS.

  16. Remember to set the WFS and STRAP background to AUTO when you move to the next target

Laser spot not detected on ACAM image. M3 misaligned.

When aligning the laser using the ACAM tool, the images are not the typical images of the laser spot. Instead, ACAM detects the laser light, but the light is very diffuse. Images with the laser shuttered show a darker sky image. See image below.
M3 is badly misaligned. Scattered light is detected as seen in the first image below.
M3 position needs to be adjusted.
  1. Set up ACAM as if you were going to coarse align the laser.
  2. Take an initial image with ACAM (10 s 1-shot).
  3. Click in the direction of the laser spot in ACAM and ensure that xy centroids were found.
  4. Retake an image and repeat coarse alignment until spot is detected. In the image sequence below, it required three coarse alignments to see the spot.
  5. after spot is coarsely centered procede with laser alignment.

Image sequence showing misaligned image followe by images with several coarse moves. The red cross hair shows the position of the cursor for each coarse align step. The last image is a typical ACAM image of the spot when aligned. M3 ACAM image sequence

Noise on the LBWFS

The noise builds up on the LBWFS. It is seen on the individual LBWFS images, on the LBWFS manager tool, and may be even seen on the WFS intensity display.
The problem could be related to LBWFS calibrations, bad TT corrections (wind-shake) and/or lots of TT residuals.
There a different ways to mitigate the problem:
  1. Check the seeing (use the methods described here) in the direction of propagation.
  2. Check for wind shake or secondary mirror vibration as well (from the optimization tool).
  3. Check the LBS stage or TSS stage have not faulted.
  4. Try switching to 11 modes from the LBWFS Manager (recommended option).
  5. Try lowering the Cog Control Gain using the LBWFS Manager.
  6. You could also try to increase the integration time on the LBWFS. This has to be done between observing sequences though, as the LBWFS operations are sync. with the observing sequences.
  7. Last, you can try to adjust the integrator leak factor. This is a parameter that determines how much of the current COG file is used in the next COG file. A factor of 0.9 means that the current COG file is multiplied by 0.9 and this is added to what the LBWFS sees to determine the next COG file.
    Usually, the leak is set to 0.95. Try setting the leak factor to a lower value, such as 0.9 (but no lower). show -s ao aolbleak
    modify -s ao aolbleak=0.9
  8. If this problem persists throughout the night, even on axis, it is almost certainly due to a bad DM-to-LBWFS registration. Make sure that the DM-to-LBWFS registration and the LBWFS calibrations are performed again before the next observing night!

Cannot measure the unstacked LGS spot size on ACAM

We are unable to measure the unstacked LGS spot size on ACAM, either because the spot is too big or you are having problems with ACAM.
The spot might be too elongated
The spot size is needed to set the minimum and maximum spot size that the reconstructor uses. You can set these keywords manually. Estimate what the smallest and largest spot sizes are.
  1. modify -s ao recmnlgs=smallest spot size.
  2. modify -s ao recmxlgs=largest spot size.

Express LGS AO Checkout

We are late to get on sky and we want to observe in LGS AO ASAP.
We normally perfom the full LGS AO check-out, but we want to bypass many of the steps.
The Express LGS AO Checkout is a flexible, yet minimal procedure to align the laser, perform the LGS AO check-out on a bright star and then go to the first target.
While you are reading this, send laser spotters outside ASAP!-)
See on KeckShare the procedure

Checking STRAP

The TT control works well in NGS AO mode but does not in LGS AO mode. Or STRAP does not behave as expected.
STRAP rarely has a problem, but that could be the day...

TSS: DAR and TSS FOCUS control in LGS mode

You are suspicious something in wrong and would like to check manually TSS tracking for DAR and the focus correction.
The DAR software functions are used to compensate for the differential atmospheric refraction while the "focus function" on TSS is used to compensate for the field curvature when TSS moves off axis (do not get confused with the LBWFS focus here!). Both DAR and TSS focus functions are implemented in the SC and the OBS real time software with a lot of their key-parameters that are set during the night time script and from the LGS AO acquisition widget depending on the observing setup.
Forget about doing anything manually! The layers of software around TSS control are really complicated. r.g., the IDL LGS AO acquisition widget is used for setting the TSS gold numbers depending on the SOD stage (sodiumDichroic or beamSplitter) and the location of the optical-axis on ACAM.
Check the offset to the TSS gold numbers: the keywords are aotsdx, aotsdy and aotsdz. The first two should be zero unless we are doing non-siderial tracking. The third should be set to whatever focus value we measured on the LBWFS during the LGS AO checkout.
There are scripts that can be operated to check the TSS and its functions:

LGS AO VNC session Xserver seems dead

Pulldown menus appear but do not launch applications and/or starting applications from the command line gives this type of error:
Xlib: connection to "polo:17.0" refused by server
Xlib: Client is not authorized to connect to Server
xrdb: Can't open display 'polo:17.0'
xauth is confused and does not believe you are authorized to start applications on that display

M3, Laser Steering Mirror is Completely Lost

Software indicates Laser is propogating but laser it not evident in dome or on sky by spotters and web cams.
M3, the laser steering mirror has driven itself so far out of alignment that the laser is not exiting the laser enclosure

Transponder-Based Aircraft Detection

TBAD Basics

TBAD is on K2 only
K1 still requires human spotters
Spotter packs are required
Even though they are unused, the laser safety system still requires the presense of the spotter packs on K2
What triggers a TBAD shutter?
  1. More than 3 "in-beams" within any 10 second time frame -- an aircraft is approaching the beam.
  2. oSaturation -- omni-directional antenna saturation, may indicate a "Top Gun" pilot buzzing the dome
  3. dSaturation -- directional antenna saturation, may indicate a "Top Gun" pilot buzzing the dome
TBAD permissive triggers are not latched
If TBAD goes to deny (RED), it will go to grant (GREEN) as soon as the condition clears
TBAD permissive triggers do latch in the AO alarm handler
One must acknowledge the permissive change to clear. If not, it may cause confusion later.

TBAD xshow Timestamp Incorrect

Timestamp in TBAD Status xshow is not within 2 minutes of the current time
TBAD is not healthy -- Cease laser operations until fixed!
If propagating:
  1. Run "tbad_log_tail" as k2obsao on k2aoserver. Laser operations my continue if the log file timestamp is current and updating and the following keywords have these values:
    • LSTBDSHUT = grant
    • LSTBDSTATE = grant
    • LSTBDBYPAS = notBypassed
  2. At your earliest convenience, stop propagating and follow the "not propagating" steps
If not propagating:
  1. Do not propagate
  2. Login to k2laserserver as k2obs
  3. cd tbad/logger
  4. Good Status (exactly one process running):
    k2laserserver{k2obs}% status
    16100 /usr/local/bin/python 2
  5. If your status is not good:
    1. Issue: " stop monitor" until " status" returns zero monitor processes.
    2. Issue: " start monitor" to restart the monitor.
    3. Verify the log file timestamp is current and the keywords have the proper values.

Wavefront Controller

Wavefront Controller Crash

WFS intensity does not update
Fault detector may trigger an audio alarm
TT graphs do not update
The wavefront controller may have crashed.
To determine whether there are problems with the WFC, please refer to the Observing with the NGWFC webpage. There are several failure modes associated with the WFC; many of them are not crashes but can be fixed by rebooting the WFC.
To reboot the WFC, click on MAORI → Menu → Menu → Reboot WFC. Then reacquire the star using the AO acquisition widget.

Powering up the tip-tilt mirror

The tip-tilt mirror is being commanded but does not appear to be moving
The tip-tilt mirror is not powered up
Go to the K1 or K2 web-based interface.
Select "Browser Web Control".
Turn all three power supplies on.

Loading new WFS centroid origin calibrations

Need to load new COG file
None, just dunno how to do it!
  1. From an xterm on k2aoserver as k2obsao, type cdcog
  2. Then dir will list all centroid origins calibration files sorted by date.
  3. The name reflects the instrument and the date. For a NIRC2 night in NGS AO mode for 2x2 binning with the 2.4 arcsec plate scale, the default file would be 24NIRC2-N2x2.cog.
  4. In IDL, type LOADCOG,'the-one-you-want.cog'
  5. The new calibration file should be loaded on the system.

Querying TRS

Want to query TRS to extract data or parameters
None, just dunno how to do it!
Refer to the TRS queries webpage.

AO Rotator

Who controls the AO rotator?

The AO rotator needs to be moved to set the sky PA and sometimes fixed if it has FAULTED.
Who should command the rotator moves?
Who should fix it when it faults?

How to detect a ROTATOR FAULT in NGS or LGS AO

At acquistion on ACAM, the OA does not manage to send the reference star to the desired pointing origin from Xguidee
TT offload does not work well; The TT graphs display the TT mirror position off-center, although the WFS average centroids (X, Y) are close to 0.0.
The rotator may not be tracking well and not read yet a FAULT status.
From the AO acquisition widget, run the "reset AO/DCS coms".

Recover from a ROTATOR FAULT in NGS or LGS AO

Rotator status could read FAULT on telescope FACSUM.
An audio message informs the AO operator that the rotator has FAULTED.
Need to fix the rotator and set it back to TRACKING
Drastic Measures

OBS device FAULTs

Why do these stages fault?

The Optics Bench Subsystem (OBS) hosts as many as 21 stages on K2. Most stages have more than 1 axis; their moves need to be fast and accurate (~< 10 microns).
PMAC boards control the way each stages move for each mode (manual, tracking, simul). The OBS works very well most of the times, but the system is not 100% reliable and there are problems from time to time,
Also, there is a basic design flaw in the OBS stages to my mind - and some of it is due to the fact that some stages have coordinated moves - : each motor tries to SLEW to any demanded position, w/o checking whether the demanded position is within the range limits.
Some FAULTs happen because the stage hit its positive or negative SOFTWARE LIMIT.
Other times, it could be a TRACKING problem.
Other times, the stage could be slightly detuned stage and the FAULT is a "max following error"
Let the automated recovery script run for the FSM, the ROTATOR, the FCS [and the TSS?]
The philosophy to recover from a FAULT is:

Repeated FSM FAULTs

The FSM FAULTs are happening for moves that are within the range of the FSM field-of-regard.
The FSM error message displayed in the IDL Observing Tools window reads"Max following error"
One of the FSM motors (FSM1x, FSM1y, FSM2x, FSM2y) could be detuned and/or experiencing problems.
First, one needs to find which FSM motor experiences the problem.
  1. From AO SC GUI, bring an OBS DM screen, click on the FSM button to bring up the FSM dm screen.
  2. Dismiss the AO acquisition widget (to avoid triggering the automated recovery)
  3. Reproduce the FAULT and leave the stage in the FAULT state.
  4. From the FSM dm screen (ecPDev4.dl), note which motor displays a red indicator for the following error.
  5. All FSMS are on PMAC3, FSM1x is on motor5, FSM1y → motor6, FSM2x → motor7, FSM2y → motor8.
  6. From the OBS dm screen, click on PMC3, click on "MOTOR-X IVARs" where X is 5, 6, 7 or 8.
  7. Enter new value for Ix11 in yellow field, increase value by 10 to 20% in a first attempt to solve the problem (300 or 350k). Hit return and click update.
  8. Init the FSM from scgui and try to reproduce the fault.
  9. If the fault persists, call an AO support person.
  10. Don't forget to document the changes to the I variables in the nightlog
  11. Bring up AO acquisition widget and resume operations.

FCS FAULT in tracking mode (LGS AO)

FCS reads fault, or gets into a FAULT state for every new target. Once faulted, the stage does not track.
In LGS AO mode, the FCS stage needs to TRACK as a function of telescope elevation and sodium altitude. If it is not tracking, observing must stop and the problem needs to be fixed.
In NGS AO, the stage does not track so it should be less a problem and have a lesser impact on the observations; yet the fault needs to be cleared as soon as possible.
The FCS fault should be automatically detected by the AO acquisition widget and be reported to the operator. Another script will try to recover automatically from the fault and restore the FCS to its initial state.


TSS faults would occur either:
  1. if the requested offset is out of the TSS range,
  2. if the stage does not transition well to or from servo-in mode
  3. when the stage experiences problem to track in servo-in mode.
It will likely impact the observations and could require to reboot OBS if the TSS tracking mode is required for the observations.
We have developed an automated recovery script for the TSS so hopefully it will work well.. If not:
  1. Turn DAR and TRACKING OFF from LGS AO acq widget
  2. Put the TSS in MANUAL mode from MAORI->Track
  3. from SC GUI, try to send TSS to optsodstrap (yes, it won't go there...)
  4. then click HALT and INIT for the TSS stage
  5. Return the TSS to SERVO mode from MAORI->Track

Other resources and help pages

Where can I learn about all these acronyms??!!

Cindy has put a page together here.
There is also some information posted here.

Observing without ACAM

ACAM dies, its shutter locks in the closed position, or some other problem prevents you from using ACAM. This technique was successfully used in 2003 April 21 to continue observing with a locked shutter.
How do you center stars on the wavefront sensor?
NIRC2 or OSIRIS (imager)
NIRSPAO and other small field instruments
SCAM has such a small field of view that you might easily miss the star on both SCAM and the wavefront sensor. If so, follow this procedure:
  1. Point to the star
  2. Assuming you don't see it on the wavefront sensor, take a WFS sky image here.
  3. Put in PMFM (primary mirror focus mode) of about 500 nm. This will create 36 segment images and increase your chances of finding one.
  4. If you still don't see a segment, either use the handpaddle to search around until the WFS sees a bright segment, or take images with SCAM and look for a segment. The latter was used most successfully in April 2003, but selecting SCAM from xguide and looking for segments in "video mode" might be equally successful.
  5. Once you have a segment on the WFS or on SCAM, try to estimate where the center of the 36-segment PMFM pattern is. Roughly center the pattern (i.e. get several segments on the image, not just one.)
  6. Put in PMFM of 300, and immediately start a SCAM exposure that will last for a few seconds. (You can increase the number of coadds to get the total exposure up to 5 seconds or so.) The moving segments will create streaks that point roughly back towards the pattern center.
  7. Again estimate the pattern center from the radial streaks and recenter.
  8. You can repeat with a PMFM of 150, but once you have the pattern center on SCAM you should be able to go straight to zero PMFM (normal images) and center the star on SCAM.
  9. The star should now be on the WFS. (If it's not, check your FSM -(If it's not, check your FSM position.)
  10. Once the WFS sees the target, you can lock on and proceed with the observations.

I Can't See Light on the WFS!

I Can't See Light on the WFS!

Problems with User Interfaces (IDL, etc)

Problems with User Interfaces