| Fault | Mode(s) | Keyword(s)/Channel(s) | Autorecovery (if any) | Telescope(s) | Change since previous version | Autorecovery time (N/A if not autorecovery) | |||
| RTC-CE connection lost while RTC is NORMAL | NGS and LGS | ao.trssshsts | No autorecovery. When ao.wcstat is NORMAL and the TRS ssh status is OFF, shows a dialog telling the user the kXaoserver-new -> RTC-CE connection is lost and to reboot the RTC if anomalies are seen. Logs the UT time for the nightlog. The alarm is then silenced for 30 min (1800 check cycles). | K1 and K2 | New - not in previous version | N/A | |||
| ao.wcstat | |||||||||
| RTC-TRS connection lost while RTC is NORMAL | NGS and LGS | ao.trstgtsts | No autorecovery. Shows a dialog asking the user to reboot the TRS from MAORI->Menu->RTC->Reboot TRS if there are anomalies or frozen GUIs, and to reboot the RTC if that does not fix it. Logs the UT time for the nightlog. The alarm is then silenced for 30 min (1800 check cycles). | K1 and K2 | New - not in previous version | N/A | |||
| ao.wcstat | |||||||||
| Whether the ALPAO DM electronics counters stop incrementing while the DM loop is closed | NGS and LGS | ao.AME0HCNT | No autorecovery (loops are not opened; that line is commented out). Shows a dialog saying the DM electronics counter has paused and asking the user to open the loop and reboot the DM electronics, or run Reboot Hardware from MAORI -> RTC. The user can ignore it if the RTC is being reset or rebooted. Logs the UT time. The alarm is then silenced for 30 min. | K2 only | New - not in previous version | N/A | |||
| ao.AME1HCNT | |||||||||
| ao.AME2HCNT | |||||||||
| ao.AME3HCNT | |||||||||
| ao.dmlp | |||||||||
| Whether overillumination protection has been triggered while loops are closed | NGS and LGS | ao.o1protec | Runs when ao.o1protec is ON, the DM loop is closed and the EM gain is not 1. Opens loops, then writes the current EM gain back to ao.o1smgn to clear the protection, waits 1 s and checks ao.o1protec again. If it cleared, the user is told to close loops manually if there is enough light on the WFS. If not, a dialog says 'Star is too bright for the WFS. Please reduce the EM gain or increase the frame rate (on MAORI) till counts are less than 8000 on the WFS and take a new background before closing loops'. | K1 and K2 | Updated - autorecovery now tries to clear the protection before warning | < 10 seconds | |||
| ao.dmlp | |||||||||
| ao.o1smgn | |||||||||
| Whether OCAM2K flow fault is triggered | NGS and LGS | kX:ao:wc:env:im:ai1 (0 = good, 24 = bad) | Opens loops and shows a message asking the user to turn on OCAM2K from MAORI->Menu->OCAM2K when glycol flow is back to normal. Tells the user to check flow status on the RTC environment display (the message says k2rtcenv.dsp on both telescopes). This check runs whether or not the loops are closed. | K1 and K2 | Updated - details added | N/A | |||
| Whether BM2 is saturated during closed loop | LGS (ao.aoopsmode = 2) | ao.aoopsmode | No autorecovery. When the uplink TT loop is closed in LGS and |bm2Xcmd| or |bm2Ycmd| stays above 7 for more than 7 s, shows a dialog: 'BM2 is saturated for more than 7 seconds of closed loop. Please adjust BM2 C/F in the FST->BM2->Conversion Params to de-saturate'. Logs the UT time. | K1 and K2 (k1: / k2: EPICS channels) | New - not in previous version | N/A | |||
| ao.utlp | |||||||||
| kX:ao:ls:bm2:bm2Xcmd | |||||||||
| kX:ao:ls:bm2:bm2Ycmd | |||||||||
| Check for TT/HVC fault | NGS and LGS | ao.hvcstat | Runs when ao.hvcstat is ERROR. Opens loops and runs hvcrecovery, which steps the HVC ERROR->LOADED->STANDBY->NORMAL and shows a dialog before each attempt. It tries up to 5 times. If the HVC is still in ERROR, it asks the user to reset the RTC pipeline from MAORI->Menu->RTC and then tries again. At the end it reports whether the TT/HVC recovered or failed. | K1 and K2 | Updated - corrected sequence and retry logic | < 2 minutes | |||
| Check for STRAP health when loops are closed in LGS but STRAP HV is uninitialized | LGS | ao.sthvstat | Runs when the DT loop is closed, ao.ststate or ao.sthvstat is 0, and the RTC is in an LGS mode. On K2 that means wcmode 13 (LGS_57x57) or 15 (LGS_29X29), updated May 2026 for HAKA. On K1 it means wcmode 6 (LGS_ocam2k), 9 (GLAO) or 10 (LTAO). Opens loops and runs the recover STRAP script (rtcRecoverSTRAP). It then checks STRAP again and either says STRAP recovered or asks the user to check kXstrap.dsp, initialize STRAP from MAORI -> WFC, and reboot the RTC if needed. | K1 and K2 | Updated - ao.wcmode condition added (differs K1/K2) | < 2 minutes | |||
| ao.ststate | |||||||||
| ao.dtlp | |||||||||
| ao.wcmode | |||||||||
| Check for FSM fault | NGS and LGS | ao.obfmstst | Opens loops, initializes the FSM and sets it to 0,0 (check_fsm). Logs the FSM error string. The user closes loops manually. | K1 and K2 | Updated - loops opened; error string logged | < 1 minute | |||
| ao.obfmerrs (logged) | |||||||||
| Check for TSS fault | NGS-ST and LGS (ao.aoopsmode = 1 or 2) | ao.obtsstst | Opens loops and initializes the TSS (check_tss). Then tells the user to reset AO/DCS comms and re-acquire the star. | K1 and K2 | Updated - mode corrected; loops opened | <3 minutes | |||
| ao.obtserrs (logged) | |||||||||
| Check for LBS fault | NGS-ST and LGS (ao.aoopsmode = 1 or 2) | ao.oblbstst | Initializes the LBS (CHECK_DEVICE 'lbs') and tells the user to resume AO operations. | K1 and K2 | Updated - mode corrected | <3 minutes | |||
| ao.oblberrs (logged) | |||||||||
| Check for FCS fault | NGS and LGS | ao.obwfstst | Initializes the FCS (check_fcs) and tells the user to resume AO operations. | K1 and K2 | No change | <3 minutes | |||
| ao.obwferrs (logged) | |||||||||
| Check for Rotator Fault | NGS and LGS | ao.obrtstst | Opens loops right away and initializes the rotator (CHECK_DEVICE 'rot'). Logs the error string and rotator position. The user closes loops manually. | K1 and K2 | Updated - loops opened | <3 minutes | |||
| ao.obrterrs, ao.obrt (logged) | |||||||||
| Check for Rotator Stall | NGS and LGS | ao.obrtstal | Runs only when the DT loop is closed and the rotator is TRACKing with the stall flag set. There is no dialog: the fault detector prints a console warning and sounds an alarm advising the user to re-initialize the rotator if stalls happen often. | K1 and K2 | Updated - corrected (console message and sound, not a dialog) | N/A | |||
| ao.obrtstst | |||||||||
| ao.dtlp | |||||||||
| Check for FCS stall | NGS and LGS | ao.aofclgct | Dialog message which
advises the following: 'An FCS stall has been detected. Please clear the
stall and continue. The fault may be cleared by 1. Setting FCS to MANUAL 2. Moving the FCS stage 3. Setting FCS to TRACKING'. The fault detector then pauses for 10 s. |
K1 and K2 (K1 also requires both keywords to read successfully) | Minor update | N/A | |||
| ao.obwfstal | |||||||||
| Checking for STRAP gate being 100% open and STRAP health (overcurrent, overcounts, EEPROM and temperature control status) | NGS-ST and LGS (ao.dtsensor = STRAP) | ao.dtsensor | Runs only when the DT sensor is STRAP and the DT loop is closed. Gate: if ao.stgtact is not 100%, the fault detector sets ao.stgate to Open, waits 4 s and checks again. Only if the gate is still not at 100% does it show a dialog asking the user to open the DT loop, attenuate the light with the SFW, open the STRAP gate and close the loop. Health: it shows an error dialog for overcurrent, overcounts, temperature control OFF, overtemperature or EEPROM failure. There is no autorecovery for health faults. | K1 and K2 | Updated - automatic gate re-open attempt added | N/A | |||
| ao.dtlp | |||||||||
| ao.stgtact | |||||||||
| ao.stgate | |||||||||
| ao.stocrstat | |||||||||
| ao.stoctstat | |||||||||
| ao.sttcstat | |||||||||
| ao.stotstat | |||||||||
| ao.stepstat | |||||||||
| Check that the AU keyword server is alive | NGS and LGS | ao.aotel | If ao.aotel cannot be read, runs restart_aus to restart the AU keyword server. | K1 and K2 | New - not in previous version | <3 minutes | |||
| Check for RTC Fault | NGS and LGS | ao.wcstat | If ao.wcstat is ERROR: shows a dialog, resets the pipeline (kXrtcreset) and then checks the RTC state (CHECKWFC). It reports success or asks the user to reboot the RTC. If ao.wcdmstat or ao.wcdtstat is in ERROR: shows an error dialog, runs the Initialize RTC script (kXrtcInit.csh) and checks the RTC state again. It reports success or asks the user to reset the pipeline or reboot the RTC. K1 only: the check is skipped when these keywords are disconnected, to avoid false alarms. | K1 and K2 | Updated - verification step added; K1 disconnect guard | <3 minutes | |||
| ao.wcdmstat | |||||||||
| ao.wcdtstat | |||||||||
| Check for STRAP communication problem | NGS and LGS | ao.stinttim | No autorecovery. If ao.stinttim cannot be read, waits 3 s and tries again. If it still cannot be read, shows a dialog asking the user to check the STRAP status display and STRAP GUI screens, and to reboot if needed. | K1 and K2 | New - not in previous version | < 10 seconds | |||
| Legend: | |||||||||
| New check (not in previous version) | |||||||||
| Existing check - description updated/corrected | |||||||||
| Source: /kroot/rel/ao/default/bin/linux-x86_64/faultdetector.pro (-> ao/ops/default) on k1ao and k2ao, plus the routines it calls (detectwfcfault, checkwfc, hvcrecovery, check_au_kwds). Reviewed 2026-09-30. kX = k1 or k2 depending on telescope. | |||||||||