Contents

GENERAL MAGIQ PROBLEMS

Cannot move telescope

Symptom
MAGIQ GUI is running but cannot move telescope.
Problem
There are various possible causes:
  • MAGIQ server not running
  • MAGIQ server running but cannot communicate with DCS
  • TDC/AUX not running
Diagnosis and Recovery
  • Verify that TCD/AUX are running, ie. check FACSUM is displaying information correctly. If not, Stby/Init or reboot TDC/AUX.
  • Check if there are dcs error messages in the log window or log file as shown below. If there are, restart MAGIQ server. 
    2011-01-03 14:07:53.952 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:07:53.952 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:03.962 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:08:03.962 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:13.972 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:08:13.972 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:23.982 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
  • Check the title bar of the MAGIQ GUI window. It shows the name of the server and the connection status. If it says disconnected, then MAGIQ server is not running or not ready (ie short after a restart). Check the MAGIQ server status. If MAGIQ server was restarted, then wait until GUI reconnects, otherwise restart MAGIQ server.
  • Check that fields in the MAGIQ GUI are updating, ie LST, AZ, EL. If not, restart MAGIQ server.

MAGIQ GUI frozen

Symptom
  • GUI looks OK, but the fields are not updating.
  • GUI window is blank or not updating.
  • Button pressed but GUI is not coming back.
Problem
Possible causes are:
  • GUI is busy doing some intensive computing, should freeze more than 5s.
  • Communication with MAGIQ server disconnected.
  • Window manager olwm or fmwm failed.
Diagnosis and Recovery
  • Verify that other GUIs are working, ie FACSUM, Observer UI, DCSUI. If they are OK, then problem is with MAGIQ GUI, restart GUI. If other GUI are also frozen, then restart window manager or logout and login again.
  • Sometimes too many connections to retrieve DSS image or Star catalog can cause the GUI to freeze, restart GUI and avoid retrieveing DSS or star catalog for a few minutes.

GENERAL GUIDER CAMERA PROBLEMS

The general recovery procedure has four steps:
  1. Restart camera/mech/motor servers
  2. If problem persists, power cycle electronics via the DCS/MAGIQ GUI
  3. If problem persists, restart MAGIQ server and MAGIQ GUI
  4. If problem persists, call support

Mechanism Fails to Move or Images/exposures not functioning correctly

Symptom
One of the following symptoms is occuring
  • Mechanism Problems
    • ND Filter wheel doesn't move
    • Photometric Filter wheel doesn't move
    • IQL doesn't move
    • Focus doesn't change
  • Image Exposure Problems
    • Exposure time does not change
    • No IQL images
    • No full frames or guide frames, only IQL frames
Problem
Communication with mech server (galil) failed.
Recovery
To recover from the following known issues:
  • first try to initialize the mechanisms, From the OA GUI top menu bar select "Camera -> Diagnostics." From the Diagnostics pannel select "Init" and then "Load defaults" to put mechanisms at a nominal position.
  • If above fails, power cycle the galils using the DCS/MAGIQ menu option. If the mechanism status is unknown, please then Init and then Load Defaults (power cycling should do this automatically).
  • If above all fails, call support.

Image Quality: Only guide box is visible

Symptom
Only guide box visible while the rest of the image is black.
Problem
No full frames are being read out.
Recovery
Change full interval.

Image Quality: image details washed out / contrast problem

Symptom
Image details are washed out or there is a bad contrast
Problem
Object is either too faint or bright
Recovery
Increase/decrease exposure time or change ND filter

NIRSPEC Specific Problems

The recovery procedure should be used for the following known issues:
Symptom Problem
Can't select nirspecm Camera nirspecm was not available or failed to start up when MAGIQ server started
Not getting images; GUI slow or frozen Camera server not running or frozen
Images look like scrambled puzzles Camera read out and exposure control got out of sync
Noisy images Focus, filter wheel, or IQL motor may remain turned on
Recovery
  1. Restart MAGIQ server from DCSGUI tear-off menu
  2. Wait until MAGIQ UI reconnects (30s) or restart MAGIQ GUI from tear-off menu
  3. Reselect nirspecm
  4. Check settings: camera focus, filters, exposure time, binning
  5. If the problem persists then restart camera server nirspecm from DCSGUI MAGIQ tear-off menu
  6. If the problem persists, power cycle camera electronics and Galil controller from DCSGUI MAGIQ tear-off menu
  7. Check settings: camera focus, filters, exposure time, binning. Click "Init" and then "Load Defaults" if mechanism states say "unknown" in status display.
  8. If the problem persists, call support

NIRSPEC: No SCAM images

Symptom
Not receiving SCAM images in the MAGIQ gui. MAGIQ log in GUI may show RED errors.
Problem
SCAM may not be running or there was a startup problem.
Recovery
First try this (Ensure SCAMGUIDE is on):
  1. Stop Exp from MAGIQ GUI
  2. Select NULL camera
  3. Login to waimea (as nirspec or nspeceng, etc)
  4. Issue m scamguide=1
  5. Select scam camera from MAGIQ GUI
  6. Try to expose
Next try this (Restart SCAM cam_server):
  1. Stop Exp from MAGIQ GUI
  2. Select NULL camera
  3. Select "NIRSPEC" -> "Restart SCAM" from DCSGUI MAGIQ tear-off menu
  4. Wait about 15 seconds
  5. Login to waimea(as nirspec or nspeceng, etc)
  6. Issue m scamguide=1
  7. Select scam camera from MAGIQ GUI
  8. Try to expose

LRIS

LRIS (same as LRIS-ADC) has two MAGIQ cameras: lrisSlit and lrisOffset.

LRIS guider image acquisition failed with the Slitmask Alignment Tool

Symptom
Guider images are not acquired and displayed in external guis such as the LRIS Slitmask Alignment Tool
Problem
  1. MAGIQ Keyword server is not functioning.
  2. MAGIQ server is not connecting to the keyword server
Recovery
Restart the MAGIQ keyword server and MAGIQ server in that order:
  • log into k1-magiq-server
  • run magiqAdmin start MagiqKwd
  • from MAGIQ pulldown menu select Restart Server

Troubleshooting procedures for non-MAGIQ cameras

HOW-TOs

To start/stop MAGIQ server

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu.

To start/stop MAGIQ GUI

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu. See above.

Note: When stopping and starting the MAGIQ server, it is recommended to verify that the processes are actually stopped and restarted by checking MAGIQ status. See below.

To start/stop MAGIQ camera servers

When MAGIQ server starts, it checks if the needed camera servers are running and starts them if necessary. No extra manual steps are necessary. When MAGIQ server stops, it does not stop the camera servers. To recover from an error or to trouble shoot problems, start or stop MAGIQ camera and mech servers via DCS GUI MAGIQ tear-off menu. Example for NIRSPEC:

Only one set of camera servers can be running at the same time for one camera. To ensure that the start-up script first forces any running camera servers to quit before starting a new one. This is why the menu to start the camera server is labeled with "Restart". The menu item "Power cycle" is available for those cases when a simple restart does not restore the camera functionality and a full power cycle is required. A power cycle always includes:

In some cases, a manual restart of the MAGIQ server and MAGIQ OA GUI is necessary after a power cycle of the camera electronics and Galil controller.

To check MAGIQ Status

There are several ways to check MAGIQ status:
  1. MAGIQ status web page
  2. Keck weather page
  3. Via DCSGUI MAGIQ tear-off menu button "Check MAGIQ status"
  4. From an xterm on k1-magiq-server or k2-magiqserver as k1obs or k2obs:
    • magiq status

    • The example above shows two GUIs: MAGIQ OA GUI and Observer GUI. MAGIQ server is running. NIRSPEC camera and mech servers are running on k2-magiq-camserver1. And MAGIQ keyword server is running. Note that there are two cam_rpc_svc processes for each camera server.

Power control

The DCSGUI MAGIQ tear-off menu provides the buttons to power cycle the NIRSPEC MAGIQ camera electronics and Galil controller. See magiq script section for more details. NIRSPEC SCAM SCAM is available via the legacy camera server. To use SCAM the NIRSPEC keyword scamguide must be set to 1. This is usually done via the NIRSPEC eavesdrop GUI. The DCSGUI MAGIQ tear-off menu has an option to start the SCAM legacy camera server. See above.

MAGIQ SCRIPT

Note: The commands and scripts description in this section and in the next section are intended to be invoked via GUIs and for trouble shooting purposes. In normal operation, there is not need to use these scripts directly. The magiq script provides the means to manage MAGIQ processes and it is available on k1-magiq-server or k2-magiq-server. The general syntax is: 
 magiq command Subsystem [parameters]
where command is one the following: and Subsystem is one of the following:

MagiqServer

There can be only one MagiqServer running.
magiq start MagiqServer
magiq restart MagiqServer
Starts the MagiqServer. The script always kills any running MAGIQ server before starting a new one. Thus start is same as restart.
magiq stop MagiqServer
Stops the MagiqServer.

MagiqGui

Multiple MagiqGui can run simultaneously.
magiq start MagiqGui
Starts a new MAGIQ OA GUI using the current DISPLAY environment variable.
magiq stop MagiqGui
Stops ALL running MAGIQ OA GUIs.
magiq restart MagiqGui
Stops ALL running MAGIQ OA GUIs and start ONE MAGIQ OA GUI.

ObserverUI

Multiple ObserverUI can run simultaneously.
magiq start ObserverUI
Starts a new MAGIQ Observer GUI using the current DISPLAY environment variable.
magiq stop ObserverUI
Stops ALL running MAGIQ Observer GUIs.
magiq restart ObserverUI
Stops ALL running MAGIQ Observer GUIs and start ONE MAGIQ Observer GUI.

IqmUI

Multiple IqmUI can run simultaneously.
magiq start IqmUI
Starts a new MAGIQ IQM GUI using the current DISPLAY environment variable.
magiq stop IqmUI
Stops ALL running MAGIQ IQM GUIs.
magiq restart IqmUI
Stops ALL running MAGIQ IQM GUIs and start ONE MAGIQ IQM GUI.

CameraDiagnostic

Multiple CameraDiagnostic can run simultaneously.
magiq start CameraDiagnostic
Starts a new MAGIQ CameraDiagnostic GUI using the current DISPLAY environment variable.
magiq stop CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs.
magiq restart CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs and start ONE MAGIQ CameraDiagnostic GUI.

MagiqCacheGenerator

The cache generator caches catalog and image archive look-up results on local disk to speed up subsequent look-ups. Multiple cache generators can be running simultaneously. The cache directory is:
magiq start MagiqCacheGenerator star-list catalog-name dss-name
Starts the MAGIQ cache generator. Star-list is the full pathname of the star list in Keck format. Catalog-name is one of:
  • GSC23 (online)
  • NOMAD (online)
  • 2MASS (online)
  • USNOB10 (online)
  • VGSC2 (online) Vizier GSC2
  • GSC (local)
  • SAO (local)
  • HIP (local)
dss-name is one of:
  • DSS2R (online)
  • DSS2I (online)
  • DSS2B (online)

MagiqKwd

The MAGIQ keyword agent broadcasts MAGIQ keywords to an UNIX-IOC and listens for MAGIQ keyword changes and forwards them to the MAGIQ server. There can be only one MAGIQ keyword agent running.
magiq start MagiqKwd
Starts the MAGIQ Keyword Agent.
magiq stop MagiqKwd
Stops the MAGIQ Keyword Agent.
magiq restart MagiqKwd
Stops the MAGIQ Keyword Agent and restarts a new one.

Audio

The MAGIQ audio alarm script magiqAudio is a shell script that monitors the MagiqServer.log file and issues an audio alarm by playing an .au file. There can be only one audio alarm script running.
magiq start Audio
Starts the MAGIQ audio alarm script.
magiq stop Audio
Stops the MAGIQ audio alarm script.
magiq restart Audio
Stops the MAGIQ audio alarm script and restarts a new one.

CameraServer

The MAGIQ script invokes another script called magiqCamera, which runs k1-magiq-camserver1 or k2-magiq-camserver2 dependent on the telescope. The magiqCamera script starts and stops the actual camera server and also controls the power of the camera electronics and Galil controller.
magiq start CameraServer camera_host camera_name
Starts the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers. For example: 
magiq start CameraServer k2-magiq-camserver1 nirspecm
magiq start CameraServer k1-magiq-camserver1 lrisSlit
magiq start CameraServer k1-magiq-camserver1 lrisOffset
magiq stop CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers.
magiq restart CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host and restarts a new one. This includes the needed motion/mech servers.
magiq poweroff CameraServer camera_host camera_name
Turns off power to camera_name on camera_host. This affects camera electronics only. For example: 
magiq poweroff CameraServer k1-magiq-camserver1 lrisSlit
magiq poweroff CameraServer k1-magiq-camserver1 lrisOffset
magiq poweroff CameraServer k2-magiq-camserver1 nirspecm
magiq poweron CameraServer camera_host camera_name
Turns on power to camera_name on camera_host. This affects camera electronics only.
magiq powercycle CameraServer camera_host camera_name
Turns off power to camera_name on camera_host and turns power back on. This affects camera electronics only.

MechServer

magiq poweroff MechServer camera_host camera_name
Stops camera/mech server(s) and turns off power to Galil controller. For example: 
magiq poweroff MechServer k2-magiq-camserver1 nirspecm
stops the camera server, the motion control server and the mechanism server and then turns off power to the Galil controller.
magiq poweron MechServer camera_host camera_name
  • Stops the camera/mech server(s) and turns on power to Galil controller and camera electronics.
  • Initializes the EDT board via initcam from EDT.
  • Then restarts the camera/mech servers(s).
  • Then initializes mechanisms.
magiq powercycle MechServer camera_host camera_name
  • Stops the camera/mech server(s) and turns on power to Galil controller and camera electronics.
  • Initializes the EDT board via initcam from EDT.
  • Then restarts the camera/mech servers(s).
  • Then initializes mechanisms.
Note: powercycle calls poweroff and then poweron.
magiq powercycle_restart MechServer camera_host camera_name
Same as powercycle, but it stops the MAGIQ server first and then restarts the MAGIQ server after power cyling the mech server.

MAGIQ CAMERA SCRIPT

MAGIQ CAMERA SCRIPT

The MAGIQ camera script, magiqCamera, is available on k1-magiq-camserver1 and k2-magiq-camserver1. It is invoked by the MAGIQ script. It can be invoked manually also. This script sets up the environment variables and calls other instrument dependent scripts. The syntax is: 
magiqCamera command camera-name
where command is one of: and camera-name is one of:
Examples

MAGIQ CAMERA CONFIGURATION FILES

Camera Server Configuration File

The configuration file for the camera server is located at: 
$KROOT/magiq/default/cam_config.xml
on the camera server host. This configuration is a XML file containing information of all MAGIQ cameras (for one telescope) including:

Mech Server Configuration Files

The mech/motor servers require the following configuration files located at: 
$KROOT/rel/default/data
on the camera server host: Configuration information includes:

MAGIQ Server Configuration Files

MAGIQ server configuration files are located on the MAGIQ server host at: 
$KROOT/magiq/default
The file magiqConfig.xml.k2server includes the following information:

To add a new instrument

To add a new guider camera