GENERAL MAGIQ PROBLEMS

Cannot move telescope

Symptom:

MAGIQ GUI is running but cannot move telescope.

Problem:

There are various possible causes:

Diagnosis and Recovery:

MAGIQ GUI frozen

Symptons:

Problem:

Diagnosis and Recovery:

GENERAL GUIDER CAMERA PROBLEMS

The general recovery procedure has four steps:
  1. Restart camera/mech/motor servers
  2. If problem persists, power cycle electronics via the DCS/MAGIQ GUI
  3. If problem persists, restart MAGIQ server and MAGIQ GUI
  4. If problem persists, call support

Mechanism Fails to Move or Images/exposures not functioning correctly

Symptom:

One of the following symptoms is occuring

Problem:

Communication with mech server (galil) failed.

Recovery:

To recover from the following known issues,

Image Quality: Only guide box is visible

Symptom:
Only guide box visible while the rest of the image is black.

Problem:

No full frames are being read out.

Recovery:

Change full interval.

Image Quality: image details washed out / contrast problem

Symptom:
Image details are washed out or there is a bad contrast

Problem:

Object is either too faint or bright

Recovery:

Increase/decrease exposure time or change ND filter

NIRSPEC Specific Problems

The recovery procedure should be used for the following known issues:
Symptom: Problem:
Can't select nirspecm Camera nirspecm was not available or failed to start up when MAGIQ server started
Not getting images GUI slow or frozen Camera server not running or frozen
Images look like scrambled puzzles Camera read out and exposure control got out of sync
Noisy images Focus, filter wheel, or IQL motor may remain turned on

Recovery:

  1. Restart MAGIQ server from DCSGUI tear-off menu
  2. Wait until MAGIQ UI reconnects (30s) or restart MAGIQ GUI from tear-off menu
  3. Reselect nirspecm
  4. Check settings: camera focus, filters, exposure time, binning
  5. If the problem persists then restart camera server nirspecm from DCSGUI MAGIQ tear-off menu
  6. If the problem persists, power cycle camera electronics and Galil controller from DCSGUI MAGIQ tear-off menu
  7. Check settings: camera focus, filters, exposure time, binning. Click "Init" and then "Load Defaults" if mechanism states say "unknown" in status display.
  8. If the problem persists, call support

NIRSPEC: No SCAM images

Not receiving SCAM images in the MAGIQ gui.

Problem:

SCAM may not be running or there was a startup problem.

Recovery:

Restart SCAM legacy camera server from DCSGUI MAGIQ tear-off menu, and retry camera acquisitions.

LRIS

LRIS (same as LRISADC) has two MAGIQ cameras: lrisSlit and lrisOffset.

LRIS guider image acquisition failed with the Slitmask Alignment Tool

Symptom:
If Guider images are not acquired and displayed in external guis such as the LRIS Slitmask Alignment Tool, there are two possible reasons:

Problem:

Recovery:

Restart the MAGIQ keyword server and MAGIQ server in that order:

Other non MAGIQ cameras

HIRES GPIB camera. See here.
DEIMOS PXL camera. See here.
ESI PXL camera. See here.
AO PXL camera. See here.
SSC GPIB camera. See here

HOW-TO'S

To start/stop MAGIQ server

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu.

To start/stop MAGIQ GUI

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu. See above.
Note:When stopping and starting the MAGIQ server, it is recommended to verify that the processes are actually stopped and restarted by checking MAGIQ status. See below.

To start/stop MAGIQ camera servers

When MAGIQ server starts, it checks if the needed camera servers are running and starts them if necessary. No extra manual steps are necessary. When MAGIQ server stops, it does not stop the camera servers. To recover from an error or to trouble shoot problems, start or stop MAGIQ camera and mech servers via DCS GUI MAGIQ tear-off menu. See below. Example for NIRSPEC:

Only one set of camera servers can be running at the same time for one camera. To ensure that the start-up script first forces any running camera servers to quit before starting a new one. This is why the menu to start the camera server is labeled with "Restart". The menu item "Power cycle" is available for those cases when a simple restart does not restore the camera functionality and a full power cycle is required. A power cycle always includes:

To check MAGIQ Status

There are two ways to check MAGIQ status:
  1. Via DCSGUI MAGIQ tear-off menu button "Check MAGIQ status"
  2. From a xterm on k1-magiq-server or k2-magiqserver as k1obs or k2obs:
    • magiq status

    • The example above shows two GUIs: MAGIQ OA GUI and Observer GUI. MAGIQ server is running. NIRSPEC camera and mech servers are running on k2-magiq-camserver1. And MAGIQ keyword server is running. Note that there are two cam_rpc_svc processes for each camera server.

MAGIQ status web page
Another way to check the status is through the MAGIQ status web page:
http://www.keck.hawaii.edu/observing/magiqStatus/magiqStatus.php

MAGIQ status is also displayed on the weather page:
http://www.keck.hawaii.edu/observing/weather/weather2/index.php

Power control

The DCSGUI MAGIQ tear-off menu provides the buttons to power cycle the NIRSPEC MAGIQ camera electronics and Galil controller. See magiq script section for more details. NIRSPEC SCAM SCAM is available via the legacy camera server. To use SCAM the NIRSPEC keyword scamguide must be set to 1. This is usually done via the NIRSPEC eavesdrop GUI. The DCSGUI MAGIQ tear-off menu has an option to start the SCAM legacy camera server. See above.

MAGIQ SCRIPT

Note: The commands and scripts description in this section and in the next section are intended to be invoked via GUIs and for trouble shooting purposes. In normal operation, there is not need to use these scripts directly. The magiq script provides the means to manage MAGIQ processes and it is available on k1-magiq-server or k2-magiq-server. The general syntax is:

magiq command Subsystem [parameters]
command:

MagiqServer

There can be only one MagiqServer running.

magiq start MagiqServer
magiq restart MagiqServer
Starts the MagiqServer. The script always kills any running MAGIQ server before starting a new one. Thus start is same as restart.

magiq stop MagiqServer
Stops the MagiqServer.

MagiqGui

Multiple MagiqGui can run simultaneously.

magiq start MagiqGui
Starts a new MAGIQ OA GUI using the current DISPLAY environment variable.

magiq stop MagiqGui
Stops ALL running MAGIQ OA GUIs.

magiq restart MagiqGui
Stops ALL running MAGIQ OA GUIs and start ONE MAGIQ OA GUI.

ObserverUI

Multiple OvserverUI can run simultaneously.

magiq start ObserverUI
Starts a new MAGIQ Observer GUI using the current DISPLAY environment variable.

magiq stop ObserverUI
Stops ALL running MAGIQ Observer GUIs.

magiq restart ObserverUI
Stops ALL running MAGIQ Observer GUIs and start ONE MAGIQ Observer GUI.

IqmUI

Multiple IqmUI can run simultaneously.

magiq start IqmUI
Starts a new MAGIQ IQM GUI using the current DISPLAY environment variable.

magiq stop IqmUI
Stops ALL running MAGIQ IQM GUIs.

magiq restart IqmUI
Stops ALL running MAGIQ IQM GUIs and start ONE MAGIQ IQM GUI.

CameraDiagnostic

Multiple CameraDiagnostic can run simultaneously.

magiq start CameraDiagnostic
Starts a new MAGIQ CameraDiagnostic GUI using the current DISPLAY environment variable.

magiq stop CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs.

magiq restart CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs and start ONE MAGIQ CameraDiagnostic GUI.

MagiqCacheGenerator

The cache generator caches catalog and image archive look-up results on local disk to speed up subsequent look-ups. Multiple cache generators can be running simultaneously. The cache directory is:

MagiqKwd

The MAGIQ keyword agent broadcasts MAGIQ keywords to an UNIX-IOC and listens for MAGIQ keyword changes and forwards them to the MAGIQ server. There can be only one MAGIQ keyword agent running.

magiq start MagiqKwd
Starts the MAGIQ Keyword Agent.

magiq stop MagiqKwd
Stops the MAGIQ Keyword Agent.

magiq restart MagiqKwd
Stops the MAGIQ Keyword Agent and restarts a new one.

Audio

The MAGIQ audio alarm script magiqAudio is a shell script that monitors the MagiqServer.log file and issues an audio alarm by playing an .au file. There can be only one audio alarm script running.

magiq start Audio
Starts the MAGIQ audio alarm script.

magiq stop Audio
Stops the MAGIQ audio alarm script.

magiq restart Audio
Stops the MAGIQ audio alarm script and restarts a new one.

CameraServer

The MAGIQ script invokes another script called magiqCamera, which runs k1-magiq-camserver1 or k2-magiq-camserver2 dependent on the telescope. The magiqCamera script starts and stops the actual camera server and also controls the power of the camera electronics and Galil controller.

magiq start CameraServer camera_host camera_name
Starts the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers.
For example:
magiq start CameraServer k2-magiq-camserver1 nirspecm

magiq start CameraServer k1-magiq-camserver1 lrisSlit

magiq start CameraServer k1-magiq-camserver1 lrisOffset

magiq stop CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers.

magiq restart CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host and restarts a new one. This includes the needed motion/mech servers.

magiq poweroff CameraServer camera_host camera_name
Turns off power to camera_name on camera_host. This affects camera electronics only.
For example:
magiq poweroff CameraServer k1-magiq-camserver1 lrisSlit

magiq poweroff CameraServer k1-magiq-camserver1 lrisOffset

magiq poweroff CameraServer k2-magiq-camserver1 nirspecm

magiq poweron CameraServer camera_host camera_name
Turns on power to camera_name on camera_host. This affects camera electronics only.

magiq powercycle CameraServer camera_host camera_name
Turns off power to camera_name on camera_host and turns power back on. This affects camera electronics only.

Galil Power

magiq poweroff MechServer camera_host camera_name
Stops camera/mech server(s) and turns off power to Galil controller. For example:

magiq poweroff MechServer k2-magiq-camserver1 nirspecm
Stops the camera server, the motion control server and the mechanism server and then turns off power to the Galil controller.

magiq poweron MechServer camera_host camera_name

MAGIQ CAMERA SCRIPT

MAGIQ CAMERA SCRIPT

The MAGIQ camera script, magiqCamera, is available on k1-magiq-camserver1 and k2-magiq-camserver1. It is invoked by the MAGIQ script. It can be invoked manually also. This script sets up the environment variables and calls other instrument dependent scripts. The syntax is:

magiqCamera command camera-name
command is one of:


Examples:

magiqCamera start lrisSlit
magiqCamera start lrisOffset

magiqCamera start nirspecm
The script checks if the camera/mech servers are already running, if not then it starts camera server and mech/motion control servers and initializes mechanisms.

magiqCamera stop lrisSlit
magiqCamera stop lrisOffset

magiqCamera stop nirspecm
Stops the camera/mech/motion servers and delete the corresponding entries from the RPC info table.

magiqCamera restart lrisSlit
magiqCamera restart lrisOffset

magiqCamera restart nirspecm
Stops the camera/mech/motion servers and restarts them, and initializes the mechanisms.
magiqCamera status
Shows all running camera/mech/motion server processes.

MAGIQ CAMERA CONFIGURATION FILES

Camera Server Configuration File

The configuration file for the camera server is located at:
$KROOT/magiq/default/cam_config.xml

on the camera server host. This configuration is a XML file containing information of all MAGIQ cameras (for one telescope) including:

Mech Server Configuration Files

The mech/motor servers require the following configuration files located at:

MAGIQ Server Configuration Files

MAGIQ server configuration files are located on the MAGIQ server host at:
$KROOT/magiq/default
  1. The directory configData contains XML files describing instruments and their guider cameras.
    The file Telescopes.xml contains the list of telescopes. In the case of Keck 1 or Keck 2, there is only one telescope in the list. This is to avoid confusion and redundant copies of configuration files. The file KECK1.xml or KECK2.xml contains the list of instruments that are available for the given telescope. The configuration files for the instruments are located in the directory KECK1 and KECK2 respectively.
    An instrument configuration includes an XML file and a directory of the same name. The XML file lists the guider cameras and subsystems. The information on subsystems are included for completeness and not used at this time. The instrument directory contains an XML file describing pointing origins. This description is only used for off-sky operations (observing planning or testing). When on-sky the pointing origin information is given via keywords that supersede the static information from the XML file.
    The directory Cameras contains configuration for guider cameras available for the given telescope. The list of these cameras is given in the file cameras.xml. Each camera has its own XML configuration file, which includes REF x/y and default pointing origin name, plate scale and other information.
    The file magiqConfig.xml.k2server includes the following information:
    • Telescope name
    • Update rates for various event loops
    • Star catalog cache directory
    • Fits header description file
    • Directory containing configData
    • Logging level
    • Simulation modes
    • Centroiding algorithms
    • Flexure compensation parameter file location
    • Keyword services

    To add a new instrument:

    • Choose a name for the new instrument, for example INSTR123.
    • Create XML file INSTR123.xml containing list of guider cameras.
    • Create directory INSTR123 containing the PointingOrigins.xml file.
    • Add INSTR123 in KECK1.xml or KECK2.xml

    To add a new guider camera:

    • Choose a new for the new camera, for example CAM123.
    • Create a XML configuration file CAM123.xml including reference pixel location, and the default pointing origin, plate scale, tv angle and flip.
    • Add CAM123 in Cameras/cameras.xml.