GENERAL MAGIQ PROBLEMS

Cannot move telescope

Symptom:

MAGIQ GUI is running but cannot move telescope.

Problem:

There are various possible causes:

MAGIQ server not running
MAGIQ server running but cannot communicate with DCS
TDC/AUX not running

Diagnosis and Recovery:

Verify that TCD/AUX are running, ie. check FACSUM is displaying information correctly. If not, Stby/Init or reboot TDC/AUX.

Check if there are dcs error messages in the log window or log file as shown below. If there are, restart MAGIQ server.

2011-01-03 14:07:53.952 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:07:53.952 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:03.962 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:08:03.962 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:13.972 [Thread-1] INFO : Restarting dcs2 Probe failed: 0
2011-01-03 14:08:13.972 [Thread-1] INFO : Shutting down gateway to dcs2
2011-01-03 14:08:23.982 [Thread-1] INFO : Restarting dcs2 Probe failed: 0

Check the title bar of the MAGIQ GUI window. It shows the name of the server and the connection status. If it says disconnected, then MAGIQ server is not running or not ready (ie short after a restart). Check the MAGIQ server status. If MAGIQ server was restarted, then wait until GUI reconnects, otherwise restart MAGIQ server.
Check that fields in the MAGIQ GUI are updating, ie LST, AZ, EL. If not, restart MAGIQ server.

MAGIQ GUI frozen

Symptons:

GUI looks OK, but the fields are not updating.
GUI window is blank or not updating.
Button pressed but GUI is not coming back.

Problem:

GUI is busy doing some intensive computing, should freeze more than 5s.
Communication with MAGIQ server disconnected.
Window manager olwm or fmwm failed.

Diagnosis and Recovery:

Verify that other GUIs are working, ie FACSUM, Observer UI, DCSUI. If they are OK, then problem is with MAGIQ GUI, restart GUI. If other GUI are also frozen, then restart window manager or logout and login again.
Sometimes too many connections to retrieve DSS image or Star catalog can cause the GUI to freeze, restart GUI and avoid retrieveing DSS or star catalog for a few minutes.

GENERAL GUIDER CAMERA PROBLEMS

The general recovery procedure has four steps:

Restart camera/mech/motor servers
If problem persists, power cycle electronics via the DCS/MAGIQ GUI
If problem persists, restart MAGIQ server and MAGIQ GUI
If problem persists, call support

Mechanism Fails to Move or Images/exposures not functioning correctly

Symptom:

One of the following symptoms is occuring

ND Filter wheel doesn't move
Photometric Filter wheel doesn't move
IQL doesn't move
Focus doesn't change

Exposure time does not change
No IQL images
No full frames or guide frames, only IQL frames

Problem:

Communication with mech server (galil) failed.

Recovery:

To recover from the following known issues,

first try to initialize the mechanisms, From the OA GUI top menu bar select "Camera -> Diagnostics." From the Diagnostics pannel select "Init" and then "Load defaults" to put mechanisms at a nominal position.
If above fails, power cycle the galils using the DCS/MAGIQ menu option. If the mechanism status is unknown, please then Init and then Load Defaults (power cycling should do this automatically).
If above all fails, call support.

Image Quality: Only guide box is visible

Symptom:

Only guide box visible while the rest of the image is black.

Problem:

No full frames are being read out.

Recovery:

Change full interval.

Image Quality: image details washed out / contrast problem

Symptom:

Image details are washed out or there is a bad contrast

Problem:

Object is either too faint or bright

Recovery:

Increase/decrease exposure time or change ND filter

NIRSPEC Specific Problems

The recovery procedure should be used for the following known issues:

Symptom:	Problem:
Can't select nirspecm	Camera nirspecm was not available or failed to start up when MAGIQ server started
Not getting images GUI slow or frozen	Camera server not running or frozen
Images look like scrambled puzzles	Camera read out and exposure control got out of sync
Noisy images	Focus, filter wheel, or IQL motor may remain turned on

Recovery:

Restart MAGIQ server from DCSGUI tear-off menu
Wait until MAGIQ UI reconnects (30s) or restart MAGIQ GUI from tear-off menu
Reselect nirspecm
Check settings: camera focus, filters, exposure time, binning
If the problem persists then restart camera server nirspecm from DCSGUI MAGIQ tear-off menu
If the problem persists, power cycle camera electronics and Galil controller from DCSGUI MAGIQ tear-off menu
Check settings: camera focus, filters, exposure time, binning. Click "Init" and then "Load Defaults" if mechanism states say "unknown" in status display.
If the problem persists, call support

NIRSPEC: No SCAM images

Not receiving SCAM images in the MAGIQ gui.

Problem:

SCAM may not be running or there was a startup problem.

Recovery:

Restart SCAM legacy camera server from DCSGUI MAGIQ tear-off menu, and retry camera acquisitions.

LRIS

LRIS (same as LRISADC) has two MAGIQ cameras: lrisSlit and lrisOffset.

The lrisSlit camera has 2 mechanisms:

Focusing mechanism
ND filter wheel

The lrisOffset camera has 4 mechanisms:

Focusing mechanism
ND filter wheel
Photometric filter wheel
IQL (image quality lens)

LRIS guider image acquisition failed with the Slitmask Alignment Tool

Symptom:

If Guider images are not acquired and displayed in external guis such as the LRIS Slitmask Alignment Tool, there are two possible reasons:

Problem:

MAGIQ Keyword server is not functioning.
MAGIQ server is not connecting to the keyword server

Recovery:

Restart the MAGIQ keyword server and MAGIQ server in that order:

log into k1-magiq-server
run magiqAdmin start MagiqKwd
from MAGIQ pulldown menu select Resart Server

Other non MAGIQ cameras

HIRES GPIB camera. See here.
DEIMOS PXL camera. See here.
ESI PXL camera. See here.
AO PXL camera. See here.
SSC GPIB camera. See here

HOW-TO'S

To start/stop MAGIQ server

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu.

Start or stop MAGIQ server via DCS GUI MAGIQ tear-off menu. See above.
Note:When stopping and starting the MAGIQ server, it is recommended to verify that the processes are actually stopped and restarted by checking MAGIQ status. See below.

To start/stop MAGIQ camera servers

When MAGIQ server starts, it checks if the needed camera servers are running and starts them if necessary. No extra manual steps are necessary. When MAGIQ server stops, it does not stop the camera servers. To recover from an error or to trouble shoot problems, start or stop MAGIQ camera and mech servers via DCS GUI MAGIQ tear-off menu. See below. Example for NIRSPEC:

Only one set of camera servers can be running at the same time for one camera. To ensure that the start-up script first forces any running camera servers to quit before starting a new one. This is why the menu to start the camera server is labeled with "Restart". The menu item "Power cycle" is available for those cases when a simple restart does not restore the camera functionality and a full power cycle is required. A power cycle always includes:

terminating any running camera and mech servers,
powering off the camera electronics and the Galil controller,
then powering on the Galil controller and the camera electronics and
then restarting the camera and mech servers. In some cases, a manual restart of the MAGIQ server and MAGIQ OA GUI is necessary after a power cycle of the camera electronics and Galil controller.

To check MAGIQ Status

There are two ways to check MAGIQ status:

Via DCSGUI MAGIQ tear-off menu button "Check MAGIQ status"
From a xterm on k1-magiq-server or k2-magiqserver as k1obs or k2obs:
- magiq status
- The example above shows two GUIs: MAGIQ OA GUI and Observer GUI. MAGIQ server is running. NIRSPEC camera and mech servers are running on k2-magiq-camserver1. And MAGIQ keyword server is running. Note that there are two cam_rpc_svc processes for each camera server.

MAGIQ status web page
Another way to check the status is through the MAGIQ status web page:
http://www.keck.hawaii.edu/observing/magiqStatus/magiqStatus.php

MAGIQ status is also displayed on the weather page:
http://www.keck.hawaii.edu/observing/weather/weather2/index.php

Power control

The DCSGUI MAGIQ tear-off menu provides the buttons to power cycle the NIRSPEC MAGIQ camera electronics and Galil controller. See magiq script section for more details. NIRSPEC SCAM SCAM is available via the legacy camera server. To use SCAM the NIRSPEC keyword scamguide must be set to 1. This is usually done via the NIRSPEC eavesdrop GUI. The DCSGUI MAGIQ tear-off menu has an option to start the SCAM legacy camera server. See above.

MAGIQ SCRIPT

Note: The commands and scripts description in this section and in the next section are intended to be invoked via GUIs and for trouble shooting purposes. In normal operation, there is not need to use these scripts directly. The magiq script provides the means to manage MAGIQ processes and it is available on k1-magiq-server or k2-magiq-server. The general syntax is:

magiq command Subsystem [parameters]
command:

start
stop
restart
poweron
poweroff
powercycle
status (see 2.4.1) Subsystem:
MagiqServer
MagiqGui
ObserverUI
IqmUI
CameraDiagnostic
MagiqCacheGenerator
MagiqKwd
Audio
CameraServer
MechServer

MagiqServer

There can be only one MagiqServer running.

magiq start MagiqServer
magiq restart MagiqServer
Starts the MagiqServer. The script always kills any running MAGIQ server before starting a new one. Thus start is same as restart.

magiq stop MagiqServer
Stops the MagiqServer.

MagiqGui

Multiple MagiqGui can run simultaneously.

magiq start MagiqGui
Starts a new MAGIQ OA GUI using the current DISPLAY environment variable.

magiq stop MagiqGui
Stops ALL running MAGIQ OA GUIs.

magiq restart MagiqGui
Stops ALL running MAGIQ OA GUIs and start ONE MAGIQ OA GUI.

ObserverUI

Multiple OvserverUI can run simultaneously.

magiq start ObserverUI
Starts a new MAGIQ Observer GUI using the current DISPLAY environment variable.

magiq stop ObserverUI
Stops ALL running MAGIQ Observer GUIs.

magiq restart ObserverUI
Stops ALL running MAGIQ Observer GUIs and start ONE MAGIQ Observer GUI.

IqmUI

Multiple IqmUI can run simultaneously.

magiq start IqmUI
Starts a new MAGIQ IQM GUI using the current DISPLAY environment variable.

magiq stop IqmUI
Stops ALL running MAGIQ IQM GUIs.

magiq restart IqmUI
Stops ALL running MAGIQ IQM GUIs and start ONE MAGIQ IQM GUI.

CameraDiagnostic

Multiple CameraDiagnostic can run simultaneously.

magiq start CameraDiagnostic
Starts a new MAGIQ CameraDiagnostic GUI using the current DISPLAY environment variable.

magiq stop CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs.

magiq restart CameraDiagnostic
Stops ALL running MAGIQ CameraDiagnostic GUIs and start ONE MAGIQ CameraDiagnostic GUI.

MagiqCacheGenerator

The cache generator caches catalog and image archive look-up results on local disk to speed up subsequent look-ups. Multiple cache generators can be running simultaneously. The cache directory is:

/s/nightly1/log/magiq/cache for Keck 1
/s/nightly2/log/magiq/cache for Keck 2
magiq start MagiqCacheGenerator star-list catalog-name dss-name
Starts the MAGIQ cache generator. Star-list is the full pathname of the star list in Keck format. Catalog-name is one of:
GSC23 (online)
NOMAD (online)
2MASS (online)
USNOB10 (online)
VGSC2 (online) Vizier GSC2
GSC (local)
SAO (local)
HIP (local) dss-name is one of:
DSS2R (online)
DSS2I (online)
DSS2B (online)

MagiqKwd

The MAGIQ keyword agent broadcasts MAGIQ keywords to an UNIX-IOC and listens for MAGIQ keyword changes and forwards them to the MAGIQ server. There can be only one MAGIQ keyword agent running.

magiq start MagiqKwd
Starts the MAGIQ Keyword Agent.

magiq stop MagiqKwd
Stops the MAGIQ Keyword Agent.

magiq restart MagiqKwd
Stops the MAGIQ Keyword Agent and restarts a new one.

Audio

The MAGIQ audio alarm script magiqAudio is a shell script that monitors the MagiqServer.log file and issues an audio alarm by playing an .au file. There can be only one audio alarm script running.

magiq start Audio
Starts the MAGIQ audio alarm script.

magiq stop Audio
Stops the MAGIQ audio alarm script.

magiq restart Audio
Stops the MAGIQ audio alarm script and restarts a new one.

CameraServer

The MAGIQ script invokes another script called magiqCamera, which runs k1-magiq-camserver1 or k2-magiq-camserver2 dependent on the telescope. The magiqCamera script starts and stops the actual camera server and also controls the power of the camera electronics and Galil controller.

magiq start CameraServer camera_host camera_name
Starts the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers.
For example:
magiq start CameraServer k2-magiq-camserver1 nirspecm
magiq start CameraServer k1-magiq-camserver1 lrisSlit
magiq start CameraServer k1-magiq-camserver1 lrisOffset
magiq stop CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host, including the needed motion/mech servers.

magiq restart CameraServer camera_host camera_name
Stops the MAGIQ camera server for the camera camera_name on host camera_host and restarts a new one. This includes the needed motion/mech servers.

magiq poweroff CameraServer camera_host camera_name
Turns off power to camera_name on camera_host. This affects camera electronics only.
For example:
magiq poweroff CameraServer k1-magiq-camserver1 lrisSlit
magiq poweroff CameraServer k1-magiq-camserver1 lrisOffset
magiq poweroff CameraServer k2-magiq-camserver1 nirspecm
magiq poweron CameraServer camera_host camera_name
Turns on power to camera_name on camera_host. This affects camera electronics only.

magiq powercycle CameraServer camera_host camera_name
Turns off power to camera_name on camera_host and turns power back on. This affects camera electronics only.

Galil Power

magiq poweroff MechServer camera_host camera_name
Stops camera/mech server(s) and turns off power to Galil controller. For example:

magiq poweroff MechServer k2-magiq-camserver1 nirspecm
Stops the camera server, the motion control server and the mechanism server and then turns off power to the Galil controller.

magiq poweron MechServer camera_host camera_name

Stops the camera/mech server(s) and turns on power to Galil controller and camera electronics.
Initializes the EDT board via initcam from EDT.
Then restarts the camera/mech servers(s).
Then initializes mechanisms.
magiq powercycle MechServer camera_host camera_name
Stops the camera/mech server(s) and turns on power to Galil controller and camera electronics.
Initializes the EDT board via initcam from EDT.
Then restarts the camera/mech servers(s).
Then initializes mechanisms. Note: powercycle calls poweroff and then poweron.
magiq powercycle_restart MechServer camera_host camera_name
Same as powercycle, but it stops the MAGIQ server first and then restarts the MAGIQ server after power cyling the mech server.

MAGIQ CAMERA SCRIPT

The MAGIQ camera script, magiqCamera, is available on k1-magiq-camserver1 and k2-magiq-camserver1. It is invoked by the MAGIQ script. It can be invoked manually also. This script sets up the environment variables and calls other instrument dependent scripts. The syntax is:

magiqCamera command camera-name
command is one of:

start
stop
restart
status camera-name is one of:
lrisSlit
lrisOffset
nirspecm

Examples:

magiqCamera start lrisSlit
magiqCamera start lrisOffset
magiqCamera start nirspecm The script checks if the camera/mech servers are already running, if not then it starts camera server and mech/motion control servers and initializes mechanisms.

magiqCamera stop lrisSlit
magiqCamera stop lrisOffset
magiqCamera stop nirspecm Stops the camera/mech/motion servers and delete the corresponding entries from the RPC info table.

magiqCamera restart lrisSlit
magiqCamera restart lrisOffset
magiqCamera restart nirspecm Stops the camera/mech/motion servers and restarts them, and initializes the mechanisms.
magiqCamera status
Shows all running camera/mech/motion server processes.

MAGIQ CAMERA CONFIGURATION FILES

Camera Server Configuration File

The configuration file for the camera server is located at:

$KROOT/magiq/default/cam_config.xml

on the camera server host. This configuration is a XML file containing information of all MAGIQ cameras (for one telescope) including:

identification (name)
offsets (biases)
temperature set point
unit number
channel number
RPC program number

Mech Server Configuration Files

The mech/motor servers require the following configuration files located at:

```
$KROOT/rel/default/data
```
on the camera server host:
- mgqmtr_keyword_config_file
- mgqmtr_keyword_monitor_list
- mgqmtr_motor_config_file Configuration information includes:
  - type of motor
  - output channel
  - encoder resolution
  - name and position of filters in encoder counts
  - list of keywords

MAGIQ Server Configuration Files

MAGIQ server configuration files are located on the MAGIQ server host at:

$KROOT/magiq/default

The directory configData contains XML files describing instruments and their guider cameras.
The file Telescopes.xml contains the list of telescopes. In the case of Keck 1 or Keck 2, there is only one telescope in the list. This is to avoid confusion and redundant copies of configuration files. The file KECK1.xml or KECK2.xml contains the list of instruments that are available for the given telescope. The configuration files for the instruments are located in the directory KECK1 and KECK2 respectively.
An instrument configuration includes an XML file and a directory of the same name. The XML file lists the guider cameras and subsystems. The information on subsystems are included for completeness and not used at this time. The instrument directory contains an XML file describing pointing origins. This description is only used for off-sky operations (observing planning or testing). When on-sky the pointing origin information is given via keywords that supersede the static information from the XML file.
The directory Cameras contains configuration for guider cameras available for the given telescope. The list of these cameras is given in the file cameras.xml. Each camera has its own XML configuration file, which includes REF x/y and default pointing origin name, plate scale and other information.
The file magiqConfig.xml.k2server includes the following information:
- Telescope name
- Update rates for various event loops
- Star catalog cache directory
- Fits header description file
- Directory containing configData
- Logging level
- Simulation modes
- Centroiding algorithms
- Flexure compensation parameter file location
- Keyword services
To add a new instrument:
- Choose a name for the new instrument, for example INSTR123.
- Create XML file INSTR123.xml containing list of guider cameras.
- Create directory INSTR123 containing the PointingOrigins.xml file.
- Add INSTR123 in KECK1.xml or KECK2.xml
To add a new guider camera:
- Choose a new for the new camera, for example CAM123.
- Create a XML configuration file CAM123.xml including reference pixel location, and the default pointing origin, plate scale, tv angle and flip.
- Add CAM123 in Cameras/cameras.xml.