Most Common

Script Timeout

OSGUI Stops Updating

Darks will go too long

Nothing on OSPEC

Filter Wheel Fault

General Info

Computers

OSIRIS has several computers necessary for operations

K1 Computer Room

K1 Rack H

NameAliasDescription
napiliosirisserver Main host. Runs all servers except detector servers, all GUIs related to observing, and all scripts.
osiris-drposirisdrp Data host. Runs DRP, QuickLook2, any IDL, and serves the data.
puunoaosirisspec SPEC detector host. Runs SPEC detector server.
kuiahaosirisimag IMAG detector host. Runs IMAG detector server.
osrstermosrsterm OSIRIS terminal server.
osrssw1osrssw1 OSIRIS network switch #1
osrssw2osrssw2 OSIRIS network switch #2
pauoaosirisspare Spare computer that quickly can be configured to replace one of osirisserver, osirisspec, or osirisimag.
osiris-odrposirisdrpspare Spare computer that can replace osirisdrp.

Server Background

OSIRIS has 15 keyword servers, one for each detector, one for each motor, 3 for temperatures, 1 for pressure, 2 for power, and 1 global server that communicates with the other 14 servers. Many have common features:

Starting
osiris start SERVER
Stopping
osiris stop SERVER
Show available keywords
show -s SERVER keywords
lock
Keyword servers can be locked so moves are not made accidentally. Lock=0 is unlocked, lock=1 is locked.
lockeng
Some keywords are effectively locked at all times via the lockeng keyword. To unlock these keywords:
modify -s SERVER lockeng=yymmdd (where yymmdd is the civil date). This keyword acts as a toggle, i.e. set it to yymmdd again to lock.

Keyword Servers

The following servers can replace "SERVER" in the above examples.

osiris
OSIRIS global server. All observing can be done using calls only to this server.
op1s
OSIRIS power server 1. Controls Power to the detector controllers.
KeywordDescriptionServers
pwstat[1-5]UnusedNone
pwstat6Imager Electronicsoids
pwstat7Spec Electronicsosds
pwstat8EC Cooling SystemNone
op2s
OSIRIS power server 2. Controls power to the pressure gauge, temperature controllers, motor controllers, and terminal server.
KeywordDescriptionServers
pwstat1Pressure Gaugeoprs
pwstat2Lakeshore 340otcs
pwstat3Dewar Lakeshore 218ot1s
pwstat4Cabinet Lakeshore 218ot2s
pwstat5Motor Controllersom1s, om2s, om3s, om4s, om5, om6s
pwstat6Terminal ServerNone
pwstat7UnusedNone
pwstat8EC Cooling SystemNone
oprs
OSIRIS pressure server. Currently, we are using the backup pressure gauge. Data plots are displayed on the Dewar Pressure Page.
ot1s
OSIRIS temperature server 1. Records dewar temperatures. Data plots are displayed on the Dewar Temps Page.
ot2s
OSIRIS temperature server 2. Records electronics cabinet temperatures. Data plots are displayed on the Cabinet Temps Page.
otcs
OSIRIS temperature control server. Maintains both detectors at a specified temperature. Data plots are displayed on the Detector Temps Page.
om1s
OSIRIS mechanism server 1. Scale Turret 1.
om2s
OSIRIS mechanism server 2. Spec Filter Wheel.
om3s
OSIRIS mechanism server 3. Scale Turret 2.
om4s
OSIRIS mechanism server 4. Lenslet Mask Stage.
om5s
OSIRIS mechanism server 5. Imager Filter Wheel 1.
om6s
OSIRIS mechanism server 6. Imager Filter Wheel 2.
osds
OSIRIS SPEC detector server.
oids
OSIRIS IMAG detector server.

Acquisition

Nothing on OSPEC

Symptom
After acquiring a star, there's nothing in the online data cube
Problem
Several possibilities:
  1. Pointing origin is off
  2. Filter change
  3. DAR issues
Solution
Most of these issues are fairly minor and can be fixed by some additional steps during acquisition. If you're well and truly lost, try offseting to the OSIRIS imager to find your target as described in the Acquistion Cookbook.
  1. To compensate for an incorrect pointing origin:
    1. Note the pixel of the star in the ORP cube from the AO checkout.
    2. Use the OTGUI to offset from that pixel in Hn3 0.020 scale to the center of your filter/scale combination.
    3. Make this offset before taking data on subsequent acquisitions.
  2. The dispersing elements in OSIRIS are fixed so different filters illuminate different lenslets. If you change filters you must:
    1. Use the OTGUI to send the object from the center of filter/scale #1 to the center of filter/scale #2.
  3. DAR = Differential Atmospheric Refraction. The OSPEC FOV is small enough that DAR is a factor. If you've done the initial offsets as described above, but still don't see light, please check:
    1. Does the AO system have the correct B-V for your object?
    2. Was DAR on from the LgsaoAcq Widget?

Scripts

Script Timeout (scriptreply)

Symptom
OSIRIS stops taking data then observer sees this statement in the OSIRIS Log GUI:
wfao: Timed out waiting for TT loop.  If TT lock is not required, please turn wait4tt OFF for future exposures.  Do you want to continue now?
wfao: Set scriptreply keyword to Y or N to respond.
wfao: Waiting for response.
Problem
The AO loops had not closed before the OSIRIS script timed out.
Solution
If the loops are actually closed...
  1. From the xterm on the osirisserver (napili) type "m scriptreply=y" or "modify -s osiris scriptreply=y"

Darks will go too long

Symptom
User started osirisTakeDarks script and now wants to stop it.
Problem
The series of darks may run into observing time.
Solution
Stop the script via "CTRL-C" (DO NOT ABORT THE EXPOSURE IN PROGRESS!)
  1. From the xterm in which the user started the osirisTakeDarks script, issue "CTRL-C"
  2. Answer with an "n" to "Do you wish to abort the current exposure?"
  3. Wait for the exposure in progress to finish.

GUIS

OSGUI Stops Updating

Symptom
OSGUI does not update upon taking exposure or changing filters, etc.
Problem
The OSGUI has crashed. Often as a result of aborting an observing sequence.
Solution
Restart the OSGUI
  1. From an xterm on the osirisserver, type "osiris stop osgui"; or select "File" --> "Exit" from the OSGUI
  2. type "osiris start osgui &"

Cannot send DDF to Queue

Symptom
Nothing happens when observer clicks "Send to Queue" on OOPGUI
Problem
Unsure.
Solution #1
Restart the OOPGUI and ODEC
  1. Kill both the OOPGUI and ODEC either via "File" --> "Exit" or from an xterm on the osirisserver typing "osiris stop oopgui" and "osiris stop odec"
  2. type "osiris start oopgui &"
  3. type "osiris start odec &"
Solution #2
If solution #1 fails, manually load DDFs into ODEC
  1. Setup observing sequence in OOPGUI
  2. Save the sequence from OOPGUI via "File" --> "Save DDF..."
  3. Load the sequence into ODEC via "File" --> "Add DDF to Queue..."
  4. Observe

Detector Servers

Recover from Aborted Exposure

Symptom
After issuing an "Abort All Immediately" command from the ODEC, one or both of the detectors does not take images.
Problem
Detector needs a re-initialization or the detector server needs a restart.
Solution
The first two steps can be performed by the observer. The restart of a detector server requires the SA (for now).
  1. From the "OSIRIS" tab of the OTGUI, Flush the unresponsive detector.
  2. Once the flush is complete, issue one of the following commands from a napili prompt:
    1. "modify -s osiris sinit=1" (for the SPEC detector)
    2. "modify -s osiris iinit=1" (for the IMAG detector)
  3. If this does not solve the problem, please contact your support astronomer to restart a failed detector server.

Restart a Failed Detector Server

Symptom
One cannot take images with one of the two detectors.
Problem
The detector server has become unresponsive, possibly due to an "abort exposure" command.
Solution
Restart the detector server and reset keywords via the osirisRecover script.
  1. Issue the command "osirisRecover [IMAG|SPEC]" from a napili prompt.
  2. The script should make everything ok. Please note that this script CAN be run while taking data with the other detector.

Exposure will not start: detector server hung I

Symptom
Can't initiate an exposure. Spec detector appear to be hung. Cannot take data.
Problem
Can't connect to the side car server.
Solution I
Try first reconnecting to the side car server if this fails, move on to Solution 2 below.
  1. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
      This will ...
    • Set osds "connect=1" if "connected" != 1
    • Will attempt osds resume=1 if connected
    • Success is osds ready=1
    • If "resume" doesn't work, try "init=1"

Exposure will not start: detector server hung II

Symptom
Can't initiate an exposure. Spec detector appear to be hung. Cannot take data.
Problem
Can't connect to the side car server.
Solution II
Restart the side car server processes by killing the side car
  1. osiris stop osds => stop the osiris spec detector server
  2. vncviewer osiris-control1 => to launch the vnc session for the side car server. You will need login info ...
  3. Kill the dos window titled "ColdSidecarHxRG"
  4. doubel-click the icon "ColdSidecarHxRG" to restart the side car server.
  5. Be patient. wait roughly 30s for things to reconnect.
  6. osiris start osds
  7. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
      This will ...
    • Set osds "connect=1" if "connected" != 1
    • Will attempt osds resume=1 if connected
    • Success is osds ready=1
    • If "resume" doesn't work, try "init=1"

Continuous Sidecar Server Disconnects

Symptom
The osds keyword "connected" equals 0 after repeated connection attempts and sidecar server restarts. May still be able to take data
Problem
Clock time on osiris-control1 (PC) is incorrect
Solution
Re-sync the time on osiris-control1
  1. Re-sync the time on control1:
    1. Open an xterm on osirisserver under any OSIRIS account.
    2. Execute the command
      	vncviewer osiris-control1
      to connect with the VNC session on the augmentix computer. The password can be found in the SA password list.
    3. Click on the time in the lower right corner to bring up the Windows time dialog
    4. Click on the "Internet Time" tab
    5. Click on "Change settings"
    6. Click "Update Now" and wait about 5 seconds
    7. Look for a "Success" message. If the sync fails, repeat the click "Update Now" step until it succeeds
  2. Restart the Sidecar Server on osiris-control1:
    1. Kill the dos window titled "ColdSidecarHxRG"
    2. doubel-click the icon "ColdSidecarHxRG" to restart the side car server.
    3. Be patient. wait roughly 30s for things to reconnect.
  3. Restart the osds server
    1. osiris stop osds => stop the osiris spec detector server
    2. osiris start osds
    3. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
        This will ...
      • Set osds "connect=1" if "connected" != 1
      • Will attempt osds resume=1 if connected
      • Success is osds ready=1
      • If "resume" doesn't work, try "init=1"

Global Server

Global Server Disconnect

Symptom
You see a popup on each Java GUI with a "KTL_OPEN" error.
Problem
Many possible causes, but essentially, the global server has lost communication with one or more sub-servers.
Solution
Restart the failed servers
  1. If it's a detector server, first recover the failed detector server. Otherwise, restart the server via "osiris stop [server]", then "osiris start [server]"
  2. The global server should automatically reconnect. If it does not, restart the global server via "osiris stop osiris", then "osiris start osiris".
  3. If the above steps fail, you may try issuing an "osirisDisconnect", then "osirisConnect" to stop and restart all the servers.

Servers (General)

Cannot start a server

Symptom
When starting osiris keyword servers, you see many RPC errors.
Problem
The RPC info table may not have been cleared.
Solution
Clear the offending entry in the rpcinfo table
  1. Login to osiris on napili: "ssh osiris@napili"
  2. View the RPC table: "rpcinfo"
     798049830    1    tcp       0.0.0.0.128.126     server_op1s 726
     798050830    1    tcp       0.0.0.0.128.137     server_op2s 726
     798082830    1    tcp       0.0.0.0.128.149     server_oprs 726
     798449830    1    tcp       0.0.0.0.128.159     server_ot1s 726
     798450830    1    tcp       0.0.0.0.128.169     server_ot2s 726
     798467830    1    tcp       0.0.0.0.128.179     server_otcs 726
     797749830    1    tcp       0.0.0.0.128.189     server_om1s 726
     797750830    1    tcp       0.0.0.0.128.199     server_om2s 726
     797751830    1    tcp       0.0.0.0.128.209     server_om3s 726
     797752830    1    tcp       0.0.0.0.128.219     server_om4s 726
     797753830    1    tcp       0.0.0.0.128.230     server_om5s 726
     797754830    1    tcp       0.0.0.0.128.241     server_om6s 726
     790000000    1    tcp       0.0.0.0.129.235     server_osiris 726
                    
  3. Remove the offending server entry: (e.g. "rpcinfo -d 797750830 1" to remove om2s)

Motor Mechanisms

Motor Error

Symptom
Result of testAll gives error on one or more of om?s servers
Problem
Mechanism is lost.
Solution
Reinit the service, the home the motor
    On osirisserver:
  1. testAll to determine the offending service (e.g. om5s)
  2. modify -s om5s init=1
  3. if you see an error immediately, see if it mentions the lockeng keyword. If so, unlock lockeng by modify -s om5s lockeng=yymmdd (where yymmdd is the civil date)
  4. modify -s om5s home=1
  5. testAll to verify functionality

Filter Wheel Fault

Symptom
Red X appears over spec filter in status gui.
Problem
Failed to achieve switch position
Solution
Trick the mechanism into thinking it is in position.
  1. show -s om2s switch
  2. show -s om2s pos
  3. show -s om2s switch17 (if, for example, result from pos was 17)
  4. modify -s om2s lockeng= 060817 (civil date)
  5. modify -s om2s switch17= 128 (if, for example, result from switch was 128 which differed from switch17 read in step 3)
  6. Note: If the problem with the switch mechanism resolves itself (as often happens) follow the above steps to restore the expected grey code back to the default value. In this case you do not need to reset the lockeng keyword.

testAll Output

testAll Sample Output—Good

Symptom
Everything is A-OK with OSIRIS
Problem
None. testAll output:
[465] osrseng@napili: testAll
============================================================

 6 of  6 motor servers running on napili
 3 of  3 power/pressure servers running on napili
 3 of  3 temperature servers running on napili
 1 of  1 global servers running on napili
 1 of  1 image servers running on kuiaha
 1 of  1 spec servers running on puunoa
10 of 10 tnet processes running on napili

The following GUIs are currently running:
username    pid    service
--------    ---    -------
osiris6     20209    Gui-ODEC
osiris6     20157    Gui-OOPGUI
osiris6     20371    Gui-OORGUI
osiris6     20261    Gui-OSGUI
osiris6     20313    Gui-OTGUI

To view the running processes, execute the command: ct

============================================================
All the required servers and tnet processes are running.
============================================================
Service  Description               Value              Status        
====================================================================
op1s     ElectronicsCab_1          PowerOn            OK           
op2s     ElectronicsCab_2          PowerOn            OK           
osds     SPEC_detector             Idling             OK           
oids     IMAG_detector             Idling             OK           
oprs     Dewar_Pressure            0.000262           OK           
otcs     SPEC_Temp                 68.00              OK           
om1s     Scale_Turret_1            0.100              OK           
om2s     Spec_Filter_Wheel         Drk                OK           
om3s     Scale_Turret_2            0.100              OK           
om4s     Lenslet_Mask_Stage        Broad              OK           
om5s     Imager_Filter_Wheel_1     Hn1                OK           
om6s     Imager_Filter_Wheel_2     Kn1                OK           
====================================================================
0 warnings generated
0 errors generated
====================================================================
        
Solution
No action needed

testAll Sample Output—EC1 Power Issue

Symptom
Something is wrong with OSIRIS
Problem
Various. testAll output:
[195] osrseng@napili: testAll
============================================================

 6 of  6 motor servers running on napili
 3 of  3 power/pressure servers running on napili
 3 of  3 temperature servers running on napili
 1 of  1 global servers running on napili
 1 of  1 image servers running on kuiaha
 1 of  1 spec servers running on puunoa
10 of 10 tnet processes running on napili

The following GUIs are currently running:
username    pid    service
--------    ---    -------
osiris6     20209    Gui-ODEC
osiris6     20157    Gui-OOPGUI
osiris6     20371    Gui-OORGUI
osiris6     20261    Gui-OSGUI
osiris6     20313    Gui-OTGUI

To view the running processes, execute the command: ct

============================================================
All the required servers and tnet processes are running.
============================================================
Service  Description               Value              Status        
====================================================================
op1s     ElectronicsCab_1          PowerOff           Error        
op2s     ElectronicsCab_2          PowerOn            OK           
osds     SPEC_detector             Error              Error        
oids     IMAG_detector             Error              Error        
oprs     Dewar_Pressure            1.000000           Warning      
otcs     SPEC_Temp                 67.88              Tspec_off_target
om1s     Scale_Turret_1            0.100              OK           
om2s     Spec_Filter_Wheel         Hn3                OK           
om3s     Scale_Turret_2            0.100              OK           
om4s     Lenslet_Mask_Stage        Broad              OK           
om5s     Imager_Filter_Wheel_1     Hn1                OK           
om6s     Imager_Filter_Wheel_2     Kn1                OK           
====================================================================
2 warnings generated by the following servers:
oprs otcs
Remedies:  Generally no action required for warnings, but situation
           should be monitored via testAll and noted in the Daylog.
           As always, if in doubt, contact the SA.

3 errors generated by the following servers:
op1s osds oids
Remedies:  If the only servers in error start with om:
           Please contact the SA.

Remedies:  If you see errors from any of op1s, op2s, osds, or oids:
           Please try to disconnect/reconnect via:
           1) osirisDisconnect
           2) ctx (to verify all servers are down)
           3) osirisConnect
           4) testAll

           If you have already tried this, contact the SA.
====================================================================
        
Solution
This is shortly after a reconfig. The pressure and temperature warnings are normal for shortly after a reconfig. The electronics cab and detector server errors are because the op1s server, which controls the Pulizzi for Electonics Cabinet 1 did not turn on properly.
    To fix:
  1. osirisDisconnect
  2. ct (to verify servers are down)
  3. osiris start op1s
  4. wait 10 seconds
  5. modify -s op1s init=1
  6. wait 10 seconds
  7. show -s op1s pwstat6 pwstat7 pwstat8
  8. 1=on, 0=off...all should be on
      if not:
    1. modify -s op1s lockeng=yymmdd (civil date)
    2. modify -s op1s pwstat6=1
    3. modify -s op1s pwstat7=1
    4. modify -s op1s pwstat8=1
    5. show -s op1s pwstat6 pwstat7 pwstat8
  9. if all good, osirisConnect