General Info

Computers

OSIRIS has several computers necessary for operations

K1 Computer Room

K1 OSIRIS/MOSFIRE Rack

Server Background

OSIRIS has 15 keyword servers, one for each detector, one for each motor, 3 for temperatures, 1 for pressure, 2 for power, and 1 global server that communicates with the other 14 servers. Many have common features:

Starting

osiris start SERVER

Stopping

osiris stop SERVER

Show available keywords

show -s SERVER keywords

lock

Keyword servers can be locked so moves are not made accidentally. Lock=0 is unlocked, lock=1 is locked.

lockeng

Some keywords are effectively locked at all times via the lockeng keyword. To unlock these keywords:
modify -s SERVER lockeng=yymmdd (where yymmdd is the civil date). This keyword acts as a toggle, i.e. set it to yymmdd again to lock.

Keyword Servers

The following servers can replace "SERVER" in the above examples.

osiris

OSIRIS global server. All observing can be done using calls only to this server.

op1s

OSIRIS power server 1. Controls Power to the detector controllers.

Keyword	Description	Servers
pwstat[1-5]	Unused	None
pwstat6	Imager Electronics	oids
pwstat7	Spec Electronics	osds
pwstat8	EC Cooling System	None

op2s

OSIRIS power server 2. Controls power to the pressure gauge, temperature controllers, motor controllers, and terminal server.

Keyword	Description	Servers
pwstat1	Pressure Gauge	oprs
pwstat2	Lakeshore 340	otcs
pwstat3	Dewar Lakeshore 218	ot1s
pwstat4	Cabinet Lakeshore 218	ot2s
pwstat5	Motor Controllers	om1s, om2s, om3s, om4s, om5, om6s
pwstat6	Terminal Server	None
pwstat7	Unused	None
pwstat8	EC Cooling System	None

oprs

OSIRIS pressure server. Currently, we are using the backup pressure gauge. Data plots are displayed on the Dewar Pressure Page.

ot1s

OSIRIS temperature server 1. Records dewar temperatures. Data plots are displayed on the Dewar Temps Page.

ot2s

OSIRIS temperature server 2. Records electronics cabinet temperatures. Data plots are displayed on the Cabinet Temps Page.

otcs

OSIRIS temperature control server. Maintains both detectors at a specified temperature. Data plots are displayed on the Detector Temps Page.

om1s

OSIRIS mechanism server 1. Scale Turret 1.

om2s

OSIRIS mechanism server 2. Spec Filter Wheel.

om3s

OSIRIS mechanism server 3. Scale Turret 2.

om4s

OSIRIS mechanism server 4. Lenslet Mask Stage.

om5s

OSIRIS mechanism server 5. Imager Filter Wheel 1.

om6s

OSIRIS mechanism server 6. Imager Filter Wheel 2.

osds

OSIRIS SPEC detector server.

oids

OSIRIS IMAG detector server.

Acquisition

Nothing on OSPEC

Symptom

After acquiring a star, there's nothing in the online data cube

Problem

Several possibilities:

1. Pointing origin is off
2. Filter change
3. DAR issues

Solution

Most of these issues are fairly minor and can be fixed by some additional steps during acquisition. If you're well and truly lost, try offseting to the OSIRIS imager to find your target as described in the Acquistion Cookbook.

1. To compensate for an incorrect pointing origin:
   1. Note the pixel of the star in the ORP cube from the AO checkout.
   2. Use the OTGUI to offset from that pixel in Hn3 0.020 scale to the center of your filter/scale combination.
   3. Make this offset before taking data on subsequent acquisitions.
2. The dispersing elements in OSIRIS are fixed so different filters illuminate different lenslets. If you change filters you must:
   1. Use the OTGUI to send the object from the center of filter/scale #1 to the center of filter/scale #2.
3. DAR = Differential Atmospheric Refraction. The OSPEC FOV is small enough that DAR is a factor. If you've done the initial offsets as described above, but still don't see light, please check:
   1. Does the AO system have the correct B-V for your object?
   2. Was DAR on from the LgsaoAcq Widget?

Scripts

Script Timeout (scriptreply)

Symptom

OSIRIS stops taking data then observer sees this statement in the OSIRIS Log GUI:

wfao: Timed out waiting for TT loop.  If TT lock is not required, please turn wait4tt OFF for future exposures.  Do you want to continue now?
wfao: Set scriptreply keyword to Y or N to respond.
wfao: Waiting for response.

Problem

The AO loops had not closed before the OSIRIS script timed out.

Solution

If the loops are actually closed...

1. From the xterm on the osirisserver (napili) type "m scriptreply=y" or "modify -s osiris scriptreply=y"

Darks will go too long

Symptom

User started osirisTakeDarks script and now wants to stop it.

Problem

The series of darks may run into observing time.

Solution

Stop the script via "CTRL-C" (DO NOT ABORT THE EXPOSURE IN PROGRESS!)

1. From the xterm in which the user started the osirisTakeDarks script, issue "CTRL-C"
2. Answer with an "n" to "Do you wish to abort the current exposure?"
3. Wait for the exposure in progress to finish.

GUIS

OSGUI Stops Updating

Symptom

OSGUI does not update upon taking exposure or changing filters, etc.

Problem

The OSGUI has crashed. Often as a result of aborting an observing sequence.

Solution

Restart the OSGUI

1. From an xterm on the osirisserver, type "osiris stop osgui"; or select "File" --> "Exit" from the OSGUI
2. type "osiris start osgui &"

Cannot send DDF to Queue

Symptom

Nothing happens when observer clicks "Send to Queue" on OOPGUI

Problem

Unsure.

Solution #1

Restart the OOPGUI and ODEC

1. Kill both the OOPGUI and ODEC either via "File" --> "Exit" or from an xterm on the osirisserver typing "osiris stop oopgui" and "osiris stop odec"
2. type "osiris start oopgui &"
3. type "osiris start odec &"

Solution #2

If solution #1 fails, manually load DDFs into ODEC

1. Setup observing sequence in OOPGUI
2. Save the sequence from OOPGUI via "File" --> "Save DDF..."
3. Load the sequence into ODEC via "File" --> "Add DDF to Queue..."
4. Observe

Detector Servers

Recover from Aborted Exposure

Symptom

After issuing an "Abort All Immediately" command from the ODEC, one or both of the detectors does not take images.

Problem

Detector needs a re-initialization or the detector server needs a restart.

Solution

The first two steps can be performed by the observer. The restart of a detector server requires the SA (for now).

1. From the "OSIRIS" tab of the OTGUI, Flush the unresponsive detector.
2. Once the flush is complete, issue one of the following commands from a napili prompt:
   1. "modify -s osiris sinit=1" (for the SPEC detector)
   2. "modify -s osiris iinit=1" (for the IMAG detector)
3. If this does not solve the problem, please contact your support astronomer to restart a failed detector server.

Restart a Failed Detector Server

Symptom

One cannot take images with one of the two detectors.

Problem

The detector server has become unresponsive, possibly due to an "abort exposure" command.

Solution

Restart the detector server and reset keywords via the osirisRecover script.

1. Issue the command "osirisRecover [IMAG|SPEC]" from a napili prompt.
2. The script should make everything ok. Please note that this script CAN be run while taking data with the other detector.

Exposure will not start: detector server hung I

Symptom

Can't initiate an exposure. Spec detector appear to be hung. Cannot take data.

Problem

Can't connect to the side car server.

Solution I

Try first reconnecting to the side car server if this fails, move on to Solution 2 below.

1. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
   This will ...
   • Set osds "connect=1" if "connected" != 1
   • Will attempt osds resume=1 if connected
   • Success is osds ready=1
   • If "resume" doesn't work, try "init=1"

Exposure will not start: detector server hung II

Symptom

Can't initiate an exposure. Spec detector appear to be hung. Cannot take data.

Problem

Can't connect to the side car server.

Solution II

Restart the side car server processes by killing the side car

1. osiris stop osds => stop the osiris spec detector server
2. vncviewer osiris-control1 => to launch the vnc session for the side car server. You will need login info ...
3. Kill the dos window titled "ColdSidecarHxRG"
4. doubel-click the icon "ColdSidecarHxRG" to restart the side car server.
5. Be patient. wait roughly 30s for things to reconnect.
6. osiris start osds
7. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
   This will ...
   • Set osds "connect=1" if "connected" != 1
   • Will attempt osds resume=1 if connected
   • Success is osds ready=1
   • If "resume" doesn't work, try "init=1"

Exposure will not start: scriptrun error

Symptom

Can't initiate an exposure. "Start next dataset" in ODEC is greyed out, cannot take data via the GUI or command line,
but osirisCheckoutDetector would show detector is in a healthy state.

Problem

The scriptrun keyword is set to a non-zero value.

Solution

Execute the following command in a napili terminal

show -s osiris scriptrun

If scriptrun=1, do

modify -s osiris scriptrun=0

May be a good idea to restart ODEC and Status GUI.

Continuous Sidecar Server Disconnects

Symptom

The osds keyword "connected" equals 0 after repeated connection attempts and sidecar server restarts. May still be able to take data

Problem

Clock time on osiris-control1 (PC) is incorrect

Solution

Re-sync the time on osiris-control1

1. Re-sync the time on control1:
   1. Open an xterm on osirisserver under any OSIRIS account.
   2. Execute the command

      	vncviewer osiris-control1

      to connect with the VNC session on the augmentix computer. The password can be found in the SA password list.
   3. Click on the time in the lower right corner to bring up the Windows time dialog
   4. Click on the "Internet Time" tab
   5. Click on "Change settings"
   6. Click "Update Now" and wait about 5 seconds
   7. Look for a "Success" message. If the sync fails, repeat the click "Update Now" step until it succeeds
2. Restart the Sidecar Server on osiris-control1:
   1. Kill the dos window titled "ColdSidecarHxRG"
   2. doubel-click the icon "ColdSidecarHxRG" to restart the side car server.
   3. Be patient. wait roughly 30s for things to reconnect.
3. Restart the osds server
   1. osiris stop osds => stop the osiris spec detector server
   2. osiris start osds
   3. run osirisCheckoutDetector. If not using the eng account run it at: /home/osrseng/scripts/osirisCheckoutDetector
      This will ...
      • Set osds "connect=1" if "connected" != 1
      • Will attempt osds resume=1 if connected
      • Success is osds ready=1
      • If "resume" doesn't work, try "init=1"

Global Server

Global Server Disconnect

Symptom

You see a popup on each Java GUI with a "KTL_OPEN" error.

Problem

Many possible causes, but essentially, the global server has lost communication with one or more sub-servers.

Solution

Restart the failed servers

1. If it's a detector server, first recover the failed detector server. Otherwise, restart the server via "osiris stop [server]", then "osiris start [server]"
2. The global server should automatically reconnect. If it does not, restart the global server via "osiris stop osiris", then "osiris start osiris".
3. If the above steps fail, you may try issuing an "osirisDisconnect", then "osirisConnect" to stop and restart all the servers.

Servers (General)

Cannot start a server

Symptom

When starting osiris keyword servers, you see many RPC errors.

Problem

The RPC info table may not have been cleared.

Solution

Clear the offending entry in the rpcinfo table

1. Login to osiris on napili: "ssh osiris@napili"
2. View the RPC table: "rpcinfo"
   

    798049830    1    tcp       0.0.0.0.128.126     server_op1s 726
    798050830    1    tcp       0.0.0.0.128.137     server_op2s 726
    798082830    1    tcp       0.0.0.0.128.149     server_oprs 726
    798449830    1    tcp       0.0.0.0.128.159     server_ot1s 726
    798450830    1    tcp       0.0.0.0.128.169     server_ot2s 726
    798467830    1    tcp       0.0.0.0.128.179     server_otcs 726
    797749830    1    tcp       0.0.0.0.128.189     server_om1s 726
    797750830    1    tcp       0.0.0.0.128.199     server_om2s 726
    797751830    1    tcp       0.0.0.0.128.209     server_om3s 726
    797752830    1    tcp       0.0.0.0.128.219     server_om4s 726
    797753830    1    tcp       0.0.0.0.128.230     server_om5s 726
    797754830    1    tcp       0.0.0.0.128.241     server_om6s 726
    790000000    1    tcp       0.0.0.0.129.235     server_osiris 726
                   

3. Remove the offending server entry: (e.g. "rpcinfo -d 797750830 1" to remove om2s)

Motor Mechanisms

Motor Error

Symptom

Result of testAll gives error on one or more of om?s servers

Problem

Mechanism is lost.

Solution

Reinit the service, the home the motor

On osirisserver:
1. testAll to determine the offending service (e.g. om5s)
2. modify -s om5s init=1
3. if you see an error immediately, see if it mentions the lockeng keyword. If so, unlock lockeng by modify -s om5s lockeng=yymmdd (where yymmdd is the civil date)
4. modify -s om5s home=1
5. testAll to verify functionality

Filter Wheel Fault

Symptom

Red X appears over spec filter in status gui.

Problem

Failed to achieve switch position

Solution

Trick the mechanism into thinking it is in position.

1. show -s om2s switch
2. show -s om2s pos
3. show -s om2s switch17 (if, for example, result from pos was 17)
4. modify -s om2s lockeng= 060817 (civil date)
5. modify -s om2s switch17= 128 (if, for example, result from switch was 128 which differed from switch17 read in step 3)
6. Note: If the problem with the switch mechanism resolves itself (as often happens) follow the above steps to restore the expected grey code back to the default value. In this case you do not need to reset the lockeng keyword.

testAll Output

testAll Sample Output—Good

Symptom

Everything is A-OK with OSIRIS

Problem

None. testAll output:

[465] osrseng@napili: testAll
============================================================

 6 of  6 motor servers running on napili
 3 of  3 power/pressure servers running on napili
 3 of  3 temperature servers running on napili
 1 of  1 global servers running on napili
 1 of  1 image servers running on kuiaha
 1 of  1 spec servers running on puunoa
10 of 10 tnet processes running on napili

The following GUIs are currently running:
username    pid    service
--------    ---    -------
osiris6     20209    Gui-ODEC
osiris6     20157    Gui-OOPGUI
osiris6     20371    Gui-OORGUI
osiris6     20261    Gui-OSGUI
osiris6     20313    Gui-OTGUI

To view the running processes, execute the command: ct

============================================================
All the required servers and tnet processes are running.
============================================================
Service  Description               Value              Status        
====================================================================
op1s     ElectronicsCab_1          PowerOn            OK           
op2s     ElectronicsCab_2          PowerOn            OK           
osds     SPEC_detector             Idling             OK           
oids     IMAG_detector             Idling             OK           
oprs     Dewar_Pressure            0.000262           OK           
otcs     SPEC_Temp                 68.00              OK           
om1s     Scale_Turret_1            0.100              OK           
om2s     Spec_Filter_Wheel         Drk                OK           
om3s     Scale_Turret_2            0.100              OK           
om4s     Lenslet_Mask_Stage        Broad              OK           
om5s     Imager_Filter_Wheel_1     Hn1                OK           
om6s     Imager_Filter_Wheel_2     Kn1                OK           
====================================================================
0 warnings generated
0 errors generated
====================================================================
        

Solution

No action needed

testAll Sample Output—EC1 Power Issue

Symptom

Something is wrong with OSIRIS

Problem

Various. testAll output:

[195] osrseng@napili: testAll
============================================================

 6 of  6 motor servers running on napili
 3 of  3 power/pressure servers running on napili
 3 of  3 temperature servers running on napili
 1 of  1 global servers running on napili
 1 of  1 image servers running on kuiaha
 1 of  1 spec servers running on puunoa
10 of 10 tnet processes running on napili

The following GUIs are currently running:
username    pid    service
--------    ---    -------
osiris6     20209    Gui-ODEC
osiris6     20157    Gui-OOPGUI
osiris6     20371    Gui-OORGUI
osiris6     20261    Gui-OSGUI
osiris6     20313    Gui-OTGUI

To view the running processes, execute the command: ct

============================================================
All the required servers and tnet processes are running.
============================================================
Service  Description               Value              Status        
====================================================================
op1s     ElectronicsCab_1          PowerOff           Error        
op2s     ElectronicsCab_2          PowerOn            OK           
osds     SPEC_detector             Error              Error        
oids     IMAG_detector             Error              Error        
oprs     Dewar_Pressure            1.000000           Warning      
otcs     SPEC_Temp                 67.88              Tspec_off_target
om1s     Scale_Turret_1            0.100              OK           
om2s     Spec_Filter_Wheel         Hn3                OK           
om3s     Scale_Turret_2            0.100              OK           
om4s     Lenslet_Mask_Stage        Broad              OK           
om5s     Imager_Filter_Wheel_1     Hn1                OK           
om6s     Imager_Filter_Wheel_2     Kn1                OK           
====================================================================
2 warnings generated by the following servers:
oprs otcs
Remedies:  Generally no action required for warnings, but situation
           should be monitored via testAll and noted in the Daylog.
           As always, if in doubt, contact the SA.

3 errors generated by the following servers:
op1s osds oids
Remedies:  If the only servers in error start with om:
           Please contact the SA.

Remedies:  If you see errors from any of op1s, op2s, osds, or oids:
           Please try to disconnect/reconnect via:
           1) osirisDisconnect
           2) ctx (to verify all servers are down)
           3) osirisConnect
           4) testAll

           If you have already tried this, contact the SA.
====================================================================
        

Solution

This is shortly after a reconfig. The pressure and temperature warnings are normal for shortly after a reconfig. The electronics cab and detector server errors are because the op1s server, which controls the Pulizzi for Electonics Cabinet 1 did not turn on properly.

To fix:
1. osirisDisconnect
2. ct (to verify servers are down)
3. osiris start op1s
4. wait 10 seconds
5. modify -s op1s init=1
6. wait 10 seconds
7. show -s op1s pwstat6 pwstat7 pwstat8
8. 1=on, 0=off...all should be on
   if not:
   1. modify -s op1s lockeng=yymmdd (civil date)
   2. modify -s op1s pwstat6=1
   3. modify -s op1s pwstat7=1
   4. modify -s op1s pwstat8=1
   5. show -s op1s pwstat6 pwstat7 pwstat8
9. if all good, osirisConnect