Following are recognized message strings that are printed into the LRIS CCD VME crate ('agnew') console, corresponding to various conditions. These message strings should aid in troubleshooting error conditions on this system.

VME output

  1. Normal CCD crate boot
  2. VME task list
  3. Normal exposure

Error conditions

  1. Sudden inoperation of CCD system
  2. CCD power is off
  3. CCD shutter is not working
  4. Fiber communication loss
CCD crate reboot procedures

A. Normal CCD crate boot

Note the message sequence at the end: 
Timeout waiting for ISERV, retrying
Got network configuration
Broadcast message sent on ln0
Broadcast not sent on lo0
is not an error. It simply indicates that the LRIS control software has not been started by the user. 
-------------


punaluu{lris}1: telnet tserver4 3010
Trying 128.171.136.21 ...
Connected to tserver4.
Escape character is '^]'.

->reboot


Press any key to stop auto-boot...
 0
auto-booting...


boot device          : ln
processor number     : 0 
host name            : punaluup
file name            : /usr/vw502b/config/sun1e/vxWorks
inet on ethernet (e) : 192.9.200.23:ffffff00
host inet (h)        : 192.9.200.25
user (u)             : kics
ftp password (pw)    : vertangle
flags (f)            : 0x0 
target name (tn)     : agnew
startup script (s)   : startup_ccd.vxworks

Attaching network interface ln0... done.
Loading... 394968 + 72752 + 19768
Starting at 0x20000...

Attaching network interface ln0... done.
Attaching network interface lo0... done.
Mounting "/usr" from host "punaluup"... done.
Loading symbol table from punaluup:/usr/vw502b/config/sun1e/vxWorks.sym ...done


 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]   
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]   
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]   
      ]]]]]]]]]]]  ]]]]     ]]]]]]]]]]       ]]              ]]]]          TM 
 ]     ]]]]]]]]]  ]]]]]]     ]]]]]]]]       ]]               ]]]]             
 ]]     ]]]]]]]  ]]]]]]]]     ]]]]]] ]     ]]                ]]]]             
 ]]]     ]]]]] ]    ]]]  ]     ]]]] ]]]   ]]]]]]]]]  ]]]] ]] ]]]]  ]]   ]]]]] 
 ]]]]     ]]]  ]]    ]  ]]]     ]] ]]]]] ]]]]]]   ]] ]]]]]]] ]]]] ]]   ]]]]   
 ]]]]]     ]  ]]]]     ]]]]]      ]]]]]]]] ]]]]   ]] ]]]]    ]]]]]]]    ]]]]  
 ]]]]]]      ]]]]]     ]]]]]]    ]  ]]]]]  ]]]]   ]] ]]]]    ]]]]]]]]    ]]]] 
 ]]]]]]]    ]]]]]  ]    ]]]]]]  ]    ]]]   ]]]]   ]] ]]]]    ]]]] ]]]]    ]]]]
 ]]]]]]]]  ]]]]]  ]]]    ]]]]]]]      ]     ]]]]]]]  ]]]]    ]]]]  ]]]] ]]]]] 
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]   
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]       Development System  
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]]   
 ]]]]]]]]]]]]]]]]]]]]]]]]]]]       VxWorks version 5.0.2b for SPARC.
 ]]]]]]]]]]]]]]]]]]]]]]]]]]       KERNEL: WIND version 2.0.
 ]]]]]]]]]]]]]]]]]]]]]]]]]       Copyright Wind River Systems, Inc., 1984-1991

                               CPU: Sun SPARCengine 1E.  Processor #0.
                              Memory Size: 0x400000.

->  tdl 0,2
msg = 0X203 0X54444c 0Xffffff
reply = 0X20002 0Xffffff
reply =        0X20002 0Xffffff
value = 0 = 0x0
-> tTimeout waiting for ISERV, retrying
Got network configuration
Broadcast message sent on ln0
Broadcast not sent on lo0
dl 0,3
msg = 0X303 0X54444c 0Xffffff
reply = 0X30002 0Xffffff
reply =        0X30002 0Xffffff
value = 0 = 0x0
-> tdl 0,3
msg = 0X303 0X54444c 0Xffffff
reply = 0X30002 0Xffffff
reply =        0X30002 0Xffffff
value = 0 = 0x0
-> Timeout waiting for ISERV, retrying
Got network configuration
Broadcast message sent on ln0
Broadcast not sent on lo0
Timeout waiting for ISERV, retrying
Got network configuration
Broadcast message sent on ln0
Broadcast not sent on lo0

B. VME task list

The processes serv_46** are suspended faults due to unanswered broadcasts. If there were many (say 20) of these, the system could hang; in this example it is probably OK. The serv_ tasks can be deleted individually by proceeding the TID with 0x:

-> td 0x3b7ed0 
-------------------------------
-> i
    
  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask   _excTask       3f8ea0   0 PEND          37f60   3f8cf0  3d0002     0
tLogTask   _logTask       3f6d48   0 PEND          37f60   3f6b98   d0003     0
tShell     _shell         3c5818   1 READY         525cc   3c5208   30065     0
tRlogind   _rlogind       3d3138   2 PEND          6f5cc   3d2d68       0     0
tTelnetd   _telnetd       3d1260   2 PEND          6f5cc   3d0fc0       0     0
tNetTask   _netTask       3f1410  50 PEND          6f5cc   3f1268  3d0002     0
tPortmapd  _portmapd      3cfd20 100 PEND          6f5cc   3cfa30      16     0
MLOG_STDOUT_start_mlog_   2b0380 100 PEND          37f60   2afe50   30065     0
cserv      _cserv         297b28 100 PEND          6f5cc   297600  3d0002     0
ccdClock   _ccdClock      27f2d0 100 READY         70f2c   27f188  3d0002     0
broad_mon  _broadcast_m   273d00 100 DELAY         70f2c   273bd0  3d0002  2397
responder  _responder     24f4d8 100 PEND          37f60   24e1f0  3d0002     0
serv_4606  _s_show_elap   3b7ed0 100 SUSPEND       70924   3b7d40  3d0002     0
serv_4644  _s_show_auto   232ab0 100 SUSPEND       70924   232920  3d0002     0
serv_4504  _s_set_time    2301e8 100 SUSPEND       70924   230058  3d0002     0
serv_4644  _s_show_auto   22d920 100 SUSPEND       70924   22d790  3d0002     0
rccd       _rccd          395368 150 PEND+T        6f5cc   3950c8  3d0002     4
value = 3951828 = 0x3c4cd4
----------------------------------------------------------------------------
-------------------------------

C. Normal exposure with shutter

-------------------------------

Set auto_shutter to 1 on camera 0
check_shutter_status: returning 0
broadcast_camera_event:  sent subject=746

 Fast wipe number 1 of 1 on camera 0 started...
done.
broadcast_camera_event:  sent subject=741
broadcast_camera_event:  sent subject=726
opening shutter for expose on camera 0
check_shutter_status: returning 0
shutter opened for expose on camera 0
Getting raw ADC channel 5 of utility board for camera 0
Got raw analog data.
Send response to host.

Elapsed time = 1, stop_flag=0, e_time=1, subint=0
stop_flag: stop_flag=1  e_time=1 total_time=1
closing shutter for exposure stop on camera 0
safe_do_shutter:  calling do_shutter.
check_shutter_status: returning 0
broadcast_camera_event:  sent subject=747
ccdClock: calling read_ccd...
table[0]= c01800
table[1]= 0
Setting video offsets:  166 156
Using namps= 2  binpix= 1124  dataline= 2048  size= 4603904 pixels
Using preline= 0  preflush= 0  overflush= 0  postline= 0
Using winstart[0]= 0  winsize[0]= 1024
Using winstart[1]= 0  winsize[1]= 1024
msg size = 3 words.  reply size = 2 words
msg = 0X103 0X4c4c4441 0X1
reply = 0X10002 0X444f4e
Load program 1 into vmeinf.
Wrote number of ccds=1 to vmeinf.
Set number of pixels to read = 4603904
 in vmeinf.
in read_ccd2
broadcast_camera_event:  sent subject=729
Camera reading broadcast sent.
calling rccdgetimage
DMA primed, sending message
Image 850789417 IMOK received from 192.9.200.25
returned from rccdgetimage
Pixels left to read = 0
broadcast_camera_event:  sent subject=730
ccdClock: back from read_ccd.
Image 850789417 resent 3 ranges, 186000 of 4603904 pixels
Image 850789417 resent 1 ranges, 12000 of 4603904 pixels
Image 850789417 complete

ERROR CONDITION 1: Sudden inoperation (CCD can not start an exposure)

If the CCD system suddenly cannot start an exposure after an extended period of normal operation, it is likely that too many serv_ tasks have piled up on the crate: 
-> i

  NAME        ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
---------- ------------ -------- --- ---------- -------- -------- ------- -----
tExcTask   _excTask       3f8ea0   0 PEND          37f60   3f8cf0  3d0001     0
tLogTask   _logTask       3f6d48   0 PEND          37f60   3f6b98   d0003     0
tShell     _shell         3c5818   1 READY         525cc   3c5208   30065     0
tRlogind   _rlogind       3d3138   2 PEND          6f5cc   3d2d68       0     0
tTelnetd   _telnetd       3d1260   2 PEND          6f5cc   3d0fc0       0     0
tNetTask   _netTask       3f1410  50 PEND          6f5cc   3f1268  3d0002     0
tPortmapd  _portmapd      3cfd20 100 PEND          6f5cc   3cfa30      16     0
MLOG_STDOUT_start_mlog_   2b0da8 100 PEND          37f60   2b0878  3d0001     0
cserv      _cserv         298550 100 PEND          6f5cc   298028  3d0002     0
ccdClock   _ccdClock      27fcf8 100 PEND          6f5cc   27f880  3d0002     0
broad_mon  _broadcast_m   2767f0 100 DELAY         70f2c   2766c0  3d0002  2595
responder  _responder     24b2f8 100 PEND          37f60   24a010  3d0001     0
serv_4644  _s_show_auto   3b7ed0 100 SUSPEND       70924   3b7d40  3d0002     0
serv_4624  _s_show_numa   231a68 100 SUSPEND       70924   2318d8  3d0002     0
serv_4644  _s_show_auto   22f1a0 100 SUSPEND       70924   22f010  3d0002     0
serv_4632  _s_show_voff   22c8d8 100 SUSPEND       70924   22c748  3d0002     0
serv_4644  _s_show_auto   22a010 100 SUSPEND       70924   229e80  3d0002     0
serv_4622  _s_show_ccdg   226710 100 SUSPEND       70924   226580  3d0002     0
serv_4624  _s_show_numa   222e10 100 SUSPEND       70924   222c80  3d0002     0
serv_4624  _s_show_numa   21f510 100 SUSPEND       70924   21f380  3d0002     0
serv_4604  _s_show_binn   21cc48 100 SUSPEND       70924   21cab8  3d0002     0
serv_4630  _s_show_voff   21a380 100 SUSPEND       70924   21a1f0  3d0002     0
serv_4604  _s_show_binn   216a80 100 SUSPEND       70924   2168f0  3d0002     0
serv_4604  _s_show_binn   2141b8 100 SUSPEND       70924   214028  3d0002     0
serv_4606  _s_show_elap   2118f0 100 SUSPEND       70924   211760  3d0002     0
serv_4604  _s_show_binn   20f028 100 SUSPEND       70924   20ee98  3d0002     0
serv_4904  _s_show_raw_   20c760 100 SUSPEND       70924   20c5d0  3d0002     0
serv_4610  _s_show_wind   209e98 100 SUSPEND       70924   209d08  3d0002     0
serv_4606  _s_show_elap   206598 100 SUSPEND       70924   206408  3d0002     0
serv_4606  _s_show_elap   203cd0 100 SUSPEND       70924   203b40  3d0002     0
serv_4502  _s_set_binni   2003d0 100 SUSPEND       70924   200240  3d0002     0
serv_4816  _s_set_utb_d   1fcad0 100 PEND          6f5cc   1fc898  3d0002     0
rccd       _rccd          395390 150 SUSPEND      3fe730   395010  3d0002     0
value = 3951892 = 0x3c4d14
SOLUTION: Reboot the CCD crate. This can be done by resetting the toggle switch on the VME Sun1E board, or simply by typing "reboot":

->reboot

ERROR CONDITION 2: CCD Leach controller (saddlebag) has no power

These messages are a result of attempted utility board reads (the utility board is in the saddlebag) when the saddlebag is powered down. These utility board reads occur automatically from the LRIS control software. 
-------------------------------

vmemsgxchng:  DSP reply timeout.  Check power/hardware.
E1:Time=849969:"util_mem_rd.c", line 31: broad_mon, 275e80, "util_mem_rd ()",
Error, #1: Undefined error condition. Bad return 12 from vmemsgxchng() sending RDM packet

E1:Time=849969:"get_dsp_data.c", line 108: broad_mon, 275e80, "get_dsp_data ()",
Error, #1: Undefined error condition. util_mem_rd returned -1007 reading loc 400007 of camera 0

E1:Time=849969:"broadcast_ccd_analog_inputs.c", line 72: broad_mon, 275e80, "broadcast_ccd_analog_inputs ()",
Error, #1: Undefined error condition. get_dsp_data returned 3 reading UTB adc chan 0 from camera 0

vmemsgxchng:  DSP reply timeout.  Check power/hardware.
E1:Time=849999:"util_mem_rd.c", line 31: broad_mon, 275e80, "util_mem_rd ()",
Error, #1: Undefined error condition. Bad return 12 from vmemsgxchng() sending RDM packet

E1:Time=849999:"get_dsp_data.c", line 108: broad_mon, 275e80, "get_dsp_data ()",
Error, #1: Undefined error condition. util_mem_rd returned -1007 reading loc 400008 of camera 0

E1:Time=849999:"broadcast_ccd_analog_inputs.c", line 72: broad_mon, 275e80, "broadcast_ccd_analog_inputs ()",
Error, #1: Undefined error condition. get_dsp_data returned 3 reading UTB adc chan 1 from camera 0

vmemsgxchng:  DSP reply timeout.  Check power/hardware.
E1:Time=850029:"util_mem_rd.c", line 31: broad_mon, 275e80, "util_mem_rd ()",
Error, #1: Undefined error condition. Bad return 12 from vmemsgxchng() sending RDM packet

E1:Time=850029:"get_dsp_data.c", line 108: broad_mon, 275e80, "get_dsp_data ()",
Error, #1: Undefined error condition. util_mem_rd returned -1007 reading loc 400009 of camera 0

E1:Time=850029:"broadcast_ccd_analog_inputs.c", line 72: broad_mon, 275e80, "broadcast_ccd_analog_inputs ()",
Error, #1: Undefined error condition. get_dsp_data returned 3 reading UTB adc chan 2 from camera 0

-------------------------------
SOLUTION: Restore power to the CCD controller on LRIS and cycle the shutter control box power.

ERROR CONDITION 3: The camera shutter will not work.

These messages occur when the Leach controller can not send open and shut signals to the shutter. 
-------------------------------

check_shutter_status: returning -1103
E1:Time=41488:"s_cshutter.c", line 491: serv_4714, 3b9310, "check_shutter_status ()",
Error, #1: Undefined error condition. Camera shutter is partially open!

E1:Time=41488:"s_cshutter.c", line 318: serv_4714, 3b9310, "do_shutter ()",
Error, #1010: Error number not found in error message table. Bad return from check_shutter_status.

E1:Time=41488:"s_set_erase.c", line 234: serv_4714, 3b9310, "set_erase ()",
Error, #1010: Error number not found in error message table. Bad return from close_shutter()

E1:Time=41488:"s_expose.c", line 254: serv_4714, 3b9310, "s_expose ()",
Error, #1: Undefined error condition. Bad return from set_erase()

-------------------------------
SOLUTION: Cycle power on the shutter control box. If this fails, check the cabling from the Leach saddlebag to the shutter control box, and from there to the shutter housing in front of the dewar.

ERROR CONDITION 4: There is no data transfer between the CCD controller and the VME crate.

The messages from this condition are the same as for ERROR 2; the system behaves as if there was no power being supplied to the Leach controller saddlebag. The problem is arising from a broken or poor fiber connection.

DIAGNOSTIC: If it is known that the CCD power is on, attempt the following diagnostic test, which tests the data link ('tdl') between the VME crate and the timing board in the saddlebag: These are examples of good returns: 

-> tdl 0,2
msg = 0X203 0X54444c 0Xffffff
reply = 0X20002 0Xffffff
reply =        0X20002 0Xffffff
value = 0 = 0x0
-> tdl 0,3
msg = 0X303 0X54444c 0Xffffff
reply = 0X30002 0Xffffff
reply =        0X30002 0Xffffff
value = 0 = 0x0
These are examples of bad returns: 
-> tdl 0,2
msg = 0X203 0X54444c 0Xffffff
vmemsgxchng:  DSP reply timeout.  Check power/hardware.
tdl:  vmemsgxchng() failed.
value = -1007 = 0xfffffc11
-> tdl 0,3
msg = 0X303 0X54444c 0Xffffff
vmemsgxchng:  DSP reply timeout.  Check power/hardware.
tdl:  vmemsgxchng() failed.
value = -1007 = 0xfffffc11
-------------------------------
SOLUTION: The fiber chain must be checked. Make sure that the proper channels are connected in the LRIS-Cass panel (Nas-deck or Cass), and at the interconnect panel in the computer room. Check all connections for cleanliness. Conduct a fiber dB-loss test if necessary on the individual segments.