1) How one can determine whether IP forwarding on keamano is enabled? i.e., what software should be running and how do you determine whether it's operating properly?

2) What remedial action was necessary? i.e., what commands did you execute on keamano and under what account are they to be run?

Short version

Check IP forwarding on keamano by typing (as either root or kics): routeadm The output will be something like:

[1016] kics@keamano% routeadm
              Configuration   Current              Current
                     Option   Configuration        System State
---------------------------------------------------------------
               IPv4 routing   disabled             disabled
               IPv6 routing   disabled             disabled
            IPv4 forwarding   enabled              enabled
            IPv6 forwarding   disabled             disabled
    As of Feb 2015 following an upgrade to keamano, the third line
    says that ip_forwarding is either enabled or disabled. Both the
    configuration and the Current System State need to be set enabled. 

  
 Turn on IP forwarding: If IP forwarding is disabled, we need to
 enable it. Ideally, once we enable ip forwarding, we will never need
 to correct this again. It was last corrected on a date near to my
 heart: 14 Feb 2015. 
     to enable it on the current configuration run
           routeadm -d ipv4-forwarding   
          
 routeadm -e ipv4-forwarding 
        
 You then need to set the current system state to apply
        the changes above: 
           routeadm -u   
  
	
 It is likely that the rotator software is in a confused state
	  since it was completely unable to talk to DCS.  Therefore restart it:
         
 as dmoseng on polo run  deirot_restart 
         
 or as kics on keamano run  deimos restart deirot

Long Version

Background

Polo and keamano have two network interfaces: one to the public .keck.hawaii.edu network, and the other to the DEIMOS private network. Roto, however, has a connection *only* to the private network. It can send packets to hosts outside the private net because it's configured to route such packets through keamano, and keamano is configured to do "IP forwarding" of packets between its own two network interfaces. Other hosts -- such as the DCS crates, DNS servers, etc -- can successfully reply to roto only if they have been configured to know that that packets to roto (192.168.6.6) must be routed via keamano. I believe that only tdc2 and aux2 have been given such "static" routes, whereas other hosts such as Keck's DNS servers have not been told a route to roto. Therefore, K2's DCS crates are the only hosts outside the DEIMOS private network that can reply to any message from roto.

Which takes us to: troubleshooting when rotation control appears to work in all aspects except that DCS tracking fails.

According to the DEIMOS master GD Wirth the Second (MK), this has happened twice in DEIMOS's history.

8 Aug 2003 - around DEIMOS commissioning
14 Feb 2015 - during the keamano upgrade

Symptoms:

On the rotator sub-panel Allowing DCS control does not toggle DCS control of the rotator.
modify -s deirot rotatdcs=yes results in the following error on the command line or in the deirot log
said:
```
	    Aug 20 18:47:55 ... DCS mode not allowed: \
			CURRINST is <unknown>; needs to be <deimos>
```

Other email communication

Below appears to be communication that was captured during troubleshooting efforts. The suggestions below should be tried with caution....

From 2003: However, if deirotd had already learned that CURRINST = deimos before IP forwarding was disabled, it is possible that the precise symptom would be different. This would be a useful test to try today: put deirot into dcs-tracking mode, disable IP forwarding, and see what happens in the logfile. ]]

In an effort to find out if this is a roto-specific problem, a deirot-specific problem, or a more general problem, try showing a DCS keyword from either polo or keamano, and then try showing the same keyword from roto.
```
	kics@polo% show -s dcs2 currinst
		currinst = DEIMOS
```
If this hadn't worked, the problem would plainly be more widespread than roto. Go investigate other sources of trouble, e.g. ensure that DCS is up and functioning.
If the "show" command in step (1) does work from polo, log into roto as kics, and do the same thing:
```
	rotop:kics% show -s dcs2 currinst
```
(Note that the service name *must* be "dcs2", not "dcs"; the command will fail with the complaint that the shareable library libdcs_keyword.so cannot be found. This is done to avoid any ambiguity about which DCS service is used by roto -- we don't want roto accidentally sending rotator state updates to K1!) If the show command succeeds, the problem is within deirot itself. Try restarting the rotator software: "deimos restart deirot".
If the "show" command in step (2) fails, saying that it can't contact the DCS service, check if roto can ping the aux2 and tdc2 crates:
```
	rotop:kics% ping aux2
	...
	rotop:kics% ping tdc2
	...
```
If the ping command succeeds, the source of trouble is completely unclear to me... as an act of desperation, reboot roto by opening the Bay C panel on DEIMOS, and power cycling the rotation computer (it's in a pizza-box style enclosure).
If the ping command in step (3) fails, then roto can't talk to the crates -- yet we know they must be functioning, because the "show" command worked from polo (or keamano). That suggests that roto can't "see" outside the DEIMOS private network. This might be due to (a) roto having a misconfigured routing table, so that it doesn't know to send non-private-network packets via keamano, (b) keamano having IP forwarding turned off, or (c) aux2 and/or tdc2 not having a route back to roto.
Check roto's routes:
```
    rotop:kics 102 % netstat -nr
    Kernel IP routing table
    Destination   Gateway       Genmask       Flags MSS Window  irtt Iface
    192.168.6.0   0.0.0.0       255.255.255.0 U       0 0          0 eth0
    0.0.0.0       192.168.6.1   0.0.0.0       UG      0 0          0 eth0
```
This shows that all packets going outside the DEIMOS private network are routed through 192.168.6.1, ie keamano.
If the routing was wrong, you can try reconfiguring the routes ("netstat add default gw 192.168.6.1") or brutally rebooting as described in step (3).
If the routing in step (4) is correct, check that IP forwarding is enabled on keamano:
```
	keamano%  /etc/init.d/ip_forward
```
The output will be something like:
```
	Usage: /etc/init.d/ip_forward { start | stop }
	Currently: ndd -get /dev/tcp ip_forwarding = 1
```
The second line says that ip_forwarding is either on (ip_forwarding = 1) or off (ip_forwarding = 0). If it on, ask your Keck experts to check that aux2 and tdc2 have static routes to roto (192.168.6.6) that go via keamano.
If the test in step (5) shows IP forwarding is off, turn it on by becoming root on keamano and typing
```
	keamano# /etc/init.d/ip_forward start
```
It is likely that the rotator software is in a confused state, since it was completely unable to talk to DCS. Therefore restart it:
```
    kics@polo%  deimos restart deirot
```