1) How one can determine whether IP forwarding on keamano is enabled? i.e., what software should be running and how do you determine whether it's operating properly?
2) What remedial action was necessary? i.e., what commands did you execute on keamano and under what account are they to be run?
[1016] kics@keamano% routeadm Configuration Current Current Option Configuration System State --------------------------------------------------------------- IPv4 routing disabled disabled IPv6 routing disabled disabled IPv4 forwarding enabled enabled IPv6 forwarding disabled disabledAs of Feb 2015 following an upgrade to keamano, the third line says that ip_forwarding is either enabled or disabled. Both the configuration and the Current System State need to be set enabled.
routeadm -u
Which takes us to: troubleshooting when rotation control appears to work in all aspects except that DCS tracking fails.
According to the DEIMOS master GD Wirth the Second (MK), this has happened twice in DEIMOS's history.
Aug 20 18:47:55 ... DCS mode not allowed: \ CURRINST is <unknown>; needs to be <deimos>
From 2003: However, if deirotd had already learned that CURRINST = deimos before IP forwarding was disabled, it is possible that the precise symptom would be different. This would be a useful test to try today: put deirot into dcs-tracking mode, disable IP forwarding, and see what happens in the logfile. ]]
kics@polo% show -s dcs2 currinst currinst = DEIMOSIf this hadn't worked, the problem would plainly be more widespread than roto. Go investigate other sources of trouble, e.g. ensure that DCS is up and functioning.
rotop:kics% show -s dcs2 currinst(Note that the service name *must* be "dcs2", not "dcs"; the command will fail with the complaint that the shareable library libdcs_keyword.so cannot be found. This is done to avoid any ambiguity about which DCS service is used by roto -- we don't want roto accidentally sending rotator state updates to K1!) If the show command succeeds, the problem is within deirot itself. Try restarting the rotator software: "deimos restart deirot".
rotop:kics% ping aux2 ... rotop:kics% ping tdc2 ...If the ping command succeeds, the source of trouble is completely unclear to me... as an act of desperation, reboot roto by opening the Bay C panel on DEIMOS, and power cycling the rotation computer (it's in a pizza-box style enclosure).
Check roto's routes:
rotop:kics 102 % netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.6.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 192.168.6.1 0.0.0.0 UG 0 0 0 eth0This shows that all packets going outside the DEIMOS private network are routed through 192.168.6.1, ie keamano.
If the routing was wrong, you can try reconfiguring the routes ("netstat add default gw 192.168.6.1") or brutally rebooting as described in step (3).
keamano% /etc/init.d/ip_forwardThe output will be something like:
Usage: /etc/init.d/ip_forward { start | stop } Currently: ndd -get /dev/tcp ip_forwarding = 1The second line says that ip_forwarding is either on (ip_forwarding = 1) or off (ip_forwarding = 0). If it on, ask your Keck experts to check that aux2 and tdc2 have static routes to roto (192.168.6.6) that go via keamano.
keamano# /etc/init.d/ip_forward startIt is likely that the rotator software is in a confused state, since it was completely unable to talk to DCS. Therefore restart it:
kics@polo% deimos restart deirot