Troubleshooting Swap Space Problems

The NIRSPEC software is a conspicuous and continuous consumer of memory on the host computer waimea. This results in a steadily shrinking amount of available disk swap space. If the swap space runs to zero during observing, the system will crash and a reboot of waimea will almost certainly be necessary.

The amount of swap space available on waimea has recently (as of 18 December 2000) been increased from about 300 Mb, to about 500 Mb, and then to about 1.5 Gb (yes, that's GIGAbytes). It is devoutly to be hoped that swap space problems will not come up during normal observing after these increases, but if they do, please be sure to have the OA report them in the nightlog.
There is a cron job that checks the amount of swap space every 10 minutes. If it detects that the available space has fallen below 50 Mb, a recorded announcement will be played in Remote Ops 2: "Warning! Waimea's tmp directory has fallen below 50 megabytes. You will run out of swap space soon!" If you hear this, take it seriously!

1. How To Check The Available Swap Space
2. Monitoring Swap Space Usage Over Time
3. What To Do When You Hear The Recorded Warning
4. How To Prevent Swap Space Problems Before They Start

1. How To Check The Available Swap Space

   On any terminal window with a waimea> prompt, enter the command:

   waimea> swap -s

   You should see something like the following output:

   total: 237832k bytes allocated + 37280k reserved = 275112k used, 1310608k available

   The last number, the number of kilobytes of swap available, is the critical number. If this drops below 50000k you will start to hear the recorded warnings.

2. Monitoring Swap Space Usage Over Time

   Available swap space is logged 7 times per hour (every 10 minutes plus an extra entry 48 min after the hour) by a cron job running on waimea. The log file is:

   • /sdata600/logs/swapwatch.log

   Check the last 12 hours' worth of entries with the command:

   • tail -84 /sdata600/logs/swapwatch.log

   issued in a window on waimea. By the way, if you are logged intp a waimea window as user "nirspec", you can go straight to the log directory with the alias cdlog.

   A line of the logfile looks like this:

   • Wed Dec 20 10:50:01 HST 2000: swapspace avail in /tmp (Kbytes): 1204536

   and of course the last number is the one to watch.

3. What To Do When You Hear The Recorded Warning
   • Check the available swap space several times as you work through these steps.
   • Finish only the current observing sequence; if you are running an observing sequence from the EFS GUI, let it finish normally, but don't start another one.
   • Take whatever calibration lamp exposures have not yet been taken for your current grating and slit settings.
   • Gracefully shutdown the NIRSPEC control software by putting the mouse pointer on the blue background, holding down the right button, going into the NIRSPEC Control Menu and selecting **Close NIRSPEC & Stop Software. Wait until all the windows are gone except for the Rotator GUI and the EFS GUI; exit those two GUIs using the Quit buttons or the window manager
   • Restart the NIRSPEC control software just as you did in the afternoon by putting the mouse pointer on the blue background, holding down the right button, going into the NIRSPEC Control Menu and selecting Start NIRSPEC Control Software. Work your way through the script just as you did in the afternoon, except that you can choose the option n to start up in nighttime mode, and the option r to reload previous values for observer and data directory.

   When you hear the recorded warning, you probably have about 1 hour of operation left before the world comes to an abrupt end. This is done to give observers time to finish their current observing sequence and take any necessary calibration frames before shutting down. But don't try to push your luck on the time. It may take 5 to 10 minutes to shut down and restart, but it will take 30 minutes or more to recover from a crash if you don't shut down in time. Shutting down and restarting gracefully frees up enough memory that you should be able to finish the night without further incidents.

4. How To Prevent Swap Space Problems Before They Start

   The NIRSPEC software consumes swap space continuously while it is running, even if no data are being taken, no motors being moved, etc. Also, all the IDL processes running for the QuickLook Displays, Rotator GUI, EFS GUI, Slitmove widget and other tools consume a lot of swap. There are two very important things every observer should do to prevent having to shut down during the night:

   • Shut down the NIRSPEC control software completely before you go to bed. In addition to freeing up memory, this ensures that you will get a fresh start in a clean data directory the next day, and it leaves the instrument in a state where Observatory staff can work on it during the day if needed. So don't go to bed and leave the software up because you want darks. Take your darks in the afternoon.
   • Don't run multiple copies of QuickLook or the EFS GUI. The single QuickLook and EFS GUI for the observer's console should be the only ones running during the night. Even if the other copies are displaying on another computer's screen, they are running on waimea. These are both IDL processes which are probably burning lots of swap space.