Rebuilding Metrics

Occasionally problems with the metrics database require rebuilding parts of it. Examples are when nightlog ticket start and/or stop times are revised, when data are temporarily unavailable, but become available at some later time, and when invalid flags are set or removed.

It is important to understand that rebuilding the metrics events makes use of stored “flat files” of events. These files are contained in /h/metrics/data/nightly1 and /h/metrics/data/nightly2, organized into a directory tree the same way that the telescope nightly directories are organized. Under each of these directories is another set of directories representing the year. Under each year are directories representing the months, and under each month are directories representing each day. All of these are in UT. So a directory of /h/metrics/data/nightly1/05/11/14 would contain files with events from Keck 1 on Nov. 14, 2005 UT. Event files have an extension of .Evt.

Simply importing these flat files will only add to the existing events in the database. Usually you want to delete old events and reimport the new ones.

Adding events

Weather, Engineering, Invalid, Current Instrument events

There are a number of ways to add these events, but the common problem of a missing engineering or weather ticket, or adding an invalid flag to a night of data can be accomplished by running one of:

• /home/kics/metrics/bin/addEngEvt (to add an engineering event start/stop pair)
• /home/kics/metrics/bin/addWeaEvt (to add a weather event pair)
• /home/kics/metrics/bin/addInvEvt (to add an invalid event pair)
• /home/kics/metrics/bin/addInstrEvt (to add a “current instrument” event pair)

These programs should be run on kaaawa as kics, and require the telescope number, a date in the form “yyyy-mm-dd”, and optionally two UT times. If the UT times are omitted, 12° twilight times will be used as the start and stop times, so that the event pair will cover the entire night.

Note that this technique will write events into the flat files that metrics creates before it imports the events into the database, but it will not generate Remedy nightlog tickets. Nightlog tickets are a more time-consuming but cleaner and more permanent solution for adding engineering and weather events. You cannot add invalid events to nightlog tickets. In principle you could add “current instrument” events to a DCS log file, but this is not recommended. Normally one does not need to add such instrument events.

1. /home/kics/metrics/bin/addEngEvt 2 2004-10-19
   will add a pair of engineering events that will flag the entire night of Oct. 19, 2004 on Keck 1 as engineering.
   
2. /home/kics/metrics/bin/addWeaEvt 1 2005-01-14 10:05:00 11:35:00
   will add a pair of weather events that will mark the hour and a half between UT 10:00 and 11:35 on Jan. 14, 2005 as weather on Keck 1.

Instrument exposure events

Instrument exposure events are sometimes recreated from dumps of the files’ FITS headers created by the “savier” backup system. You will need to run the FITS headers through the appropriate instrument program in /home/kics/metrics/bin (/esi, /deimos, /nirc, etc.). These programs can be run from any computer, since they only produce a listing of events from the FITS headers. However, the savier catalog files are only visible from one of the autoloader machines (currently awalua and welewele).

As an example, lets say you wanted to recreate the exposure start/stop times from DEIMOS images that were written onto DEIMOS DVDs numbered 318 and 319. The FITS headers are contained in the files /sos/deimos/catalog.318 and /sos/deimos/catalog.319 on awalua or welewele. Logged onto awalua as kics:

1. cd /tmp (or some other directory in which you have write permissions)
2. /home/kics/metrics/bin/deimos /sos/deimos/catalog.318
3. /home/kics/metrics/bin/deimos /sos/deimos/catalog.319
4. Your data are now in deimos.Evt in your current directory. (You can actually specify the output file as a second parameter on the command line if you prefer.) Note that each time you run the command it will append to the output file, not overwrite it.
5. Extracting events from larger sets of files (like all DEIMOS files) can easily be scripted.

These data are not sorted by date, so it is generally good to write the .Evt output file to a temporary file. You can then either parse out the files you want (e.g. from a specific date), or you can run the metrics command

/home/kics/metrics/scripts/organizeInst.pl

This command will take a large file of instrument events and distribute each one of them into the appropriate flat files contained in /h/metrics/data/nightly1 and /h/metrics/data/nightly2. For instance, if the file contains both LRIS and NIRSPEC events for both 2005-02-19 and 2005-02-20, the script will append each event into either nirspec.Evt or lris.Evt in the relevant paths (05/02/19/ and 05/02/20/) in /h/metrics/data/nightly1 (for LRIS) or /h/metrics/data/nightly2 (for NIRSPEC).

To rebuild metrics for a given night

1. Log onto kaaawa as kics.
2. If you have modified Remedy nightlog tickets to correct a problem, run
   
   /home/kics/metrics/bin/nightlogFix [tel#] [yyyy-mm-dd].
   
   This will delete all Remedy events in metrics for that day, and regenerate them from Remedy. It will then run “buildActivity” to wipe the metrics activities previously determined for that night and reload them from the new events.
3. If you have not modified any nightlog tickets, but for example have recreated the instrument exposure start/stop file, or have added an invalid, weather, or engineering event using the shell scripts described in the previous section, you can instead run
   
   /home/kics/metrics/bin/buildActivity [tel#] [yyyy-mm-dd]
   
4. On your Web browser, look at the relevant metrics time line and see whether your fix has had the intended consequences.

Metrics Master
15 July 2005