NIRSPEC Data Reduction Pipeline

NSDRP can be used to reduce NIRSPEC data taken prior to the instrument upgrade in September 2019.

Introduction

NSDRP is an automatic level-1 data reduction pipeline for Keck Observatory's NIRSPEC's high-resolution (0.947 - 2.63 um) object spectra. All phases of reduction - including order location and tracing, flat fielding, background subtraction, spectral extraction, wavelength calibration and data product generation - proceed automatically without user intervention.

NSDRP is optimized for point sources but extended objects are accommodated provided there is a discernible peak in the spatial intensity profile. It can reduce a wide range of echelle and cross disperser angles and all of the high resolution slits. NSDRP works best on data that have:
  • single point source in slit
  • reasonably bright targets with detectable continuum
  • NIRSPEC-1 through NIRSPEC-7 filters
  • well-separated orders without overlapping
  • sufficient exposure times (~> 30s) with detectable sky lines
Single frames can be reduced in which case sky windows adjacent to the spatial peak are used for background subtraction. AB nod pairs can be reduced in which case sky background is removed by pair subtraction. OH sky emission lines are used for wavelength calibration. For short exposures with weak sky lines, NSDRP uses the grating equation (see software design document) to approximate the wavelength scale. A whole night of data can be reduced together in which case wavelength calibration determined from long exposure frames can be applied to short exposure frames taken in the same temporaly contiguous instrument configuration. Low-resolution mode and wavelength calibration using arc lamps have been identified for future development.

Data products generated from the reduction results include flux and noise spectra and spatial profiles for each order and wavelength calibration line identification tables in ASCII and FITS formats as well as preview plots in PNG format. Two types of log files are produced: one to record per-night summary data and another to record details of data reduction on a per object frame basis.

The latest version of NSDRP is available for download via GitHub. For detailed instructions on how to run NSDRP, including a description of command line options, see the usage section. For more information about the NSDRP design and data products please refer to the technical information.

Technical Information

NSDRP is written in the Python programming language and makes extensive use of common libraries including numpy, astropy, scipy and matplotlib. The software is written mainly in the procedural style, though the main application-specific data structures are implemented as Python classes.

Detailed information about this design and NSDRP data reduction routines, such as spectral extraction and wavelength calibration, is described in the software design document. The data products document provides examples and descriptions of generated preview plots, ASCII tables, and FITS files.

Prerequisites:

  1. Python 2.7.8 and Anaconda 2.1.0 are the recommended versions for running NSDRP, as those are the versions it was developed with. NSDRP will run on the latest Anaconda release, version 4.3.1. You can get Anaconda from https://www.continuum.io/donwloads. NSDRP requires several Python modules and Anaconda comes installed with all of them.

  2. NSDRP will run on MacOS, Windows or Linux based systems.

  3. NSDRP can be run on the standard Python installation, but additional modules may need to be installed. Python version 2.6.1 is the earliest known and tested release which is compatible with NSDRP. NSDRP will not run using Python 3.

  4. Verify correct installation by typing "python" on the command line. The terminal will display the version number and the distribution, i.e. Python2.7.8, Anaconda 2.1.0, etc.

Installation:

  1. NSDRP can be cloned or downloaded from GitHub:

    git clone https://github.com/Keck-DataReductionPipelines/NIRSPEC-Data-Reduction-Pipeline.git

    Download from https://github.com/Keck-DataReductionPipelines/NIRSPEC-Data-Reduction-Pipeline and unzip the prepared NIRSPEC-Data-Reduction-Pipeline_master.zip file

  2. To verify the installation, change directory to where the source files exist and run

    python nsdrp.py rawfits test_output

    Output will be written to the test_output directory and can be compared with the products in the downloaded sample_products.zip file.

    Any required Python modules that are not installed will be printed to the terminal and NSDRP will halt execution.

Additional information can be found in the troubleshooting guide.

Usage

Prior to runnning NSDRP for the first time be sure to read the README file contained in the source distribution.

There are two ways to run NSDRP:

  1. At the telescope in quick-look mode
  2. After observing to process an entire night's data or a single object

At the Telescope

At the telescope, NSDRP produces quick-look products for a single science exposure or a single AB pair. A flat image is required and, in the case of A-B nods, the B pair can be supplied for better sky subtraction and signal-to-noise. A script is available to run NSDRP from the command line:
nsdrp flat object [-b nod_pair]
The following arguments are required:
  • flat: FITS filename of the flat image.
  • object: FITS filename of the object frame
  • nod_pair: optional FITS filename of the B frame in the A-B nod exposure
All generated data products are saved to the 'reduced' directory in the current working directory.

After Observing

After observing, NSDRP can be run in the same way as above for a single science exposure or it can process an entire night's data. When processing an entire night's data, NSDRP should be run only on files downloaded from KOA. Only NIRSPEC files ingested into the archive are guaranteed to contain the required image type (on-sky object or calibration) FITS keyword. It also does not support A-B pair subtraction.
nsdrp.py input_dir output_dir
The following arguments are required:
  • input_dir: directory containing all FITS files to reduce
  • output_dir: directory where the data products will be written
Optional command line arguments are listed here and can also be described by executing:
nsdrp.py -h or --help

usage: nsdrp.py [-h] [-debug] [-verbose] [-subdirs] [-dgn] [-npy] [-no_cosmic]
[-no_products] [-obj_window OBJ_WINDOW]
[-sky_window SKY_WINDOW] [-sky_separation SKY_SEPARATION]
[-oh_filename OH_FILENAME] [-int_c] [-lla LLA] [-pipes]
[-shortsubdir] [-ut UT] [-gunzip] [-spatial_jump_override]
[-out_dir OUT_DIR] [-b B] [-jpg]
arg1 arg2
NSDRP
positional arguments:
arg1 input directory (KOA mode) | flat file name (cmnd line mode)
arg2 output directory (KOA mode) | object file name (cmnd line mode)
optional arguments:
-h, --help show this help message and exit
-debug enables additional logging for debugging
-verbose enables output of all log messages to stdout, always true in command line mode
-subdirs enables creation of per object frame subdirectories for data products,ignored in command line mode
-dgn enables storage of diagnostic data products
-npy enables generation of numpy text files for certain diagnostic data products
-no_cosmic inhibits cosmic ray artifact rejection
-no_products inhibits data product generation
-obj_window OBJ_WINDOW object extraction window width in pixels
-sky_window SKY_WINDOW background extraction window width in pixels
-sky_separation SKY_SEPARATION separation between object and sky windows in pixels
-oh_filename OH_FILENAME path and filename of OH emission line catalog file
-int_c revert to using integer column values in wavelength fit
-lla LLA calibration line location algorithm, 1 or [2]
-pipes enables pipe character seperators in ASCII table headers
-shortsubdir use file ID only, rather than full KOA ID, for subdirectory names, ignored in command line mode
-ut UT specify UT to be used for summary log file, overrides auto based on UT in first frame
-gunzip forces decompression of compressed FITS files, leaves both the .gz and .fits files in the source directory. Note that the compressed files can be read directly, so it is not necessary to decompress them.
-spatial_jump_override inhibit rejection of order edge traces based on 'jump' limit)
-out_dir OUT_DIR output directory in command line mode [.], ignored in KOA mode
-b B filename of frame B in AB pair
-jpg store preview plots in JPG format instead of PNG
For testing, the NSDRP download from GitHub includes a rawfits directory with some sample FITS files. Executing the following command
nsdrp.py -oh_filename /path/to/ir_ohlines.dat /path/to/rawfits /your/output/path
will result in the data products listed below. For a direct comparison, the actual data products are part of the NSDRP downloand and can be found in the sample_products.zip file.
/your/output/path/log NS.20100429.log
NS.20100429_summary.csv
NS.20100429.40280.log
/your/output/path/flats NS.20100429.32524_flat_0.log
NS.20100429.32524_flat_0.fits
/your/output/path/fitstbl/profile NS.20100429.40280_50_profile_tbl.fits
NS.20100429.40280_49_profile_tbl.fits
NS.20100429.40280_48_profile_tbl.fits
NS.20100429.40280_47_profile_tbl.fits
NS.20100429.40280_46_profile_tbl.fits
NS.20100429.40280_45_profile_tbl.fits
NS.20100429.40280_44_profile_tbl.fits
NS.20100429.40280_43_profile_tbl.fits
/your/output/path/fitstbl/wavecal NS.20100429.40280_wavecal_tbl.fits
/your/output/path/fitstbl/flux NS.20100429.40280_50_flux_tbl.fits
NS.20100429.40280_49_flux_tbl.fits
NS.20100429.40280_48_flux_tbl.fits
NS.20100429.40280_47_flux_tbl.fits
NS.20100429.40280_46_flux_tbl.fits
NS.20100429.40280_45_flux_tbl.fits
NS.20100429.40280_44_flux_tbl.fits
NS.20100429.40280_43_flux_tbl.fits
/your/output/path/fits/snr NS.20100429.40280_50_snr.fits
NS.20100429.40280_49_snr.fits
NS.20100429.40280_48_snr.fits
NS.20100429.40280_47_snr.fits
NS.20100429.40280_46_snr.fits
NS.20100429.40280_45_snr.fits
NS.20100429.40280_44_snr.fits
NS.20100429.40280_43_snr.fits
/your/output/path/fits/profile NS.20100429.40280_50_profile.fits
NS.20100429.40280_49_profile.fits
NS.20100429.40280_48_profile.fits
NS.20100429.40280_47_profile.fits
NS.20100429.40280_46_profile.fits
NS.20100429.40280_45_profile.fits
NS.20100429.40280_44_profile.fits
NS.20100429.40280_43_profile.fits
/your/output/path/fits/order NS.20100429.40280_50_order.fits
NS.20100429.40280_49_order.fits
NS.20100429.40280_48_order.fits
NS.20100429.40280_47_order.fits
NS.20100429.40280_46_order.fits
NS.20100429.40280_45_order.fits
NS.20100429.40280_44_order.fits
NS.20100429.40280_43_order.fits
/your/output/path/fits/sky NS.20100429.40280_50_sky.fits
NS.20100429.40280_49_sky.fits
NS.20100429.40280_48_sky.fits
NS.20100429.40280_47_sky.fits
NS.20100429.40280_46_sky.fits
NS.20100429.40280_45_sky.fits
NS.20100429.40280_44_sky.fits
NS.20100429.40280_43_sky.fits
/your/output/path/fits/flux NS.20100429.40280_50_flux.fits
NS.20100429.40280_49_flux.fits
NS.20100429.40280_48_flux.fits
NS.20100429.40280_47_flux.fits
NS.20100429.40280_46_flux.fits
NS.20100429.40280_45_flux.fits
NS.20100429.40280_44_flux.fits
NS.20100429.40280_43_flux.fits
/your/output/path/fits/trace NS.20100429.40280_50_trace.fits
NS.20100429.40280_49_trace.fits
NS.20100429.40280_48_trace.fits
NS.20100429.40280_47_trace.fits
NS.20100429.40280_46_trace.fits
NS.20100429.40280_45_trace.fits
NS.20100429.40280_44_trace.fits
NS.20100429.40280_43_trace.fits
/your/output/path/previews/profile NS.20100429.40280_50_profile.png
NS.20100429.40280_49_profile.png
NS.20100429.40280_48_profile.png
NS.20100429.40280_47_profile.png
NS.20100429.40280_46_profile.png
NS.20100429.40280_45_profile.png
NS.20100429.40280_44_profile.png
NS.20100429.40280_43_profile.png
/your/output/path/previews/trace NS.20100429.40280_50_trace.png
NS.20100429.40280_49_trace.png
NS.20100429.40280_48_trace.png
NS.20100429.40280_47_trace.png
NS.20100429.40280_46_trace.png
NS.20100429.40280_45_trace.png
NS.20100429.40280_44_trace.png
NS.20100429.40280_43_trace.png
/your/output/path/previews/sky NS.20100429.40280_50_sky.png
NS.20100429.40280_49_sky.png
NS.20100429.40280_48_sky.png
NS.20100429.40280_47_sky.png
NS.20100429.40280_46_sky.png
NS.20100429.40280_45_sky.png
NS.20100429.40280_44_sky.png
NS.20100429.40280_43_sky.png
/your/output/path/previews/order NS.20100429.40280_50_order.png
NS.20100429.40280_49_order.png
NS.20100429.40280_48_order.png
NS.20100429.40280_47_order.png
NS.20100429.40280_46_order.png
NS.20100429.40280_45_order.png
NS.20100429.40280_44_order.png
NS.20100429.40280_43_order.png
/your/output/path/previews/snr NS.20100429.40280_50_snr.png
NS.20100429.40280_49_snr.png
NS.20100429.40280_48_snr.png
NS.20100429.40280_47_snr.png
NS.20100429.40280_46_snr.png
NS.20100429.40280_45_snr.png
NS.20100429.40280_44_snr.png
NS.20100429.40280_43_snr.png
/your/output/path/previews/flux NS.20100429.40280_50_flux.png
NS.20100429.40280_49_flux.png
NS.20100429.40280_48_flux.png
NS.20100429.40280_47_flux.png
NS.20100429.40280_46_flux.png
NS.20100429.40280_45_flux.png
NS.20100429.40280_44_flux.png
NS.20100429.40280_43_flux.png
/your/output/path/ascii/wavecal NS.20100429.40280_wavecal.txt
/your/output/path/ascii/flux NS.20100429.40280_50_flux.txt
NS.20100429.40280_49_flux.txt
NS.20100429.40280_48_flux.txt
NS.20100429.40280_47_flux.txt
NS.20100429.40280_46_flux.txt
NS.20100429.40280_45_flux.txt
NS.20100429.40280_44_flux.txt
NS.20100429.40280_43_flux.txt
/your/output/path/ascii/profile NS.20100429.40280_50_profile.txt
NS.20100429.40280_49_profile.txt
NS.20100429.40280_48_profile.txt
NS.20100429.40280_47_profile.txt
NS.20100429.40280_46_profile.txt
NS.20100429.40280_45_profile.txt
NS.20100429.40280_44_profile.txt
NS.20100429.40280_43_profile.txt


KOA Help Desk

To report any issues with NSDRP, please visit the KOA help desk.

Acknowledging the Archive

This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExScI), under contract with the National Aeronautics and Space Administration.

Please also acknowledge the PI(s) of datasets that have been obtained through KOA.

The Keck Observatory Archive (KOA) is a collaboration between the NASA Exoplanet Science Institute (NExScI) and the W. M. Keck Observatory (WMKO). NExScI is sponsored by NASA's Exoplanet Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory (JPL).