User Interface Access to the Keck Observatory Archive

KOA STATUS

Archive is running normally.

KOA LINKS

  User Interface

  FAQ

  Filename translator

  Reducing Keck Data

  HIRES News

  HIRES Data Description

  HIRES Keywords

  HIRES Page

  KCWI Keywords

  NIRC2 News

  NIRC2 Data Description

  NIRC2 Keywords

  NIRC2 Page

  NIRES News

  NIRES Page

  NIRES Keywords

  NIRSPEC News

  NIRSPEC Data Description

  NIRSPEC Keywords

  NIRSPEC Page

  LRIS Keywords

  MOSFIRE Keywords

  DEIMOS Keywords

  ESI Keywords

  OSIRIS Keywords

  NIRC Keywords

  LWS Keywords

  Help Desk

General Questions
  1. What is KOA?
  2. Are data from all Keck instruments available in the KOA?
  3. What type of data are available?
  4. What are metadata?
  5. Where can I find more information about the instruments and data products?
  6. How far back in time are data available in the archive?
  7. What is the dark feature present in the middle of the chip in HIRES 'legacy' data?
  8. What is used to determine WCS information for NIRC2 images?
  9. I would like to obtain weather/environmental information for the data. Is this possible?
  10. Is there a proprietary period for the archived data?
  11. What is a rectification matrix, and why do I need it?
  12. How do I acknowledge KOA in my publications?
Usage
  1. Where is the KOA user interface (UI)?
  2. Do I need to register to request/download data from the KOA?
  3. How do I find data for a specific type or list of objects?
  4. I'm a PI of an observing program. How can I access my proprietary data?
  5. How can I access/change my user password?
  6. How soon after my observing run are my data available on the KOA?
  7. Can I request that tapes or DVDs of data be sent to me?
  8. Can I preview the data before downloading them?
  9. The names of the files downloaded from KOA are not in a format familiar to me. Is there a way to convert the KOA filenames back to the original filenames?
  10. I have downloaded the raw data. Now, how do I reduce them?
  11. Who do I contact for questions, comments or concerns?
Extracted Spectra (HIRES only)
  1. What are extracted spectra?
  2. How are they produced?
  3. What products are available?
  4. How do I get them?
  5. What is the “Quality Assessment” (QA)?
  6. What are some of the known issues?
  7. What software are available to work with the data?
  8. How do I use the different products?
  9. How do I do my own reduction?

General Questions
  1. What is KOA?

    KOA stands for the Keck Observatory Archive, a NASA-funded project jointly developed and operated by the W. M. Keck Observatory (WMKO) and the NASA Exoplanet Science Institute (NExScI).

  2. Are data from all Keck instruments available in the KOA?

    Yes, KOA is currently archiving and serving public data from all current and retired instruments at Keck. They include the High Resolution Echelle Spectrograph (HIRES), the Near-Infrared Camera (NIRC2), the Near-Infrared Echelle Spectrometer (NIRSPEC), LRIS, MOSFIRE, DEIMOS, ESI, OSIRIS, and the two decommissioned instruments NIRC and LWS. Data from the Keck Interferometer (KI) are also available through a dedicated KOA interface.

  3. What type of data are available?

    For all instruments, KOA serves the raw data including all calibration files necessary to reduce the raw data. For a limited number of instruments, reduced browse products are also available. We offer extracted spectra for the majority of the HIRES science data. The HIRES extracted spectra are flat-fielded, bias-subtracted, and wavelength calibrated data intended primarily for quick data quality assessment. For more information about extracted spectra go to the Extracted Spectra section. In addition, when available we also serve calibrated NIRC2 images, which include world coordinate system (WCS) information and have been flat-fielded, dark-subtracted, and corrected for bad pixels and optical distortion. The reduced data cubes for OSIRIS are also available when suitable sky frames are found. For LWS imaging data, we make available the coadded 2-D images that have been properly combined from chop-nod data of the 6-D FITS files.

  4. What are metadata?

    Metadata are data which are used to describe the parameters of a given file and are used to search for data files. The metadata are derived from the FITS header keyword information. The metadata consist of two parts: the native keywords from the original primary header unit written by the instrument during readout, and the metakeywords, which are added during the course of evaluating and preparing the FITS files for archiving. Examples of native keywords are: RA, DEC, OBJECT, DECKNAME, OUTDIR. Examples of metakeywords include: KOAID, DATLEVEL, IMAGETYP, INSTSTAT, PROGID, PROGPI.

  5. Where can I find more information about the instruments and data products?

    In general, instrument information can be found on the Keck instruments page, and keyword information is available from the KOA page.

  6. How far back in time are data available in the archive?

    All data that were available to KOA since each instrument's commissioning have been ingested into the archive. For example, for HIRES this includes data from 1994, while for NIRSPEC it includes data from its initial commissioning in 1999, and for NIRC2 since 2001.

  7. What is the dark feature present in the middle of the HIRES chip in 'legacy' data?

    This feature is a cosmetic defect caused by a pen mark near the middle of the old HIRES chip. To quote the HIRES User's Manual, S. Vogt (1994):

    One very prominent distinguishing cosmetic defect of this CCD is a large felt-tip pen mark near the center of the CCD. It was kindly added by some technician at Tektronix to remind us that this virtually flawless $100,000 CCD is only an engineering-grade device. Unfortunately, the folks at Tektronix seemed to have forgotten how to make science-grade devices, so we are stuck with this annoying blob.

  8. What is used to determine WCS information for NIRC2 images?

    The WCS information is determined from the keywords RA, DEC, ROTPOSN, using an instrument PA zero-point of 0.7 deg (keyword INSTANGL) with a correction of -0.252 deg for the narrow camera (Yelda et al. 2010), and pixel scales of 0.009952, 0.019829, and 0.039686 (arcsec/pixel) for the narrow, medium, and wide camera, respectively. In addition, for vertical angle and stationary rotator modes, the parallactic angle (PARANTEL or PARANG) and elevation (EL) are also used.

  9. I would like to obtain weather/environmental information for the data. Is this possible?

    We provide access to the Keck Observatory weather logs and all-sky images (when available) on the main KOA UI page. Click on the "More Search Options" tab. For a particular date, enter the desired date in the search box at the bottom of that page. Plots of the nights temperatures, dew point, humidity, barometric presssure, and full width at half maximum of the guide star images are also available. Detailed instructions are provided here.

  10. Is there a proprietary period for the archived data?

    Yes, the default proprietary period (PP) is 18 months from the date of observations. With a scientific justification and approval by the appropriate selecting official, this period can be extended. The PP can be specified individually for each of the three HIRES CCDs, as well as for any individual night. Full details of the PP policy are located here.

  11. What is a rectification matrix, and why do I need it?

    A rectification matrix (recmat) is needed to convert a raw 2-D OSIRIS spectral frame into a 3-D reduced data cube. This conversion requires a map of the OSIRIS instrumental PSF of each lenslet at all possible wavelengths. These maps are called rectification matrices, which are created periodically for each filter and plate scale combination. Please see this page or the OSIRIS Manual for further information.

  12. How do I acknowledge KOA in my publications?

    If you find KOA useful in your research, please use the following acknowledgement in your publications:

    This research has made use of the Keck Observatory Archive (KOA), which is operated by the W. M. Keck Observatory and the NASA Exoplanet Science Institute (NExScI), under contract with the National Aeronautics and Space Administration.

    In addition, we encourage users to also acknowledge the original PI(s) of any data that have been obtained through KOA.

Usage
  1. Where is the KOA user interface (UI)?

    The KOA user interface can be found at http://koa.ipac.caltech.edu/.

  2. Do I need to register to request/download data from the KOA?

    If you are a primary investigator (PI) on a Keck observing run using an instrument with data in the archive, you should have received an email notifying you of your account information. Non-PI, public access is available for non-proprietary data. You do not need to register to query and access the public data. However, an email address is required to submit a download request.

  3. How do I find data for a specific type or list of objects?

    KOA does not contain detailed information on the type of objects observed. You will first have to create a list of your targets (names, RA, Dec) from some other sources, then upload the list using our table upload facility. The process would look like this: 1) Get a list of objects of interest (e.g., Be stars from SIMBAD, emission-line galaxies from NED), 2) Edit list to comply to KOA format and limit it to 1000 targets, 3) Go to the KOA interface at http://koa.ipac.caltech.edu, and 4) Select the "More Search Options" tab, then select "Multiple Object Table File" and upload the list. For full details, see the use cases here and here.

    Another possibility is to search for certain keywords (e.g., "quasars" or "Be stars") in the Program Title, but there is no guarantee that this returns all valid objects, and this type of search tends to be very slow.

  4. I'm the PI of an observing program. How can I access my proprietary data?

    Visit the KOA User Interface and login with your username/password. You should have received an email from the Keck Observatory notifying you of your KOA account information and the availability of your data in the archive.

  5. How can I access/change my user password?

    Submit your request to the KOA Help Desk for a new password. For security reasons, temporary passwords will only be sent to PIs.

  6. How soon after my observing run are my data available on the KOA?

    Metadata (FITS keyword information) are generally available within 24 hours of the night of observation. The data FITS files are electronically transferred to NExScI starting mid-morning following each night's observing. At NExScI they are automatically ingested into KOA. This process usually takes a few hours to complete.

  7. Can I request that tapes or DVDs of data be sent to me?

    All data should be downloaded from the KOA interface. We do not take requests for data on hard media.

  8. Can I preview the data before downloading them?

    Yes, JPEG preview images of the raw and calibrated data are available for quick-look purposes from the UI search result page. Previews of the extracted data are also available from the user interface as GIF images, and through the use of an interactive spectra visualization tool that can pan, zoom, and overplot different spectra. See the online help for details. Postscript versions of the extracted spectra previews (when available) are also included as part of the extracted spectra download package.

    Users are encouraged to download and install DS9, a versatile FITS data visualization package, to be used with their browsers as the default application for FITS files. With DS9 you can view the raw, two-dimensional spectral images, and do cross-cut plots through any sections of the image. For a quick tutorial on how to use DS9, go here. To view the extracted spectra, users can also use VOplot or Fv.

  9. The names of the files downloaded from KOA are not in a format familiar to me. Is there a way to convert the KOA filenames back to the original filenames?

    Yes. The names of data files as ingested into the archive have been standardized into unique identifiers known as "KOAIDs" in the form II.YYYYMMDD.xxxxx.fits, where II is the two-letter code for the instrument, YYYYMMDD is the UT date of observation, and xxxxx is the five-digit index number representing the number of seconds after UT midnight. A UNIX shell script koa_translator has been provided to translate the KOA filenames back to the original filenames, which are especially useful for the original observers, who have observing logs listing original filenames, and for data reduction tools that may expect the original names. A guide on how to run the filename translator is provided in the introductory comments in the script, or can be viewed here.

  10. I have downloaded the raw data. Now, how do I reduce them?

    Please see this page for a compilation of possible data reduction tools available to reduce your Keck data.

  11. Who do I contact for questions, comments or concerns?

    Please submit your questions or comments to the KOA Help Desk.

Extracted Spectra (HIRES only)

Disclaimer: The extracted spectra available from KOA are intended to be used as a browse product only. They are produced by a fully automated pipeline, and may not be suitable for scientific publication. We recommend that you reduce the raw images into extracted spectra yourself, paying attention to the specific kind of data that you are reducing (point source vs. extended source, the types and quantities of calibration frames, etc.).

  1. What are extracted spectra?

    The raw data consist of a two-dimensional image on the CCDs, each of which contains a number of separate spectral orders spanning across it.
    The extracted spectra are one-dimensional spectra of source counts vs. wavelength. There is one extracted spectrum for each order on the raw image.
    The extraction is performed by correcting the raw image for CCD bias, applying a flat field correction, collapsing each order into a spectrum, and subtracting the background surrounding the science object. As available, separate arc lamp images are reduced to produce a wavelength per pixel scale for each order in the extracted spectra.

  2. How are they produced?

    Each raw science image is processed automatically through a spectral extraction program called MAKEE. MAKEE was written specifically to reduce HIRES data, but is not officially supported or maintained by W. M. Keck Observatory or KOA. Detailed information on MAKEE, as well as download instructions, can be found on the MAKEE page.

    Each raw science file is reduced separately. If there are multiple observations/science files of the same target, those observations are not combined. Within each raw science file, each CCD is reduced separately, producing one set of extracted spectra per CCD. The following steps briefly outline how the raw data are processed and the individual orders extracted for each CCD.

    During the automated data reduction, a set of calibration files is associated with each raw science file using a calibration association algorithm. The calibration association returns a list of appropriate bias frames, flat fields, and wavelength calibration files. If the list of calibration files is large, in some cases only a subset will be used in the processing. To keep the list manageable and to reduce processing time, a maximum of 10 flat field images are used to generate the median flat field. A similar procedure is used to produce the median bias.

    A median flat field and a median bias image were generated from the calibration set of bias and flat field frames using IRAF. The overscan regions of the science image itself are always used to remove the CCD bias. When available, the median bias frame is then used to optimize the bias subtraction from the science frame. This removes the electronic “offset” put onto the CCD pixels to allow the CCD to record negative fluxes as well as positive fluxes (i.e., to properly account for noise in images with low flux values).

    The median flat field image is normalized so that the median value of the flat field image is unity. After the bias subtraction, the science image is divided by the flat field to remove the pixel-to-pixel variations of sensitivity.

    MAKEE identifies each order by looking for the edges of the orders in the flat field. Within each order, the surrounding background is determined from the edges of the spectrum and subtracted from the spectral order on a column by column basis, removing the sky background from the flux.

    The extracted spectral fluxes are in relative counts. No absolute flux calibration is applied.

    Wavelength calibration is performed by reducing the arc lamp images in the same manner (except no background subtraction is done), identifying known emission lines in the arc lamp spectrum, determining the wavelength to pixel scale, and applying that scale to the science extracted spectrum. The emission lines are identified by correlating the arclamp spectrum with a previously calibrated dataset of arclamp spectra in order to find the approximate position of known emission lines. New centroids are computed for each of these lines and a sixth order polynomial is fit to the resulting set of (pixel, wavelength) pairs, rejecting some lines where necessary. The fits to adjacent echelle orders are interpolated to provide an additional constraint to the polynomial fit. The polynomial fits are used to transform the science extracted spectra from counts vs pixels to counts vs wavelength. The polynomial solutions are included in the FITS header of the extracted spectra.

  3. What products are available?

    The main product of the spectral extraction is the relative flux of the science target vs. wavelength. In addition, during data reduction several ancillary products are generated including S/N estimates of the spectra and pixel traces of the individual spectra, which help provide information on the accuracy and quality of the spectral extractions. They are provided to aid users in assessing the quality of the data.

    The available products include: flux in the science target, uncertainty in that flux, signal-to-noise ratio, sky spectrum, flux profile along the slit, centroids of the profiles as a function of wavelength, etc. The entire list of products, their file names and location in the directory hierarchy, and more detailed descriptions can be found in a
    separate document.

  4. How do I get them?

    To include the extracted spectra as part of the search results. From the search result page, the user may select the radio button "Include extracted spectra" in step 2, . The extracted spectra are intended only as a browse product and are not available for download without the raw data. If you select to download all the raw data resulting from a search, and to include the extracted spectra, you will get extracted spectra from all of the raw data. If you select to download a subset of raw data, you will get the extracted spectra for only those raw data. As for raw data, only extracted spectra whose proprietary period has expired are publicly available.

  5. What is the “Quality Assessment” (QA)?

    In order to interpret the quality of the extractions, KOA has an automated “quality assessment” (QA) grade, that uses the number of errors in fitting the object profile and the number of pixels rejected during sky fitting. The automated nature of this grading process is imperfect, so even the grade may not be correct in the archive. Grading is done order-by-order, and the large number of orders in the archive discourage manual grading of each order.

    The automatic QA grade is either a “P” for “pass” or “F” for “fail”. A “pass” means that reduction artifacts, while they may be present, do not obscure the underlying astronomical data. A “fail” means that no sign of the underlying data is visible in the extraction; it is dominated by reduction artifacts.

    Visual inspection of a random sample of HIRES orders, and comparison of the visual grades with the automated grades, indicate that the automated grading system does a good job on the “pass” grades; only 1.7% of the visually graded passes were erroneously graded “fail”. Of the visually graded 5s, 64.3% were erroneously graded “pass,” indicating that the automated system does not provide a real good indication of poor reduction. This emphasizes the need to inspect the browse products, and perhaps the raw images themselves, to make a good determination of the quality of the browse product in KOA.

    The automated grades for each order are shown in a table on the “Extracted Spectra” Web page.

  6. What are the known issues?

    There are many different types of data sets in the archives, including sets with missing or no calibration files, data on bright and faint objects, point sources and objects that fill the entire slit. The extraction tool, MAKEE, is optimized for point sources, and certain common methods of taking and using calibration files. Some observers may have taken unorthodox calibration frames, or may have taken even the science data in unconventional ways that are not handled properly by MAKEE. Inevitably, some data sets will not reduce properly using the automated data reduction pipeline. A separate document provides a brief guide on how to interpret the extracted products, and describes common problems that you may encounter with the pipeline reductions. For further information, consult the MAKEE page to learn why it might fail.

  7. What software is available to visualize and manipulate the data?

    KOA offers the HIRES Extracted Spectra Visualizer to interactively plot the data. Most of the products are presented in the form of FITS binary tables. These can be read by many common astronomical reduction and visualization programs, including IDL's astro library tools, IRAF, Fv, and VOplot.

  8. How do I use the different products?

    The extracted spectra provided by KOA are browse products intended to give the user some idea of what the extracted spectra might look like when properly reduced. They are not meant to be a final, science-quality product that can be directly measured and analyzed to produce publishable science. It is recommended that archive users reduce the raw image data to produce their own science-quality spectra.

    However, the browse products may give some insight into data reduction issues that users should be aware of. For example, profiles will indicate whether the science object is point-like or extended, whether it is well-centered, and whether there are multiple objects in the slit. Traces can indicate how well the pipeline reduction followed the spectrum. Identifications of the emission lines in the wavelength calibration spectra, and the residuals to the polynomial fits, can give some indication of the precision and accuracy of the wavelength fits. Log files can be used to check for problems during the pipeline reduction process. See the
    MAKEE page for details.

  9. How do I do my own reduction?

    There is a number of data reduction tools that can be used to extract and calibrate spectra. MAKEE is one, and it can be downloaded and installed on your own computer. See the MAKEE page for details. Another tool is HIRESRedux. This IDL package has been tuned for HIRES, but can also reduce data from several other instruments. See the HIRESRedux page for more details.
    Other astronomical reduction packages such as IRAF, IDL, and XVISTA can also reduce HIRES data, although these may not include reduction tools that are tuned specifically for HIRES. Also, consult this compilation of currently available tools.


Return to the KOA Home Page Contact the KOA Master
Last modified on July 13 2016 11:53:44