Create a new directory for the night (e.g. 06aug02) and enter it
Create a 'Raw/' directory and put all the raw data in it.
If the filenames do not read esi####.fits, it is quite likely
the code will be unhappy
gzip the data (gzip *.fits)
Create a 'pro/' directory and copy the files from
$XIDL_DIR/ESI/pro/redux into it
(e.g. proc_ech.pro, extract_ech.pro). These files can and should be used to run most of the steps that follow.
Launch idl in the directory above Raw/
esi_strct :: Create the esi structure.
This structure organizes
the entire night of data and is the most important file created. The
structure is listed on the next page.
The routine creates a few things:
An IDL structure in memory with whatever name you choose (e.g. esi);
The file 'esistrct.fits' which is a fits
version of the structure;
The file 'esi.list' which is an ASCII
version of the fits file which lists the values of some but not all tags.
To view the structure outside of IDL, I
recommend the program 'fv' which I think stands for fitsview. It allows
you to examine binary fits tables.
If the EDIT keyword is set, a
gui will launch which allows some editing of the esi structure
(see below).
Example: IDL>
esi_strct, esi, /MKDIR, [/EDIT]
Example: IDL>
proc_ech, esi, /MKSTRCT
Time : 1s per image
esistrct
Tag
Type
Comment
frame
0
FRAME Number
flg_anly
0
Analysis flag 0=Don't Analyse, 2 = bias sub, 4=scatt
esi_editstrct :: Modify the ESI structure.
The previous step creates the structure and takes a guess at the
initial values of many of the tags based on the header card info. It
is difficult, however, to automate all of the values for the tags and
therefore the user should carefully edit the structure. For most of
the important tags, one can use esi_editstrct. The rest must
be done from the command line by hand.
The obvious tags to modify are:
Obj :: Object name (this propagates into the final spectra
and should have no spaces!)
flg_anly :: Include in analysis (defaulted to 1 for yes)
type :: ARC, IFLT, TWI, OBJ, STD, etc.
mode :: ECH, LWD, IMG
Example: IDL>
esi_editstrct, esi
Time : User interaction
Writing the ESI structure to disk:: In IDL you can modify the values
of any of the tags. You can then save the structure in fits form and
rewrite the ASCII file with the routine esi_wrstrct.
Example: IDL>
esi_wrstrct, esi, FITS='name'
Time : fast
Reading the ESI structure from disk:: esi_ar Example: IDL>
esi = esi_ar() Example: IDL>
esi = esi_ar('esi_name.fits') Time : fast
This routine examines the esi structure and looks for calibration
files associated with the various slit widths of your ECH files.
It groups together exposures with identical Obj name and ESI mode and
sets the obj_id tags accordingly.
A summary of the ECH exposures is put in 'esiECH_summ.txt'.
Example: IDL>
esi_echsetup, esi
Example: IDL>
proc_ech, esi, /SETUP
Time : Fast
Create Bias (Zero) frame
esi_mkzero
We have found that creating a Zero frame is preferred
over an overscan subtraction. If you have taken Bias (0s) frames,
the process is straightforward. If you haven't, you can copy the image
(or make a soft link) 'BiasS.fits' from
$XIDL_DIR/ESI/CALIBS into the 'Bias/' directory.
Example: IDL>
esi_mkzero, esi
Example: IDL> proc_ech, esi, /MKZERO
Time : 3-5min
To check the Bias frame : xatv, 'Bias/Bias1x1S.fits'
Pinhole Map
One may have taken a series of pinhole traces using the
pinhole mask and the QTZ lamp to trace out the light path along
the CCD. If so, the following task can be used to fit the pinhole
traces and create a MAP which is required for the data reduction.
In lieu of your own pinhole images, you will need to copy the files
(or make soft links)
hole_fit.idl, ECH_map.fits, img_hole.fits from
$XIDL/ESI/CALIBS to Maps/. At this point, I would recommend
using the CALIBS files over creating new ones.
esi_echtrcholes ::
This routine will combine all pinhole
images (denoted by the 9.99 slit width) and then trace the pinholes.
Output is 'Maps/img_hole.fits' and 'Maps/hole_fit.idl'
Example: IDL>
esi_echtrcholes, esi
Example: IDL>
proc_ech, esi, /MKMAP
Time : <10min
esi_echmkmap ::
This routine takes all of the good pinhole traces and makes an optical
map. This is used to rectify the arcs for 2D fitting.
Example: IDL>
esi_echmkmap, esi
Time : <5min
Process Flats
These routines processes your QTZ calibration frames to (1) trace
the edges of each order, (2) calculate the gain mismatch between multiple
amplifiers, and (3) correct pixel-to-pixel variations.
Dome flats show significantly less scattered light than the internal
flats and should be used when possible.
Use the /IFLAT switch to use internal flats.
esi_echmkflat :: The first routine simply combines the flats. The output is
a bias subtracted, median flat placed in 'Flats/'. If one does not specify
the slit size, the program loops through all sizes.
Example: IDL> esi_echmkflat, esi, [slit], [/IFLAT]
Example: IDL>
proc_ech, esi, [slit], /MKFLAT [/IFLAT]
Time : 30s per flat
esi_echfltsct :: The next step is to subtract scattered light and normalize the flat.
The scattered light is quite significant for Internal flats but also
important for Dome flats. This code requires the slit size
explicitly. The routine also normalizes the flat and also matches the
gain on the two halves of the amplified CCD. If you are using the
archived pinhole map (most likely), then you should use the keyword
/KLUDGEOR to deal with an anomoly in Order 14. The
output is 'Flats/FlatECHslit_{bin}N.fits' where slit = 50, 75, or 10 and bin = 1x1, 2x1, etc.
Example: IDL>
esi_echfltsct, esi, slit, [/IFLAT],
[/KLUDGEOR]
Example: IDL>
proc_ech, esi, slit, /NRMFLAT, [/IFLAT]
Time : 1min
Check the Flat: xatv, 'Flats/FlatECHslitN.fits'
Arc Image
This is the most important set of steps. The sky subtraction
algorithm will only succeed with a good Arc Image. It should be
emphasized that we only create one Arc Image per slit size and apply
this to all of the exposures taken with that slit size. The following
routines are well automated but should be checked carefully if you are
doing a 'final' reduction. I think it is possible that one could use
an Arc image from a previous run or the CALIBS database
but wouldn't recommend it just
yet. All of the following routines require a specific slit size.
esi_echmkarc
:: Process the Arcs.
This step bias subtracts
and flat fields the arc images. If there is an image taken with all
the lamps on (arclamp = 7) then the code will take that. Otherwise,
it will the Arcs together to make a CuAr+HgNe+Xe image. Output
is: 'Arcs/ArcECH_slit.fits'
Example: IDL>
esi_echmkarc, esi, slit
Example: IDL>
proc_ech, esi, slit, /MKARC
Time : 1-15min
esi_echfitarc
:: This routine extracts a 1D spectrum down the
center of each order and determines the wavelength solution.
There are two levels of interatction with this routine. The most
interaction (not recommended) is to use /INTER which prompts the
user to identify and fit the Arc lines. I recommend using the
option /PINTER which has the program attempt to identify a set of
lines in each order. The user than interactively fits the lines
using the routine x_identify which calls x1dfit.
As long as your arc lines are within several
pixels of my solution, things ought to run smoothly.
The output is an IDL file containing the polynomial fits for each
order: 'Arcs/ArcECH_slitfit.idl'. The program also creates
a ps file to examine the quality of fits. One can input their
own set of linelists or restrict the code to use only CuAr lines
(/CUAR).
Example: IDL>
esi_echfitarc, esi, slit, /PINTER
Example: IDL>
proc_ech, esi, slit, /FITARC
Time : 2min per order
esi_fit2darc
:: This routine takes the good lines from esi_echfitarc and
then performs a 2D fit to the lines. This is useful for
interpolating through the edges of the blaze.
Example: IDL>
esi_fit2darc, esi, slit
Example: IDL>
proc_ech, esi, slit, /FIT2DARC
Time : 2min per order
esi_echtrcarc
:: This routine traces arc lines in each
order. It first straightens the curved orders, but does not rectify
the arc lines. It simply traces them and then maps back to the
original data frame in the following step. I recommend interactively
choosing the lines the program uses to trace but on can also let
the program choose (/AUTO). Output is a series of files in the
'Arcs/TRC' directory (1 per order) including ps files for rough
inspection.
Example: IDL>
esi_echtrcarc, esi, slit, /AUTO
Example: IDL> proc_ech, esi, slit, /AIMG (This does the following step too)
Time : 2min
(>10 min interactively)
esi_echmkaimg
:: This routine takes the arc traces and
uses them to create a full Arc Image in the data frame.
The final wavelength image is then converted to vacuum wavelengths.
Output is : 'Arcs/ArcECH_slitIMG.fits'
Example: IDL>
esi_echmkaimg, esi, slit, /CHK
Time : 5min
Check output with: xatv, 'Arcs/ArcECH_slitIMG.fits'
Process the Standard
This step is not necessary but will help with
tracing the data and fluxing. Alternatively, one can grab the standard
trace from the CALIBS directory.
esi_echtrcstd :: Process the standard star (bias, flatten),
centroid it using a small region in order 11 (this can be modified),
skysubtract, and then trace it through the 10 orders.
The output are a processed image in 'Final/', an Obj structure in
'Extract' (see below), and the file
'Extract/STD_ECHslit_TRC.fits' which is the trace for the
standard star.
Example: IDL>
esi_echtrstd, esi, slit
Example: IDL>
proc_ech, esi, slit, /PROCSTD
Time : 10min (mainly sky sub)
0. Extraction
The following routines all apply to a single object, i.e. multiple exposures of that object will be reduced together.
I suggest you use a .pro file like extrct_ech.pro
to guide the process. It provides a convenient way
of stepping through the reduction.
Most of the following routines take the esi structure
and obj_id.
Process the Image
esi_echproc
:: Bias subtract and flat field the Raw image.
This routine takes the index number of the esi structure as input.
This is the integer in the first column of the file 'esi.list'.
One can subtract scattered light with the flag (/SUBSCAT), but I
currently don't recommend it. The contribution is small for almost all
objects.
Output is in 'Final/' and is a flattened flux and variance fits file
(one gzipped fits file with two exentsions per image).
Example: IDL>
esi_echproc, esi, indx, [/SUBSCAT]
Example: IDL>
extrct_ech, esi, obj_id, /PROCOBJ, [/SUBSCAT]
Time : 1min per image
Check : xatv, 'Final/f_esi#.fits'
specobjstrct
Tag
Type
Comment
field
' '
Name of field
slit_id
0L
Used to store the order number (0-9) corresponding to
physical order (15-6)
obj_id
' '
ID value (a=primary, b-z=serendip, x=NG)
flg_anly
0
0=No analysis
exp
0.
xcen
0L
Column where obj was id
ycen
0.
flg_aper
0
0=boxcar
aper
fltarr(2)
Widths of aperture, 0/1 = bottom/top (pixels)
skyrms
0.
RMS of sky fit
trace
fltarr(5000)
npix
0L
wave
fltarr(5000)
fx
fltarr(5000)
var
fltarr(5000)
rejected pix
flg_flux
0
0=f, 1=f
flux
fltarr(5000)
Fluxed data
sig
fltarr(5000)
Err in fluxed data
date
0.0d
UT
' '
img_fil
' '
slit_fil
' '
instr_strct
' '
e.g. wfccdstr fits file
Identify the Object
esi_echfndobj
:: Allows the user to interactively identify
the science objects (default is auto selection). It smashes a
region of the spectrum and finds all peaks containing 4
continuous pixels with 3 sigma significance. The region is taken
to be from 5600-5610A in order 4L (physical 11) by default. These
can be changed via the keyword REFWV=fltarr(2) and REFORDR=#L.
If not interactive, the program chooses the brightest object within
20 pixels of the center of the slit. Set the keyword /INTER to
interactively identify the object and a an aperture to mask for
sky subtraction (recommended).
It then creates an object structure which
stores the spectra of all objects identified in the slit.
The IDL structure created is described below.
The routine also does a pseudo-trace to the object based on the
paths of the pin-hole traces. This trace is used to mask out the
object for sky subtraction. At present the code assumes that there
are no serendipitous objects in the slit to mask.
Output is 'Extract/Obj_esi##.fits'
Example: IDL>
esi_echfndobj, esi, obj_id, [exp],
[REFORDR=, REFWV= /NOCLOB, /INTER, /CHK, /STD]
Example: IDL>
extrct_ech, esi, obj_id, /FNDOBJ
Time : fast
esi_echcopyfnd :: If desired, the user can copy the
trace from another object using this routine. The program takes
the trace from the alternate object and allows the user to choose
a different aperture for masking. The keyword CPYIDX is required
and refers to the index number of the template object. An offset
can also be applied to this trace using OFFSET=.
Example: IDL>
esi_echcopyfnd, esi, obj_id, [exp],
CPYIDX=, [APER=, OFFSET=]
Time : fast
Sky Subtraction
esi_echskysub ::
This procedure subtracts the sky from each order, one by one for
each exposure with a given obj_id. The main recipe adopts either
(a) a series of low order POLYNOMIALS row by row for orders 0 to 5
(physical orders 15 to 11) or (b) a 1D
Bspline fit to the wavelength, flux pairs for orders 6-9.
I have found that the polynomials work best in the low sky regions and
the BSPLINE works significantly better in the orders with strong
sky lines.
There are some fancy features added in to deal with
bright sky lines which get switched off or reduced for short exposures.
Current issues:: One current failing of the code is sky subtraction
in order 11 where the amplifiers meet. The ratio of the gain is calculated
from the Flat image and this ratio is not constant in time. This
sometimes leads to over subtraction of sky in some regions.
Options ::
BORDR= 5L :: Sets the order (non-physical) where the Bspline
routine takes over.
ORDR= [0L, 9L] :: Sets the first and last order for sky subtraction.
This is useful for repeating sky subtraction in an order or two
with improved parameters.
SKLFIL= ' ' :: Input file for fine tuning skyline subtraction with
the Bspline routine. Entires have this format:
ordr # wvmin wvmax flag
where the flag = 1 places breakpoints at the center of each row
and 2 places one at each end. One usually turns on /NOVAC with this
keyword
/NOVAC :: Does not apply a vacuum correction to the skyline info.
This probably would only be used with SKLFIL
Output is the sky subtracted 2D image appended to the unsubtracted
flux and the variance 2D images (e.g. 'Final/f_esi#.fits')
Example: IDL>
esi_echskysub, esi, obj_id
Example: IDL>
extrct_ech, esi, obj_id, /SKYSUB, [/SKY_FCHK]
Time : 10min per exposure
Examine the final product (and get wavelength info) by:
IDL>
xatv, 'Final/f_esi#.fits', getsky=2,
WVIMG='Arcs/ArcECH_slitIMG.fits'
Trace OBJ
esi_echtrcobj ::
This procedure carefully traces the science objects.
By default, it will combine multiple exposures to improve tracing.
It uses the Standard star as a guide and a crutch.
This new trace overwrites the one in the Obj structure.
Turn the keyword /FAINT on for faint objects. This step can (should)
be skipped if you have used esi_echcopyfnd for
the trace above.
Example: IDL>
esi_echtrcobj, esi, obj_id,
[exp], [/FAINT, /CHK, /USESTD, STDFIL=]
Example: IDL>
extrct_ech, esi, obj_id,
[exp], /TRCOBJ, [/USESTD, STDFIL=, /TRCCHK, /FAINT]
Time : fast
Extraction
esi_echextobj ::
This procedure performs a boxcar (default) or optimal
extraction of the object in each
order. It then rebins to a 1D spectrum using linear interpolation
of the wavelength solution. This is the only binning of the data
during the entire reduction scheme. All spectra are also shifted to
zero heliocentric velocity and placed on a wavelength array with
33000 pixels, w0 = 3900 Ang and constant velocity pixels at 10km/s.
All of the output is written to the Obj structure
Example: IDL>
esi_echextobj, esi, obj_id, [/OPTIMAL, /CHK]
Example: IDL>
extrct_ech, esi, obj_id, /EXTOBJ, [/OPTIMAL]
Time : 3min (box), 10min (optimal)
Examine the final product:
IDL>
esi_echspecplt
Combine Multiple Exposures (REQUIRED)
esi_echcombspec ::
This procedure adds up multiple exposures of the same object, order
by order, weighting by S/N after scaling each exposure to have the
same flux as the first. It appropriately deals with CR's. For objects
with more than 2 exposures, it clips 5 sigma outliers as well. Even
if you have only a single exposure, you need to run this routine prior
to fluxing and collapsing to a 1D spectrum as it creates the final
spectrum structure that is necessary for fluxing and coadding to 1D.
Output is a fits file in 'FSpec' named 'Obja_ech.fits' where
Obj is the name in the Obj tag and
the letter 'a' signifies the primary science object. In the future,
I might allow multiple objects to be extracted and would label
them 'b', 'c'..
Example: IDL>
esi_echcombspec, esi, obj_id
Example: IDL>
extrct_ech, esi, obj_id, /COMBOBJ
Time : fast
Flux the Spectra
esi_echflux ::
This procedure fluxes each order of the combined frame. Generally,
I'd suggest using the flux calibration that is defaulted
(i.e. ECH_FLUX#.fits in CALIBS).
This program overwrites the fx array in the file created by
esi_echcombspec.
Example: IDL>
esi_echfluxfin, esi, obj_id
Example: IDL>
extrct_ech, esi, obj_id, /FLUX, [/CLOBBER]
Time : fast
Examine the final product:
IDL>
esi_echspecplt, /fspec
Collapse to 1D
esi_echcoaddfin ::
This procedure coadds the various orders into a 1D spectrum. It does
weight by S/N but is otherwise rather simple minded. This includes the
fact that various regions of the spectra are ignored, especially data
in the lower left hand of the CCD.
Output is two fits files in 'FSpec' with names 'Obja_F.fits' and
'Obja_E.fits'.
Example: IDL>
esi_echcoaddfin, esi, obj_id
Example: IDL>
extrct_ech, esi, obj_id, /COADD
Time : fast
Examine the final product:
IDL>
x_specplot, 'FSpec/Name_F.fits',
'FSpec/Name_E.fits'
0. LWD Reduction
The following routines setup and then proceed through the
reduction of data taken in the LWD mode with ESI.
I suggest you use a .pro files like
proc_lwd.pro,extrct_lwd.pro
to guide the process.
Most of the following routines take the esi structure,
and the slit size or the obj_id.
Create Bias (Zero) frame
We have found that creating a Zero frame is preferred
over an overscan subtraction. If you have taken Bias (0s) frames,
the process is straightforward. If you haven't, you can copy the image
'BiasbinS.fits' from $XIDL_DIR/ESI/CALIBS
into the 'Bias/' directory.
Example: IDL>
esi_mkzero, esi
Example: IDL>
proc_lwd, esi, /MKBIAS
Time : 3-5min
This routine examines the esi structure and looks for calibration
files associated with the various slit widths of your LWD files.
It groups together exposures with identical Obj name and ESI mode and
sets the obj_id tags accordingly.
A summary of the LWD exposures is put in 'esiLWD_summ.txt'.
Example: IDL>
esi_lwdsetup, esi
Example: IDL>
proc_lwd, esi, /SETUP
Time : Fast
Arc Image
This is the most crucial step. The sky subtraction
algorithm will only succeed with a good Arc Image. It should be
emphasized that we only create one Arc Image per slit size and apply
this to all of the exposures taken with that slit size. The following
routines are well automated but should be checked carefully if you are
doing a 'final' reduction. I think it is possible that one could use
an Arc image from a previous run but wouldn't recommend it just
yet. All of the following routines require a specific slit size.
For LWD mode, only the HgNe and Xe lamps are useful. Therefore
we focus on these images (i.e. arclamp = 6).
esi_lwdmkarc
:: Process the Arcs. This step bias subtracts,
trims and combines all of the Arc images.
Note that we find it is best not to flat field the Arc, at least not
with an unnormalized arc. Output is: 'Arcs/ArcLWD_slit.fits'
Example: IDL>
esi_lwdmkarc, esi, slit
Time : 2min
esi_lwdfitarc
:: This routine extracts a 1D spectrum along
row 820 (default) of Arc image and determines the wavelength solution.
Currently, I recommend using the /INTER mode although you can
try and let the code run automatically.
As long as your arc lines are within 3 pixels of my
solution, things ought to run smoothly in AUTO mode.
The output is an IDL file containing the polynomial fits for each
order: 'Arcs/AFIT_LWDslit.fits'. The program also creates
a ps file to examine the quality of fits.
Example: IDL>
esi_lwdfitarc, esi, slit, [/INTER]
Time : 2min
esi_lwdtrcarc
:: This routine traces arc lines throughout
the image. It will parse out the ends of traces where the algorithm
tends to have trouble.
Output is an anonymous structure written to the binary fits
file 'Arcs/ATRC_LWDslit.fits'.
Example: IDL>
esi_lwdtrcarc, esi, slit
Time : 2min
esi_lwdmkaimg
:: This routine takes the arc traces and
uses them to create a full Arc Image in the data frame. The main
routine performs a 2D polynomial surface fit to the traces from
the previous step. This is very expensive but needs to be only
run once per night per slit width.
Output is : 'Arcs/ArcECH_slitIMG.fits'
Example: IDL>
esi_lwdmkaimg, esi, slit, [/CHK]
Time : 20min
Check output with: xatv, 'Arcs/ArcECH_slitIMG.fits'
To run all three steps at once, use:
IDL>
proc_lwd, esi, slit, /MKAIMG, [/INTER]
Process Flats
esi_lwdmkflat ::
This routine processes the QTZ calibration frames, producing
a normalized flat useful to correcting pixel-to-pixel variations.
The routine bias subtracts and combines individual exposures.
It then fits a high order bspline to a strip of the flat in order
to normalize to unity.
Output is 'Flats/FlatLWD_slit.fits'
Example: IDL>
esi_lwdmkflat, esi, slit
Example: IDL>
proc_lwd, esi, slit, /MKFLAT
Time : 5-10min
Check the Flat: xatv, 'Flats/FlatLWD_slit.fits'
Extract
esi_lwdproc ::
This routine bias subtracts and flattens the science frame.
Output is 'Final/f_esi####.fits'
Example: IDL>
esi_lwdproc, esi, obj_id, [/CLOBBER]
Example: IDL>
ext_lwd, esi, obj_id, /PROCOBJ
Time : 1min/image
Check: xatv, 'Final/f_esi####.fits'
esi_lwdproc ::
This routine finds the science object and any other serendips in the
slit. I highly recommend you use the INTER mode. With this, you
need to set the aperture size around the obj for sky subtraction
(not extraction!). The routine also creates the Obj structure in Extract.
Output is 'Extract/Obj_esi####.fits'
Example: IDL>
esi_lwdfndobj, esi, obj_id, [/CLOBBER]
Example: IDL>
ext_lwd, esi, obj_id, /FNDOBJ
Time: fast