Steve Allen
UCO/Lick Observatory
2003-Nov-21

Sometime shortly after the date 2002-10-09 the DEIMOS mosaic CCD readout software was modified to improve the displaying of images in real time. The change was to set the BLANK keyword in each of the FITS IMAGE extension HDUs. By setting the value of BLANK the real-time display ignored pixels which had not yet been read. Unfortunately, it is not correct to set BLANK in this fashion for the DEIMOS FITS images.

Complete technical details of the error are found in the attached appendix.

The effect of this error is that fully-conforming FITS applications are instructed to ignore any pixels which have a CCD data value of exactly 32768. Fortunately, in most exposures only a few pixels have exactly this data value.

Furthermore, correct handling of the BLANK keyword requires special operations by a FITS application, and for the sake of simplicity many FITS applications ignore the BLANK keyword. Any applications which do ignore BLANK will use all data values in the correct fashion.

At this time I know that ds9 and fv applications for viewing FITS images do handle the BLANK keyword. They indicate that such pixels have a value of NULL or NaN.

I do not know whether the IDL-based data reduction application used by the DEEP team handle the BLANK keyword. Some IRAF applications do pay attention to the BLANK keyword, and others do not.

The DEIMOS mosaic CCD readout software will be modified as soon as we can make arrangements with the DEIMOS instrument scientist. After that modification, all subsequent DEIMOS images will be correct when they are written to disk.

Archival FITS files from DEIMOS can be fixed quite simply. Inside each FITS file are multiple IMAGE extension HDUs.

If the BLANK keyword does not occur (and this will be true for images from before 2002-10-09) then the file is intact and needs no change.

If the BLANK keyword does occur and has a value

	BLANK = 0

then the HDUs are incorrect. They can be remedied by setting

	BLANK = -32768

If the BLANK keyword does occur and has value

	BLANK = -32768

then the HDU is correct.

---

Appendix: Technical Details

From: Steve Allen
To: FITSbits
Subject: [fitsbits] BLANK keyword misinterpretation
Date: Thu, 20 Nov 2003 16:22:14 -0800

We are currently testing of the new CCD mosaic which is being readied for shipment to Keck to replace the old detector in the HIRES spectrograph. The laboratory setup permits illumination patterns which are hard to duplicate in the instrument after deployment. In the course of examining the images we discovered that I had made a misinterpretation of the FITS standard.

I believe that the FITS standard is unambiguous. Nevertheless, slight changes to the wording of the standard could be very useful in preventing other such misinterpretations. Until it becomes possible to make such changes other implementors should be cautious when using the BLANK keyword. It is also possible that changes to the APIs that are used for writing FITS images would help.

Our data originate via a 16-bit AtoD conversion which produces bits corresponding to C "unsigned short" on modern big-endian processors. The pixel values are always offset such that all values are positive. We store the bits directly in the image arrays as unsigned short integers using the traditional trick of offsetting the physical data values using the keywords

	BITPIX   = 16
	BZERO    = 32768
	BSCALE   = 1

The complete HDU structure of the mosaic FITS files is created at the beginning of the CCD readout, and at this time the values in the data arrays of all IMAGE extensions are initialized to be all zero bits. During the readout the FITS file is stored in shared memory where it can be accessed by FITS viewer to provide real-time display of the incoming image. In our systems the pixels may not arrive in sequence, so the images being constructed may have "holes" during the readout.

The "holes" in the image during the readout confound the automatic contrast setting algorithm in the FITS viewer. However, the FITS viewer that we use understands the usage of the BLANK keyword. In order to instruct it ignore pixels that do not yet have data I modified the FITS headers to include the keyword

	BLANK    = 0

Whereas this had the intended effect during real-time readout, this was a mistake for the archival data.

FITS is defined by NOST 100-2.0 as printed here.

The interpretation of BITPIX is shown by Table 5.2 in section 5.4.1.1. I believe it would be helpful to many implementors if the text were slightly enhanced such that the entries

		16-bit twos-complement binary integer
		32-bit twos-complement binary integer

were changed to read

		16-bit twos-complement (signed) binary integer
		32-bit twos-complement (signed) binary integer

The definition of the data representation for BITPIX = 16 is given by section 6.2.2. It is completely unambiguous.

The definition of the BLANK keyword is given in section 5.4.2.5. Its relevant sentence reads:

    The value field shall contain an integer that specifies the
    representation of array values whose physical values are
    undefined.

This is unambiguous, but I misinterpreted it. I would have got the interpretation right if there had been one extra sentence akin to this:

    This number corresponds to the data array values before applying
    any transformation indicated by BSCALE and BZERO.

The problem is that with these values of BZERO and BSCALE the signed FITS array value of 0 corresponds to the unsigned CCD data value of 32768. The result of my misinterpretation is that images with a gentle ramp of CCD data values in the vicinity of 32768 have sprinkles of single pixels where a conforming FITS viewer flags the pixels as NULL or undefined.

In our software the fix for this is relatively straightforward. The image in shared memory during readout is currently used only for real-time display. There is a final pass over each HDU where keywords are modified before the image is written to disk for archiving. In that pass the software will reset the keywords such that

	BLANK    = -32768

This change will have no significant impact on real-time performance.

We are using CFITSIO as the API for creating these FITS images. This problem could also be alleviated if the API provided an explicitly defined means of indicating BLANK along with other parameters of the image.