The Canadian Astronomy Data Centre 
Standard Pipeline ProcessingAt the JCMT, the ORAC-DR pipeline is used to calibrate raw data and combine single observations into larger ones with more areal coverage and/or deeper sensitivity. The system is still under active development. This page describes the standard products that will be generated and stored at the CADC in the fully operational system, but with the more advanced features grayed-out until they are available. This discussion applies to both ACSIS and SCUBA-2 data, with the differences between the two systems noted in the text. Grouping AlgorithmsThere are numerous ways to group observations together. Standard processing should create the kinds of groups listed in Table 1.
The GRPTYPE (association type) column shows the character string that will be used to build file names, as discussed below. ProductsFor each association, the pipeline will produce a set of standard data products. Simplifying the process a bit, the products can be divided in "basic" data products that can be derived directly from the raw observations, and advanced data products that are derived from the basic data products. The products discussed in this section all contain calibrated data and will be stored as FITS files with complete WCS information. ACSIS maps can be very large and individual datacube files are limited in size to 512 MB or less for user convenience, so large maps will be broken into disjoint datacubes that tile the region in space and frequency. The character string in the column PROD in the following tables will be used to build file names, as discussed below, and is divided into a leading alphabetic string and a tile number that is a fixed-length decimal number with the illustrated number of digits. SCUBA-2 products are intrinsically much smaller than ACSIS products, because they only have one "channel" on the frequency axis. In principle they do not need to be tiled, but for consistency with ACSIS products will also be tiled with a single tile for each map. Thus, the numerical part of PROD for SCUBA-2 products all has the tile number '001'. The basic data products are listed in Table 2.
"Cube" files are minimally processed and are provided as an alternative form of raw ACSIS data for the benefit of those who do not wish to use Starlink software for data reduction. They are are produced only for individual observations. The choice of raw (sdf) or cube files can be made using radio buttons on the "Search for ACSIS Raw Data" pages (both public and proprietary). "Reduced" files contain fully reduced datacubes and will be generated for both ACSIS and SCUBA-2 data. For ACSIS data, the reduced files will have had baselines and the noisy ends of the spectra removed. The detailed algorithms to generate "fully reduced" datacubes are still under development. The "cont" files will be produced for ACSIS data taken in continuum mode and will show the integrated continuum flux outside the lines in the datacube. The detailed algorithm that will be used to generate the continuum images is still under development. "Reduced" products (and "cube" products for ACSIS) will be generated for every observation to allow the observing team to assess the quality of the data. Observations will only be combined into larger groups if they are marked as "good" in the OMP database tables maintained at the JAC. Authorized users can check the quality of each observation in their project by the colour-code of the entry for each observation in the nightly observation logs of the OMP interface at the JAC, and can request changes by adding shift comments through that same interface. Other data products that will be derived from reduced datacubes for ACSIS data are listed in Table 3.
As with the basic data products, detailed algorithms are still under development for all these products. For example, the algorithm to generate the "max" product may need to filter out noise spikes. The FITS HISTORY keywords in each file will record its detailed processing history. VersionsThe standard pipeline will always generate products with VERSION=000. If these products are regenerated because of newly arrived data or a change in the data reduction algorithm, the new products will overwrite the old products and will retain VERSION=000. Because of this policy, the contents of these files can change without warning. Anyone using the standard pipeline products for scientific analysis are advised to download and retain their own copy of the data. It is intended that in the future it will be possible for authorized users to upload products that will be assigned non-zero version numbers. These files should be stable so that new copies are assigned new version numbers. File Names
File names produced by the standard pipeline for ACSIS will have the
form
For example, a typical reduced cube for a single ACSIS observation would have a
filename like:
An integrated
intensity image for a line in the cube produced as a night product
from observations 30, 35, 40 and 41 on the night of 2007-01-23 would
have a name like:
Thumbnails
Every reduced product should have an associated set of "thumbnail" images
that can be used to preview the data. The thumbnail images are stored as PNG files
and will be displayed on the "Results" page for each query. The thumbnail file names
are generated from the reduced file name by removing the tile number (thumbnails are
never tiled) and appending a vertical size in pixels (_64, _256, or _1024). Thus,
the reduced file in the previous section will have thumbnail files with names like:
For ACSIS, preview images are always generated from the representative spectrum (rsp) discussed in the section "Products" above. If a representative image was also generated, the thumbnail will both the image and the spectrum on the left and right sides of the image respectively. For SCUBA-2, only the image thumbnail will be generated. |
