NAME

sgdpipeline - SGD reprocessing tool

USAGE

sgdpipeline indir outdir steminputs entry_stage exit_stage

DESCRIPTION

sgdpipeline duplicates most of the pipeline (not trend data) for the sgd. It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

SGD Pipeline Stages

The sgdpipeline is divided into 3 stages:

  1. Calibration
  2. Data screening
  3. Product creation
Each stage may be run singly or in combination with the preceeding stages. This is controlled by the entry_stage and exit_stage parameters.

SGD Stage 1 consists of the following steps:

  1. Run hxisgdsff
  2. Run hxisgdpha
  3. Run sgdevtid

The data screening (Stage 2) is identical to that in the production pipeline, when default parameters are used. For details on the default screening applied to the SGD events (respectively), see:

Default GTI used for screening data are:

The cleaning process occurs twice for SGD. Once to produce cleaned event files, the second to produce pseudo event files.

The product creation (Stage 3) is identical to that in the production pipeline, when default parameters are used. Extractor is run to create a lightcurve and spectra for each cleaned event file.

INPUT

The input to sgdpipeline is specified using (at minimum) the indir parameter. This should be specified as the sgd event_uf level sequence directory, e.g.:

sgdpipeline indir=/path/to/100039010/sgd/event_uf

Paths to specific sgd housekeeping and satellite data files can be specified using the attitude, housekeeping, extended_housekeeping, makefilter, orbit and obsgti parameters. The attiude, orbit and sgd housekeeping files are required for stage 1 calibration.

OUTPUT

Filenames, etc.

The number of output files depends on both pipeline processing stage(s) and the options selected. All output files are written to the directory specified by the outdir parameter. The archive directory structure is NOT reproduced (i.e. all output files will be in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However the usual "ahXXXXXXXXX" prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the stemoutputs parameter. This defaults to the value set by the steminputs parameter.

PARAMETERS

indir [string]
sgdpipeline:
Input directory

outdir [string]
sgdpipeline:
Output directory

steminputs [string]
sgdpipeline:
stem inputs

stemoutputs [string]
sgdpipeline:
stem outputs

instrument [string]
sgdpipeline:
Instrument (SGD,SGD1,SGD2)

entry_stage [integer]
sgdpipeline:
Entry stage

(exit_stage = 2) [integer]
sgdpipeline:
Exit stage

(sgd_start = 0.0) [real]
sgdpipeline:
SGD CALDB start time

verify_input [boolean]
sgdpipeline:
Verify with ftverify (yes, no)

(sgd_mkflabel = SGDSFFA1#) [string]
Label to use for SGD MKF GTI creation. The hash will be replaced by a respective compton camera: CC1, CC2 or CC3
For pseudo events "PSE" will be appended to the end of the label

(sgd_ehklabel = SGDSFFA1#) [string]
Label to use for SGD EHK GTI creation. The hash will be replaced by a respective compton camera: CC1, CC2 or CC3
For pseudo events "PSE" will be appended to the end of the label

(sgd_evtlabel = SGDSFFA1#) [string]
Label to use for SGD event screening. The hash will be replaced by a respective compton camera: CC1, CC2 or CC3
For pseudo events "PSE" will be appended to the end of the label

(ra = -999.99999) [real]
The Right Ascension in decimal degrees of the point to appear in the center of SKY coordinate images. If ra is within the range 0 <= ra <= 360 deg., then the value of the parameter is used. If ra is outside this range, then the ra value is read from the event header of the input file, searching first for the RA_NOM keyword and if found using its value, or if not found, then searching for the RA_PNT keyword and using its value. If neither keyword is found, then coordevt exits with an error. The default value for ra triggers the keyword look up.

(dec = -999.99999) [real]
The declination in decimal degrees of the point to appear in the center of SKY coordinate images. If dec is within the range -90 <= dec <= +90 deg., then the value of the parameter is used. If dec is outside this range, then the dec value is read from the event header of the input file, searching first for the DEC_NOM keyword and if found using its value, or if not found, then searching for the DEC_PNT keyword and using its value. If neither keyword is found, then coordevt exits with an error. The default value for dec triggers the keyword look up.

(roll = 0.0) [real]
The roll angle about the center of the SKY coordinate system in decimal degrees. The roll angle is the angle measured counterclockwise from Celestial North to the positive SKY Y axis.

(sg1_optfocx = -999.99999) [real]
SGD1 optical focx coordinate

(sg1_optfocy = -999.99999) [real]
SGD1 optical focy coordinate

(sg1_optskyx = -999.99999) [real]
SGD1 optical detx coordinate

(sg1_optskyy = -999.99999) [real]
SGD1 optical dety coordinate

(sg2_optfocx = -999.99999) [real]
SGD2 optical focx coordinate

(sg2_optfocy = -999.99999) [real]
SGD2 optical focy coordinate

(sg2_optskyx = -999.99999) [real]
SGD2 optical detx coordinate

(sg2_optskyy = -999.99999) [real]
SGD2 optical dety coordinate

(sg1_ra_pnt = -999.99999) [real]
RA of SGD1 pointing [deg]

(sg1_dec_pnt = -999.99999) [real]
DEC of SGD1 pointing [deg]

(sg2_ra_pnt = -999.99999) [real]
RA of SGD2 pointing [deg]

(sg2_dec_pnt = -999.99999) [real]
DEC of SGD2 pointing [deg]

(extended_housekeeping = ah1001.ehk) [string]
ahgtigen:
Extended housekeeping file

(makefilter = ah1001.mkf) [string]
ahgtigen:
Makefilter file

(obsgti = ah1001_gen.gti) [string]
ahscreen:
Observation GTI file

(remapfile = CALDB) [filename]
hxisgdsff/sgdevtid:
File describing the remapping of ASIC_ID and READOUT_ID to sequential numbers in the columns ASIC_ID_RMAP and READOUT_ID_RMAP. Specify CALDB to retrieve the file from the calibration database.

(gainfile = CALDB) [filename]
hxisgdpha:
FITS table describing the gain calibration of PHA for the HXI or SGD camera. The domain of PHA values is divided into intervals, and a set of cubic polynomial coefficients is given for each interval. Specifying CALDB causes this file to be retrieved from the calibration database.

(badpixfile = CALDB) [filename]
hxisgdpha/sgdevtid:
FITS table giving the active and bad channels (pixels) of the HXI or SGD camera. Specifying CALDB causes this file to be retrieved from the calibration database.

(fluorefile = CALDB) [filename]
sgdevtid:
Calibration file containing the energy ranges of fluorescence electrons in CdTe and Si. Default: CALDB.

(probseqfile = CALDB) [filename]
sgdevtid:
Calibration file assigning the probability of each possible sequence of interactions.

(probfovfile = CALDB) [filename]
sgdevtid:
Calibration file containing the probability for photons to be in the SGD camera field of view, based on incident angle.

(leapsecfile = REFDATA) [filename]
ahgtigen/ahscreen:
Input leap second file (or CALDB, [REFDATA])

(selectfile = CALDB) [filename]
ahgtigen/ahscreen:
Input file with the selection expressions

(outnsubcol = no) [boolean]
hxisgdpha:
Output the PHA_NSUB column (yes/no). This column is the PHA minus the ASIC common mode noise (ASIC_CMN column in the SFF).

(datamode = NONE) [string]
hxisgdpha:
Substitute DATAMODE in place of event value (or NONE)

(rejectbgo = no) [boolean]
sgdevtid:
Reject events for which BGO trigger occurred (yes/[no]).

(skipreco = no) [boolean]
sgdevtid:
If yes, READALL and CALMODE occurrences will not be reconstructed. Default: NO.

(outtracefile = NONE) [filename]
sgdevtid:
Output reconstruction tracing file (or NONE)

(numsignal = 48) [integer]
sgdevtid:
Maximum number of signals to analyze. Reconstruction is not attempted on occurrences with more than this number of signals.

(d10 = 3.2) [real]
sgdevtid:
[mm] Orthogonal distance between two adjacent pixels in the same detector layer; used in identifying charge sharing events.

(d1a1a = 5.0) [real]
sgdevtid:
[mm] Diagonal distance between two adjacent pixels in same detector layer; used in identifying possible CdTe-CdTe fluorescence signals in a single layer.

(d1a1b = 5.0) [real]
sgdevtid:
[mm] Distance between two CdTe layers; used in identifying possible CdTe-CdTe fluorescence signals in different layers.

(d1a2 = 14.0) [real]
sgdevtid:
[mm] Distance between CdTe and Si layers; used in identifying possible Si-CdTe fluorescence signals.

(d1a3 = 5.0) [real]
sgdevtid:
(d1a3 = 5.0) [mm] Distance between Si and Si layers; used in identifying possible electron scattering signals.

(a = 3.0) [real]
sgdevtid:
Acceptance tolerance for F test in Step 2

(b = 3.0) [real]
sgdevtid:
Acceptance tolerance for G test in Step 2

(probaccept2 = 0.1) [real]
sgdevtid:
Probability threshold for M=2 acceptance in Step 3

(probaccept3 = 0.1) [real]
sgdevtid:
Probability threshold for M=3 acceptance in Step 3

(probaccept4 = 0.1) [real]
sgdevtid:
Probability threshold for M=4 acceptance in Step 3

(distz = 1000000.0) [real]
sgdevtid:
Very large distance of target object in Step 4 [mm]. The Z-direction is toward the sky, so this distance represents the distance to an object far away from the detector.

(paraoffset0 = 1.6) [real]
sgdevtid:
Offset parameter combined with G[k,0] for the figure-of-merit (FOM) discrimination in Step 4 above.

(paraoffset1 = 1.0) [real]
sgdevtid:
Offset parameter combined with G[k,1] for the figure-of-merit (FOM) discrimination in Step 4 above.

(paraoffset2 = 1.0) [real]
sgdevtid:
Offset parameter combined withG[k,2] for the figure-of-merit (FOM) discrimination in Step 4 above.

(weight0 = 1.0) [real]
sgdevtid:
Parameter used in weighting G[k,0] for the figure-of-merit (FOM) discrimination in Step 4 above.

(weight1 = 0.0) [real]
sgdevtid:
Parameter used in weighting G[k,1] for the figure-of-merit (FOM) discrimination in Step 4 above.

(weight2 = 0.0) [real]
sgdevtid:
Parameter used in weighting G[k,2] for the figure-of-merit (FOM) discrimination in Step 4 above.

(weight3 = 0.0) [real]
sgdevtid:
Parameter used in weighting Prob[k] for the figure-of-merit (FOM) discrimination in Step 4 above.

(delgmethod = ANALYTIC) [string]
sgdevtid:
Method to be used to calculate the error in cos(theta_G) in the test of consistency between geometric and kinematic scattering angles. The possible values are "analytic" and "corner."

(occurrenceid = -1) [integer]
Occurrence to process (if >0)

(randomize = yes) [boolean]
Allow randomization ([yes]/no)

(seed = 0) [integer]
Random number generator seed (0=use system time)

(stemreport = ) [string]
File stem for log and temporary files. If the parameter is not set the script will automatically set the stem to "sgdpipeline_YYYYMMDDTHHMMSS_" and will append log file and temp file names as needed. Intended to be set by ahpipeline.

(numerrs = 0) [string]
Number of errors from sgdpipeline (output)

(cleanup = yes) [boolean]
Delete temporary files ([yes]/no)

(clobber = no) [boolean]
Overwrites the existing output file if set to yes (yes/[no]).

(chatter = 1) [integer]
Chatter level for output. Set to 0 to suppress output, or to 1, 2, or 3 for increasing the chatter of the output.

(logfile = !DEFAULT) [string]
Log filename. If set to DEFAULT uses the name of the task and, if preceded by '!', overwrite the file if it exists. If set to NONE no log file is created.

(debug = no) [boolean]
Diagnostic output is printed out on the screen if set to yes (yes/[no]).

(history = yes) [boolean]
Records tool parameters in HISTORY ([yes]/no).

(mode = ql) [string ql|hl|q]
Mode to query the parameter file. Acceptable values include: 'ql' (query and learn/remember), 'hl' (hidden and learn/remember), 'q' (query but don't remember)\ , 'h' (hidden).

THE FOLLOWING PARAMETERS ARE SET BY THE SCRIPT INTERNALLY:

HXISGDSFF:

infile [filename]
Set based on the unfiltered event files found in indir directory during initialization.
Input First FITS File (FFF) from HXI or SGD.

outfile [filename]
Set to the same name as infile, but in the outdir directory.
Output Second FITS File (SFF). May be the same or a different file from infile. If outfile is the same file as infile, then clobbering must be enabled or the tool exits with an error. Clobbering is enabled by specifying clobber=yes (see below) or by prepending an exclamation point (!) to the output file name. Either of these methods will cause the file conversion to be done in-place by modifying the input file.

HXISGDPHA:

infile [filename]
Set based on the unfiltered event files found in indir directory during initialization and processed through hxisgdsff.
Input Second FITS File (SFF) from HXI or SGD.

outfile [filename]
Set to the same name as infile, but in the outdir directory.
Output Second FITS File (SFF). If it is the same as infile, then specifying clobber=yes (see below) will cause the file conversion to be done in-place by modifying the input file. Prepending an exclamation point (!) to the output file name has the same effect.

HXISGDEXPAND:

infile [filename]
Set based on the unfiltered event files found in indir directory during initialization and processed through hxisgdsff and hxisgdpha.
Input file of event data from an HXI instrument. This is the Second FITS File (SFF) after PHA calibration by the tool hxisgdpha.

outfile [filename]
Set to the same name as infile, but in the outdir directory and changing the extention from "_uf.evt" to "exp_ufa.evt".
Output event file containing results of event reconstruction.

HXIEVTID:

infile [filename]
Set based on the unfiltered event files found in indir directory during initialization and processed through hxisgdsff and hxisgdpha.
Input file of event data from an HXI instrument. This is the Second FITS File (SFF) after PHA calibration by the tool hxisgdpha.

outfile [filename]
Set to the same name as infile, but in the outdir directory and changing the extention from "_uf.evt" to "rec_ufa.evt".
Output event file containing results of event reconstruction.

AHGTIGEN:

mkffile [filename]
Set to the makefiler file parameter the first time ahgtigen runs.
Set to the extended_housekeeping file parameter the second time ahgtigen runs (unless the files being generated are pseudo event files).

outfile [filename]
Set to ${sgdfile_out} . gtimkf, when the input file is set to the makefiler file parameter.
Set to ${sgdfile_out} . gtiehk, when the input file is set to the extended_housekeeping file parameter.

(instrume = NONE) [string]
(SGD1 or SGD2) Read from the header of the file being processed.

(label = NONE) [string]
("SGDSFFA1" . ${detnam}) or ("SGDSFFA1" . ${detnam} . "PSE") depening on if the files being generated are "clean" or "pseudo" respectively.

AHSCREEN:

infile [string]
Set systematically to the elements in an array which is constructed from any reconstructed unfiltered event files produced in the calibration stage.
If the calibration stage is being skipped, this array is filled from searching the the input directory instead.

outfile [filename]
Set to the same name as infile, but in the outdir directory and changing the extention from "rec_ufa.evt" to "rec_cl.evt".

(gtifile) [string]
Set as an array of gtifiles with cooresponding extentions; pointing, tel, mkf, and (where applicable) ehk.

(upkeyword = YES) [boolean]
Update timing keywords from input file(s) in output file
Hardcoded as yes

(label = NONE) [string]
("SGDSFFA1" . ${detnam}) or ("SGDSFFA1" . ${detnam} . "PSE") depening on if the files being generated are "clean" or "pseudo" respectively.

EXAMPLES

  1. The following command will recalibrate (stage 1) and re-screen (stage 2) all SGD data for sequence 100039010 that currently resides in the directory /data/100039010/sgd/event_uf, and the output will be stored in a directory called /data/100039010_reproc/:

    sgdpipeline indir=/data/100039010/sgd/event_uf outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 steminputs=ah100039010 obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

  2. The following command will re-screen (stage 2 only) SGD data for the same data set as in the previous example.

    sgdpipeline indir=/data/100039010/sgd/event_uf outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

  3. The following command will create products (stage 3 only) SGD data for a calibrated data set:

    sgdpipeline indir=/data/100039010/sgd/event_cl outdir=/data/100039010_reproc entry_stage=3 exit_stage=3 steminputs=ah100039010

NOTES

None, but see help for individual parameters above.

SEE ALSO

ahcalctime ahpipeline hxisgdsff hxisgdpha sgdevtid ahfilter ahgtigen ahscreen

LAST MODIFIED

February 2016