NAME

USAGE

DESCRIPTION

sxipipeline duplicates most of the pipeline (not trend data) for the SXI. It allows the user to run all or part of the pipeline processing and to vary the calibration files and filtering (screening) criteria used. A number of other pipeline processing parameters can also be changed.

SXI Pipeline Stages

The sxipipeline is divided into 3 stages:

1. Calibration
2. Data screening
3. Product creation

SXI Stage 1 consists of the following steps:

For each event file:
1. (optional) Run coordevt on hot pixel file
2. Run coordevt
3. Run sxiphas
4. Run sxiflagpix
5. Run sxipi
6. Run sxipi
   1. Create flickering pixel file
      1. Filter SXI events STATUS[1]==b0
      2. Run searchflickpix
      3. Run coordevt on flickering pixel events
7. Run sxiflagpix with flickering pixels

The data screening (Stage 2) is identical to that in the production pipeline, when default parameters are used. For details on the default screening applied to the SXI events (respectively), see:

• ahfilter - Create the EHK from attitude & orbital data and create the MKF from housekeeping data based on the CALDB mkfconf file
• ahgtigen - Create the GTI from the EHK and MKF parameters based on CALDB selection file
• ahscreen - Screen the data based on GTI and CALDB selection file

Default GTI used for screening data are:

• GTIPOINT
• GTITEL
• GTI_<DATACLASS>
• GTIEHK
• GTIMKF

Optionally, sximodegti can be run to create new GTI specific to the DATACLASSes of the input SXI event file.

The product creation (Stage 3) is identical to that in the production pipeline, when default parameters are used. For SXI events extractor is run on SKY coordinates and a lightcurve, spectra and images are created for each cleaned event file. No gif images are created.

INPUT

The input to sxipipeline is specified using (at minimum) the indir parameter. This should be specified as the sxi event_uf level sequence directory, e.g.:

sxipipeline indir=/path/to/100039010/sxi/event_uf

Paths to specific sxi housekeeping and satellite data files can be specified using the attitude, housekeeping, extended_housekeeping, makefilter, orbit and obsgti parameters. The attiude, orbit and sxi housekeeping files are required for stage 1 calibration.

OUTPUT

Filenames, etc.

The number of output files depends on both pipeline processing stage(s) and the options selected. All output files are written to the directory specified by the outdir parameter. The archive directory structure is NOT reproduced (i.e. all output files will be in a single directory).

The names of files produced are the same as those found in the HEASARC archive. However the usual "ahXXXXXXXXX" prefix, where "XXXXXXXXX" is the sequence number, can be replaced by a character string set by the stemoutputs parameter. This defaults to the value set by the steminputs parameter.

PARAMETERS

indir [string]

Directory containing the input data. This should be the path to the event_uf (or event_cl for stage_3 processing)

    /path/to/805062010/sxi/event_uf

outdir [string]

Output directory used for all output data, as well as the report file.

CAUTION: If clobber is set, and this directory already exists, then this task will overwrite files as needed in this directory.

steminputs [string]

Stem for FITS input files, e.g. ah100039010. This string is used as the base filename for finding input files. For example, if steminputs=ah100039010, then to find the attitude file, ahpipeline matches the regular expression /ah100039010sxi_[ps][\d][0-9a-f]{8}_uf\.evt(\..+)?$/ against all files found in the indir directory.

(stemoutputs = DEFAULT) [string]

Base (stem) output name used for creating output files. If set to "DEFAULT", then steminputs will be used.

entry_stage = 1 [1|2|3]

Entry stage, 1 or 2.

Stage 1: Re-calibrate unfiltered event files.

Stage 2: Start from existing unfiltered event files.

exit_stage = 2 [1|2|3]

Exit stage, 1 or 2.

Stage 1: Produces calibrated unfiltered event files.

Stage 2: Produces screened event files.

attitude [string]

Attitude file

(extended_housekeeping = ah1001.ehk) [string]

Extended housekeeping file

(makefilter = ah1001.mkf) [string]

Makefilter file

orbit [string]

Orbit file

gtifile [string]

Input GTI file

housekeeping [string]

SXI Housekeeping file

(regionfile = NONE) [string]

Input region file

(sxi_mkflabel = SXI#SCI) [string]

Label to use for SXI MKF GTI creation. The hash will be replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_ehklabel = SXI#SCI) [string]

Label to use for SXI EHK GTI creation. The hash will be replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_evtlabel = SXI#SCI) [string]

Label to use for SXI event screening. The hash will be replaced by a respective mode: CCDW1, CCD12WA or CCD34WF

(sxi_start = 0.0) [real]

SXI CALDB start time

(ra = -999.99999) [real]

RA of nominal pointing [deg]

(dec = -999.99999) [real]

Dec of nominal pointing [deg]

(roll = 0.0) [real]

Roll of nominal pointing [deg]

(optdetx = -999.99999) [real]

SXI optical detx coordinate

(optdety = -999.99999) [real]

SXI optical dety coordinate

(optfocx = -999.99999) [real]

SXI optical focx coordinate

(optfocy = -999.99999) [real]

SXI optical focy coordinate

(optskyx = -999.99999) [real]

SXI optical skyx coordinate

(optskyy = -999.99999) [real]

SXI optical skyy coordinate

(ra_pnt = -999.99999) [real]

RA of sxi pointing [deg]

(dec_pnt = -999.99999) [real]

DEC of sxi pointing [deg]

(calc_hotpix = no) [bool]

Run coordevt on hot pixel file (yes/[no])

(calc_modegti = yes) [bool]

Calculate SXI data mode GTI ([yes]/no)

(teldeffile = CALDB) [string]

Input teldef file (or CALDB)

(leapsecfile = REFDATA) [file]

ahscreen: Input leap second file (or CALDB, [REFDATA])

(selectfile = CALDB) [file]

ahscreen: Input file with the selection expressions

(badpixfile = CALDB) [file]

sxiflagpix: badpixfile

(maskfile = CALDB) [file]

sxiflagpix: maskfile

(vtevnoddfile = CALDB) [file]

sxipi: evenodd

(ctifile = CALDB) [file]

sxipi: cti

(chtrailfile = CALDB) [file]

sxipi: chtrail

(spthfile = CALDB) [file]

sxipi: splitth

(gainfile = CALDB) [file]

sxipi: gain

(patternfile = CALDB) [file]

sxipi: grade

(dattfile = datt.out) [string]

output datt file with drift corrections

(coordevt_startsys = LOWEST) [string]

Starting coordinate system

(stopsys = HIGHEST) [string]

Final coordinate system

(annaber = no) [string]

Apply annual aberration correction (yes, [no], INVERT)

(followsun = no) [bool]

Recalculate the Sun position for each event (yes, [no])

(orbaber = no) [string]

Apply sat orbital aberration correction (yes, [no], INVERT)

(attinterp = LINEAR) [string]

Sky attitude interpolation method (LINEAR, CONSTANT)

(dattinterp = LINEAR) [string]

Delta attitude interpolation method (LINEAR, CONSTANT)

(attdt = 32.) [real]

Allowed margin for time extrapolation in attfile [s]

(dattdt = 0.5) [real]

Allowed margin for time extrapolation in dattfile [s]

(chkattgap = no) [bool]

Limit attitude interpolation if gaps present (yes, [no])

(chkdattgap = yes) [bool]

Limit delta attitude interpolation if gaps present ([yes], no)

(attext = ATTITUDE) [string]

Attitude extension

(attcol = QPARAM) [string]

Attitude column

(attform = QUAT) [string]

Attitude format ([QUAT], EULER)

(orbext = ORBIT) [string]

Orbit extension

(orbcol = VELOCITY) [string]

Orbital velocity column

(orbform = VECTOR) [string]

Orbital velocity format ([VECTOR], COMPONENTS, KEPLERIAN)

(coordevt_randomize = TELDEF) [string]

Randomize coordinates when rebinning ([TELDEF], yes, no)

(randsys = TELDEF) [string]

Starting system for randomization (or TELDEF)

(randscalesys = TELDEF) [string]

System to determine randomization amount (or TELDEF)

(infileext = EVENTS) [string]

Event extension

(inclfloatcol = no) [bool]

Write non-rounded coordinate columns (yes, [no])

(inclfloatskycol = no) [bool]

Write non-rounded sky coordinate columns (yes, [no])

(floatcolsuffix = _FLOAT) [string]

Suffix for non-rounded coordinate columns

(startwithfloat = no) [bool]

Start with non-rounded startsys coordinates (yes, [no])

(blankcol = yes) [bool]

Assign null values to columns not calculated ([yes], no)

(btnull = 255) [int]

TNULL for byte (B) columns

(itnull = -999) [int]

TNULL for short (I) columns

(jtnull = -999) [int]

TNULL for long (J) columns

(ktnull = -999) [int]

TNULL for long long (K) columns

(sbtnull = 255) [int]

TNULL for signed byte columns

(uitnull = -999) [int]

TNULL for unsigned short columns

(ujtnull = -999) [int]

TNULL for unsigned long columns

(colbound = -32768) [string]

TNULL, TLMIN, TLMAX for PHAS

(chipcol = CCD_ID) [string]

Chip column (or NONE)

(xcol = ACTX) [string]

X coordinate column

(ycol = ACTY) [string]

Y coordinate column

(chancol = PI) [string]

Pulse height column (or NONE)

(gradecol = GRADE) [string]

Event grade column (or NONE)

(grade = 0) [string]

Event grade for clean (or ALL)

(n_division = 1) [int]

Divide total observation time into the given number

(cleanimg = no) [bool]

Output cleaned image for debugging (yes, no)

(cellsize = 7) [int]

Poisson clean cell size (odd integer > 1)

(impfac = 320) [double]

Factor for gamma function

(logprob1 = -5.6) [double]

Log Poisson probability threshold

(logprob2 = -5.6) [double]

Log Poisson probability threshold for second step

(iterate = yes) [bool]

Iterate the second step Poisson clean (yes, no)

(flagedge = no) [bool]

Zero chip edge pixels (yes, no)

(bthresh = 3) [int]

Zero background threshold

(duration = no) [bool]

Perform detailed search for flickering duration (yes, no)

(sigma = 3.0) [double]

Significance level for flickering duration

(firstchip = TLMIN) [string]

Min value for chip number

(lastchip = TLMAX) [string]

Max value for chip number

(xmin = TLMIN) [string]

Min value for X coordinate

(xmax = TLMAX) [string]

Max value for X coordinate

(ymin = TLMIN) [string]

Min value for Y coordinate

(ymax = TLMAX) [string]

Max value for Y coordinate

(chanmin = TLMIN) [string]

Min pulse-height value for clean (inclusive)

(chanmax = TLMAX) [string]

Max pulse-height value for clean (inclusive)

(outbadpix = no) [boolean]

Output bad pixel file (yes/[no]). This parameter is not a boolean in sxiflagpix but rather a filename. For ahpipeline and sxipipeline this is boolean to account for multiple files

(outbadimg = yes) [boolean]

Output bad pixel image ([yes]/no). This parameter is not a boolean in sxiflagpix but rather a filename. For ahpipeline and sxipipeline this is boolean to account for multiple files

(npixnbr = 1) [int]

Pixel distance defining a neighbor

(nboundnbr = 1) [int]

Pixel distance defining neighbhor from CCD/window/segment boundary

(citrailnbr = 2) [int]

Pixel distance trailing CI row

(ciprenbr = 1) [int]

Pixel distance preceding CI row

(bad_status = 3:9,11,12,16:19,25:28,30) [string]

Bad status list, colons can used to specify a range (eg. 1:3,5 = 1,2,3,5)

(copyphas = yes) [bool]

Copy original PHAS before processing ([yes]/no)

(resetflags = yes) [bool]

Reset all sxiflagpix STATUS flags ([yes]/no)

(hkext = HK_SXI_USR_USER_HK1) [string]

HK extension with video temperatures

(hkcolstem = SXI_USR_HKTBL_) [string]

Column name stem for video temperatures

(hkvideoid = A,B,B,B) [string]

Video card ID for gain correction of CCD1-4

(startcol = PHAS) [string]

Starting point of correction

(evnoddcor = yes) [bool]

Enable even-odd correction [yes/no]

(chtrailcor = yes) [bool]

Enable charge trail correction [yes/no]

(cticor = yes) [bool]

Enable CTI correction [yes/no]

(gaincor = yes) [bool]

Enable gain correction [yes/no]

(ctigrade = no) [bool]

Use grade information in CTI correction [yes/no]

(copygrade = no) [bool]

Copy existing GRADE and PHA columns [yes/no]

(phcut = CALDB) [string]

Pulseheight cut for CTI correction, or CALDB

(badpixopt = 2) [int]

Options for events with bad pixels: ignore bad pixels (1), null bad pixels (2), null whole event (3)

(spthiter = yes) [bool]

Enable split threshold iteration [yes/no]

(spthcaldb = yes) [bool]

Use split thresholds from spthfile [yes/no]

(spthoffset = 15.) [real]

Split threshold offset value (if spthcaldb = no)

(spthslope = 0.) [real]

Split threshold slope value (if spthcaldb = no)

(evtthre = DEFAULT) [string]

Event threshold (or DEFAULT)

(negthre = -5000) [int]

Minimum PHAS value for normal event

(deltatime = 8) [int]

Max allowed time gap in HK temp search [s]

(debugcol = no) [bool]

Write out the debug columns [yes/no]

(randomize = yes) [bool]

Allow randomization in PI to UPI conversion (yes, no)

(timecol = TIME) [string]

Time column

(seed = 0) [int]

Random number generator seed (0=use system time)

(stemreport = ) [string]

File stem for log and temporary files. If the parameter is not set the script will automatically set the stem to "sxipipeline_YYYYMMDDTHHMMSS_" and will append log file and temp file names as needed. Intended to be set by ahpipeline.

(numerrs = 0) [string]

Number of errors from sxipipeline (output)

(cleanup = yes) [bool]

Delete temporary files ([yes]/no)

(clobber = no) [boolean]

Overwrites the existing output file if set to yes (yes/[no]).

(chatter = 2) [integer]

Chatter level for output. Set to 0 to suppress output, or to 1, 2, or 3 for increasing the chatter of the output.

(logfile = !DEFAULT) [string]

Log filename. If set to DEFAULT uses the name of the task and, if preceded by '!', overwrite the file if it exists. If set to NONE no log file is created.

(debug = no) [boolean]

Diagnostic output is printed out on the screen if set to yes (yes/[no]).

(history = yes) [boolean]

Records tool parameters in HISTORY ([yes]/no).

(mode = ql) [string ql|hl|q]

Mode to query the parameter file. Acceptable values include: 'ql' (query and learn/remember), 'hl' (hidden and learn/remember), 'q' (query but don't remember)\ , 'h' (hidden).

EXAMPLES

1. The following command will recalibrate (stage 1) and re-screen (stage 2) all SXI data for sequence 100039010 that currently resides in the directory /data/100039010/sxi/event_uf, and the output will be stored in a directory called /data/100039010_reproc/:

   sxipipeline indir=/data/100039010/sxi/event_uf outdir=/data/100039010_reproc entry_stage=1 exit_stage=2 steminputs=ah100039010 attitude=/data/100039010/auxil/ah100039010/ah100039010.att orbit=/data/100039010/auxil/ah100039010/ah100039010.orb obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti housekeeping=/data/100039010/sxi/hk/ah100039010sxi_a0.hk makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

2. The following command will re-screen (stage 2 only) SXI data for the same data set as in the previous example, as well as recalculate the SXI mode GTI:

   sxipipeline indir=/data/100039010/sxi/event_uf outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 calc_modegti=yes obsgti=/data/100039010/auxil/ah100039010/ah100039010_gen.gti makefilter=/data/100039010/auxil/ah100039010/ah100039010.mkf exended_housekeeping=/data/100039010/auxil/ah100039010/ah100039010.ehk

3. The following command will create products (stage 3 only) SXI data for a calibrated data set:

   sxipipeline indir=/data/100039010/sxi/event_cl outdir=/data/100039010_reproc entry_stage=2 exit_stage=2 steminputs=ah100039010 regionfile=none

NOTES

None, but see help for individual parameters above.

SEE ALSO

LAST MODIFIED

February 2016