POINTLESS (CCP4: Supported Program)

NAME

pointless

SYNOPSIS

pointless [-c[opy]] [[HKLIN] foo_in.mtz] [XDSIN foo_in.HKL] [SCAIN foo_in.HKL] [HKLREF reference_file.mtz] [XYZIN reference_coords.pdb] [HKLOUT output_file.mtz] [XMLOUT xml_output.xml]
[Keyworded Input]

References
Input and Output files
Examples
Release Notes

DESCRIPTION

Pointless scores all the possible Laue groups consistent with the crystal class, which is based on cell dimension restraints. It does this by matching potential symmetry equivalent reflections. For chiral systems, the Laue group uniquely implies the point group. It then checks sets of reflections which may be systematically absent to suggest a possible spacegroup. There is also a check for lattice centering, ie a check for whole classes of reflections having essentially zero intensity, including a check for obverse/inverse twinning in rhombohedral systems.
[Note: strictly speaking, the program determines the Patterson group rather than the Laue group, since the Laue group is a point group, not including any lattice centring type (P,C,I,F,H,R). Lattice centring is included in the reported Laue group, and reindexing from the original setting may change the lattice type.]

The principal (test) reflection input is taken from one or more unmerged MTZ files (HKLIN) such as those from Mosflm (or Combat), or alternatively unmerged XDS ASCII files (XDS_ASCII or INTEGRATE, XDSIN), unmerged Scalepack files (SCAIN) (produced with the Scalepack option "nomerge original index"),  SHELX files (also SCAIN) or SAINT files (also SCAIN). Also free-format files containing HKL I sigI (see SCAIN). Input types may be mixed. Merged files may be checked for under-merging & for systematic absences. Files given as HKLIN will be checked for their filetype, so other types will be recognised.

For some crystal classes and when there is more than one input file, it is necessary to establish first that a consistent indexing convention has been used for all runs (see http://www.ccp4.ac.uk/dist/html/reindexing.html for an explanation and list). Note that if there are multiple input files and no reference file, the first input file is used as a temporary reference for subsequent files, to check for consistent indexing in cases of ambiguity.This may not work correctly if this first file is indexed in the wrong Laue group: the keyword TESTFIRSTFILE may be used to force a Laue group check on the first file before adding in the others.

For the alternative indexing check, a reference file (HKLREF, merged or unmerged) or a reference coordinate list (XYZIN) may be given and the indexing of the test file will be checked against this. The space group of the reference data is assumed to be correct.

If there is more than one input file the program checks the batch numbering schemes do not overlap, and if necessary it renumbers batches.

Multilattice files from Mosflm/FECKLESS can be read: the different lattices are for now (pending refactoring of the data objects) are mapped into different runs, with BATCH numbers incremented by multiples of 1000 (in FECKLESS). By default only the first Run, corresponding to the first Lattice, is used in symmetry determination, but all runs are written to the output HKLOUT file. Other runs may be selected with the USE command.

If HKLOUT is assigned it produces a sorted reflection list (possibly after reindexing and renumbering batches) which is ready for input to AIMLESS or SCALA.

It has other subsidiary uses described in more detail below: it can convert other formats to MTZ, check and reindex an input file to match an existing reference set of amplitudes, or apply a pre-selected reindex operator. The command line option "-copy" (or "-c") may be used just to copy files to MTZ (equivalent to the COPY command).

Description of functions:

1. Determination of Laue group and space group (LAUEGROUP mode).

This mode is the default, if no HKLREF file is specified.

Given test data comprising unmerged observations (file(s) HKLIN, XDSIN or SCAIN), the program looks for possible symmetry based on the unit cell in order to determine the Laue group, ie the symmetry of the diffraction pattern. It then examines axial reflections (and zones for non-chiral crystals) to look for systematic absences to determine the space group.
 
If a file is assigned to HKLOUT, the reindexed file will be written out from the HKLIN file, using the selected reindexing. The hklout file is assigned to the "best" space group or point group (or the SPACEGROUP if given).

Warning Notes
:

• The optimum scoring method remains to be discovered. The true solution may not have the top score, but should be close to the top

• The program will find symmetry: there is no guarantee that this symmetry is crystallographic or correct! Caveat emptor.

2. Testing of alternative indexing schemes (INDEX mode)

This mode is selected if an HKLREF dataset or an XYZIN coordinate file is specified.

Given a test dataset, merged or unmerged (file(s) HKLIN), and a merged or unmerged reference dataset in a known space group (file HKLREF), the program tests all possible alternative indexing schemes of the test dataset to find which one best matches the reference set. Alternative indexing schemes arise in high symmetry space groups when the lattice symmetry is higher than the point group symmetry (eg for trigonal space groups), but may also arise in any space group from special relationships between cell parameters (eg an orthorhombic cell with a=b or more complicated examples involving cell diagonals).

If a file is assigned to HKLOUT, then the reindexed file will be written out from the HKLIN file, using the best reindexing. The hklout file is assigned to the spacegroup of the reference file.

3. Checking the centre of the diffraction pattern

It is occasionally (though rarely) possible to misindex the diffraction pattern by (say) +-1, usually on a long axis approximately parallel to the rotation axis. In this case, the centre of symmetry of the diffraction pattern is apparently not at hkl 0,0,0 as it should be. The CENTRE option does an R-factor search for the correct centre. Note that if the Laue group symmetry is wrong, this search may be less reliable: a LAUEGROUP command may be given to reset the symmetry. If the centre is wrong, the images should be re-integrated, as there will be other errors which cannot be easily corrected.

4. Reindexing

The program may also be used just to reindex or change the spacegroup, if both HKLIN and HKLOUT files are defined, and keywords SPACEGROUP and/or REINDEX are given. If the HKLIN file is a merged file, then any anomalous difference columns will be negated, and F+/- I+/- columns swapped, if the hand of the index is changed by the reduction process. Note that in this case there is no reliable check made that the reindexing operator is consistent with the space group: if a SPACEGROUP command is not given, the reindexed space group will be used if it is valid, otherwise you will have to specify the space group. Any reflections with non-integral indices after reindexing are discarded, so this may be used to select reflections corresponding to a sub-cell. Files containing phase columns can now be reindexed, as long as you are just changing the setting, not the space group (eg I2<->C2, P2 21 21 <-> P21 21 2 etc).

5. Converting XDS, Scalepack, SHELX or SAINT format to MTZ

The command line option "-copy" (or "-c") may be used just to convert an XDS, Scalepack or SHELX file to the MTZ format. The NAME keyword is required, and for SCALEPACK or SHELX input the CELL must be given. For SAINT input it is recommended to give a CELL (as it may be more accurate).

Choice of solution for the output file

A “solution” is a space group (or point group) and a reindexing operator. In INDEX mode, the solution is always chosen to give the best match to the reference file. In LAUEGROUP mode, the default is to choose the solution with the highest joint score in the Laue group search and the systematic absence test. If several non-enantiomorphic space groups have the same score (due to lack of information), then the corresponding point group is chosen (eg P222 for Laue group Pmmm). This automatic choice may be over-ridden in several ways, usually after examining the scores from an initial run.

  1. The command CHOOSE carries out the Laue group search and the systematic absence test, but the choice of solution is specified explicitly rather than chosen automatically

  1. CHOOSE SOLUTION <n>

    Choose a Laue group solution by number in the ranked list: the systematic absence test will then be done as normal

  2. CHOOSE LAUEGROUP <name>

    Choose a Laue group solution by name, choosing the first match in the ranked list: the systematic absence test will then be done as normal

  3. CHOOSE SPACEGROUP <name>

    Choose the corresponding Laue group solution by name, choosing the first match in the ranked list, then choose the specified space group in the systematic absence test list

  1. The command LAUEGROUP skips the Laue group search, and just uses the given Laue group, typically with a given REINDEX operator. The systematic absence test will then be done as normal.

  2. The command SPACEGROUP skips all searches, and just uses the given space group, typically with a given REINDEX operator. This is equivalent to the operation of the CCP4 program REINDEX.

Note that when a Laue group name is needed, either the actual Laue group name (as printed) may be given, or the name of any space group in that Laue group (eg P2/m may be given as P2)

Details of LAUEGROUP mode

  1. The intensity data are read from the HKLIN file(s) (or XDSIN or SCAIN). If multiple files are given, the first one is treated as a reference, and subsequent files are placed on the same indexing scheme in the same way as INDEX mode, if there are alternatives, using the symmetry from the first file (multiple files defined with wild-cards are assumed by default to have the same indexing, see ASSUMESAMEINDEXING). Note that the default is to assume that the first file has valid symmetry and indexing: the keyword TESTFIRSTFILE may be used to force a determination of the Laue group before adding subsequent files. Batch numbers are forced to be unique if necessary by incrementing them by multiples of 1000. In a "file series" defined by an HKLIN command with wild-cards in the file name, files which cannot be read (eg because they are are still being written) are ignored, and by default acceptance of files is terminated on a file which is out of chronological order.

  2. The maximum lattice symmetry consistent with the unit cell dimensions from the HKLIN file is determined, within an angular tolerance of 2 degrees (or that given on the TOLERANCE command). Alternatively, if the command ORIGINALLATTICE is given, the lattice symmetry corresponding to the space group in the HKLIN file is used, or the LAUEGROUP command may be used to specify a particular Laue group.

  3. The data are reindexed in the asymmetric unit of the lattice symmetry (if necessary), and sorted to bring potentially equivalent observations together.

  4. The intensities are normalised to E2 , making <E2> = 1, using an overall B-factor and a further correction smoothed on resolution bins, plus a time-dependent B-factor within each run. The batch number relative to the start of the run is used as a proxy for "time", and this correction is not done if there are only a few batches in the run (< 20 batches).  Unless resolution limits are explicitly set (RESOLUTION command), an automatic high resolution limit is applied, at the approximate point where <I>/<sigmaI> < IsigLimit (default value 4.0, set with ISIGLIMIT command). It is best to exclude weak high resolution data from the scoring functions, as they contain no useful information for this purpose.

  5. All rotational symmetry elements of the lattice symmetry are first scored separately. For example, in a tetragonal lattice, the symmetry elements are: 4-fold axis along c; 2-fold axes along a, b, c, (110) and (1-10). If the input file was merged, then symmetry elements implicit in the merged group are flagged as present.

  6. The most useful scoring function seems to be a correlation coefficient (CC) on E2 , calculated for pairs of observations related by a particular symmetry element. Another score calculated is  Rmeas, the multiplicity-weighted R-factor. In order to allow for small samples, the CC score is converted to a likelihood, and to "significance" score or Z-score by dividing by an estimated standard deviation. This is calculated by taking many pairs of observations at the same resolution which cannot be related by symmetry, dividing them into groups of the same size as the test sample (with a maximum of 200), the score calculated for each group and their mean & standard deviation calculated. Then Z(score) = [Score - Mean(UnrelatedScore)]/Sigma(UnrelatedScore)]

  7. All Laue groups which are sub-groups of the lattice group are generated by combining pairs of symmetry elements (including the identity) and completing the groups. For merged files, sub-groups of the merged symmetry are excluded, The sub-groups are then scored by combining the scores for the individual elements, counting scores for elements present in the sub-group as positive & those elements not in the sub-group as negative. 
    It is not clear what is the best way of combining the element scores: at present two scores are printed

    1. A "likelihood" (probability) estimate: this seems to be the most useful.

    2. Combined Z-Score ("Zc" in logfile)
      The correlation coefficients are recalculated (summed) over all "for" and over all "against" elements, Z(for) and Z(against) are calculated, then NetZ = Z(for) - Z(against)

  8. The potential Laue groups are ranked according to scoring method (i), and tested for acceptance for further output and testing. A group is accepted if its likelihood score is greater than (AcceptanceLimit * Largest likelihood score)  where AcceptanceLimit is set by the ACCEPT command [default 0.2]  

    If too many of the symmetry elements have no observations, then only the Laue group from the HKLIN file is accepted, unless the command LAUEGROUP ALL is given.

  9. All accepted Laue groups are tested for relevant systematic absences. These are scored to produce a combined score for possible space groups. 

    1. Each relevant "zone" in the lattice group is tested for absences. Zones are typically axes tested for absences due  to screws, or (in the case of non-chiral spacegroups) zones to be tested for glide planes.

    2. Observations which lie in the zone, such as along an axis, are scored by a Fourier analysis of I'/sigma(I). The score used is the peak height at the appropriate point in Fourier space, eg at 1/2 for a 2(1) screw, relative to the origin: a perfect score for exact absences is thus 1.0, a score for no absences might be 0.0. I' is an “adjusted” intensity, calculated by subtracting a small fraction (see NEIGHBOUR, default 0.02) of neighbouring intensities along the axis, to allow for possible contamination of a weak reflection by a neighbouring strong one.

    3. For each spacegroup in the Laue group a probability estimate is made from the Laue group score and the systematic absence score, and the space groups are ranked accordingly. In some case, more than one space group may have identical scores.

A note on setting conventions

In some space-groups there may be a difference between the IUCr "standard" setting and the "reference" setting given in given as the principle setting in International Tables. For chiral space groups, this affects only primitive orthorhombic (P 2x 2x 2x) and centred monoclinic (C2) groups. For primitive orthorhombic space groups the conventional standard setting has a < b < c irrespective of which axes are screws, eg space group 18 may be P 2 21 21 rather than the reference setting P 21 21 2. This convention allows the indexing program (eg Mosflm) to label the axes with a < b < c, without you needing to reindex the data at a later stage when the correct space group becomes clear. For a centred monoclinic cell, the choice for the standard is either C2 (side-centred) or I2 (body-centred), whichever has the smaller beta angle.

Pointless offers a choice of using the standard (CELL-BASED) or the reference (SYMMETRY-BASED) settings, see the SETTING command.

For more information see:
http://nvlpubs.nist.gov/nistpubs/jres/107/4/j74mig.pdf
    (doi: http://dx.doi.org/10.6028/jres.107.030)
http://nvlpubs.nist.gov/nistpubs/jres/106/6/j66mig.pdf
    (doi: http://dx.doi.org/10.6028/jres.106.050)

Rhombohedral lattices: these will generally be handled in a hexagonal centred-lattice (H lattice). There is limited support for the primitive rhombohedral lattice setting (R lattice, a=b=c, alpha=beta=gamma), but at present use of a reference file in the R setting is not supported. If a space group or Laue group is explicitly given as eg R 3, this will taken as a request for the hexagonal H setting: if you want the R-lattice setting, you must give the name as eg "R 3 :R"

KEYWORDED INPUT - DESCRIPTION

Keywords are:
ACCEPT, ALLOWASSUMESAMEINDEXING, BLANK, CELL, CENTRE, CHOOSE, COPY, EXCLUDE, HKLIN, HKLOUT, HKLREF, ISIGLIMIT, LABIN, LABREF, LAUEGROUP, MULTIPLYNAME, NEIGHBOUR, CHIRALITY, ORIGINALLATTICE, PARTIALS, POLARIZATION, REINDEX, RESOLUTION, RUNSCAIN, MERGED, SETTING, SPACEGROUP, SYSTEMATICABSENCES, TESTFIRSTFILE, TOLERANCE, USE, WAVELENGTH, WILDFILE, XDSIN, XMLOUT, XYZIN

All input is optional. Only the first four characters of each keyword are significant, and keywords are case-insensitive.

TESTFIRSTFILE

An input "reference" file (HKLREF) is always assumed to have the correct symmetry, but if there is no reference file and more the one input test file is given (HKLIN etc),  then the first defined file is used as a reference for subsequent test input files (HKLIN). By default, this first HKLIN file will be assumed for this purpose to have to correct symmetry and indexing, but  this command TESTFIRSTFILE  makes the program determine the Laue group for the first file before appending subsequent files. The Laue group test will be repeated after all data files have been read. NOTESTFIRSTFILE turns off this check, and is the default.

EXCLUDE  [FILE | SERIES <series_number>] BATCH <batch range> | <batch list>

Exclude batches from all calculations and the output file. Batches may be specified as a range <b1> TO <b2>  or as a list  <b1> <b2> <b3> ...
If there are multiple HKLIN input files, specifying the batch number may be complicated by the automatic renumbering of batches which is done to make them unique.  In this case the keyword FILE (or its synonym SERIES) may be used to specify the file number or file series number in the original input file: in the absence of this keyword, the batch numbers refer to the numbers after any renumbering. Files are numbered from 1 in the order they are specified on the command line or on HKLIN commands. File series specified by wild-cards on an HKLIN command or the command line have the same file number for this purpose.

RUN <Nrun>  [FILE | SERIES <series_number>] BATCH <b1> to <b2>

RUN BYFILE

Define a "run" : Nrun is the Run number, with an arbitrary integer label (i.e. not necessarily 1,2,3 etc). A "run" defines a set of reflections which share a set of scale factors. Typically a run will be a continuous rotation around a single axis. The definition of a run may use several RUN commands. BYFILE indicates that all batches from each file should be put into the same run. If no RUN command is given then run assignment will be done automatically, with run breaks at discontinuities in dataset, batch number or Phi (this may be defined explicitly as RUN AUTO). If any RUN definitions are given, then all batches not explicitly specified will be excluded.
The FILE or SERIES has the same effect as for the EXCLUDE command.

ORIGINALLATTICE

Use the original lattice symmetry from the file instead of determining the maximum lattice symmetry from the cell dimensions.

RESOLUTION [[LOW] <ResMin>] [[HIGH] <ResMax>

Resolution limits in A, either order or with keys HIGH or LOW. If this command is absent, the program imposes an automatic high resolution limit based on a minimum value for <I>/<sigmaI> within resolution shells, or CC(1/2) in P1 (see ISIGLIMIT). Limits given here override the automatic limits. Note that the automatic limits affect only the point-group determination, and do not apply to the systematic absence tests, nor affect the resolution in the output file. Explicit resolution limits given here cut the data for all purposes and restrict data in the output file.

ISIGLIMIT [<minimum<I>/<sigmaI>>] [CCHALF <CChalflimit>]

Minimum value for <I>/<sigmaI> within resolution shells [default value 6.0], or limit for CC(1/2) in P1 [default value 0.6]. These are used to set the maximum resolution for inclusion of data in the scoring. This is overridden by explicit RESOLUTION limits.

CHIRALITY CHIRAL | NONCHIRAL | CENTROSYMMETRIC

If this is NONCHIRAL the lists of possible space groups include non-chiral (or just centrosymmetric) ones as well as the chiral ones. The default is derived from the space group given in the input MTZ file.

LABREF  [F | I =]<columnlabel>

Only for INDEX MODE (ie if HKLREF is assigned). For the reference dataset, this defines the column label for intensity or amplitude (which will be squared to an intensity). If this command is omitted, the first intensity column (or if no intensities, the first amplitude) will be used. The next column is assumed to contain the corresponding sigma, though this is not compulsory for F.

LABIN  [F | I =]<columnlabel>

Only if the test dataset is merged. For the test dataset, this defines the column label for intensity or amplitude (which will be squared to an intensity). If this command is omitted, the first intensity column (or if no intensities, the first amplitude) will be used. The next column is assumed to contain the corresponding sigma, but this not compulsory. If the column is an I(+) or F(+), then a set of 4 columns is expected (eg I(+), sigI(+), I(-), sig(I-)).

LAUEGROUP  HKLIN || <Laue group name> || ALL

Specify a Laue group instead of testing all possible ones, ie set one solution for further processing in the systematic absence testing. A REINDEX command should be be given to specify a particular reindexing operator except in the case of a change of setting for the same Laue group (eg C2/m to I2/m, "LAUEGROUP I2"). The keyword HKLIN indicates that the Laue group from the input HKLIN file should be used, otherwise the Laue group name.   The keyword ALL may be used to force the program to accept all possible Laue groups for systematic absence analysis, even if there appear to be insufficient symmetry-related observations to distinguish them.

SPACEGROUP  HKLIN || <Space group name>

Specify a space group to write to the output HKLOUT file. A REINDEX command should be given to specify a particular reindexing operator, except in the case of a change of setting for the same space group (eg C2 to I2, "SPACEGROUP I2"). The keyword HKLIN indicates that the Laue group from the input HKLIN file should be used, otherwise the space group name.  In this case all that the program does is to reindex the data into the given space group and write it out.

CHOOSE [SOLUTION <SolutionNumber>] || [LAUEGROUP <group name>]

Choose a Laue group solution from the ranked list in the search, usually chosen from a previous run: a solution includes the reindexing operator. The Laue group solutions are ranked by the probability score, so solution 1 is always the most probable which is chosen by default. This choice may be over-ridden with this command, either as a solution number or as a group name: the systematic absence check will then be done to try to determine the space group. The Laue group name may be given either as the name as printed or as any space group name belonging to that Laue group. This command may also be used to explicitly choose C2/m or I2/m settings for centred monoclinic system, overriding the default choice (qv SETTING), eg CHOOSE LAUEGROUP I2

CHOOSE SPACEGROUP <group name>

Choose a space group solution from the search: first the equivalent Laue group solution is chosen from the ranked list in the search, usually selected from a previous run: a solution includes the reindexing operator. Then systematic absence check will then be done and the specified space group is chosen.

SETTING CELL_BASED || SYMMETRY-BASED || C2

This affects only primitive orthorhombic space groups (and non-chiral I & F centred orthorhombic groups), notably P 2 2 21 (17) and P 21 21 2 (18), when a space group is being chosen based on the systematic absences, and also centred monoclinic ones (eg Laue group C 1 2/m 1), but only in the absence of a reference file (which defines the space group & setting, and overrides any other convention). In the primitive orthorhombic cases, the axis order may either be set according to the general orthorhombic (International Tables) convention from the unit cell lengths with a < b < c (CELL-BASED), or in a “reference” setting which puts the “unique” axis along c. In the first convention (CELL-BASED, the default), for example, space group 18 may be set as P 2 21 21, P 21 2 21, or P 21 21 2. If SYMMETRY-BASED is selected, then eg space group 18 always P 21 21 2 irrespective of axis lengths.

The default CELL-BASED option also affects the Laue group setting for centred monoclinic lattices: in this case the body-centred setting (I 1 2/m 1, eg space group I2) will be chosen if it leads to a less oblique cell (smaller beta angle) than the C-centred setting (C 1 2/m 2, eg space group C2). The SYMMETRY-BASED or C2 options will always select the C2 setting.

See note above.

REINDEX  [LEFTHANDED] [<reindex operator>] ||  [MATRIX <H11> <H21> <H31>  <H12> <H22> <H32> <H13> <H23> <H33>]

Specify a reindex operator, either in the form eg "k,h,-l", as a matrix, or as the equivalent real-space change of basis eg "y,x,-z". The matrix is given in the same order as that in the XDS REIDX command,
ie h' = H11 h + H21 k + H31 l;  k' = H12 h + H22 k + H32 l; l' = H13 h + H23 k + H33 l

Note that the real and reciprocal space operators correspond to mutually transposed matrices, eg "x-y,-y,-z" corresponds to "h,-h-k,-l".

Reciprocal space reindexing operators may include a translation (eg h,k,l+1), real space operators may not. It is better to re-integrate the data with the correct indexing.

If  a space group is specified (SPACEGROUP), then that is used, otherwise the spacegroup from the HKLIN file is changed by the reindex operator. The reindexed data are written to the HKLOUT file. Note that there is no check that the operator is sensible and consistent with the space group, so be careful. If no space group is specified, the reindex transformation will be applied to the input space group, but this may lead to an unrecognised space group (eg non-cyclic permutations in P 21 21 21), in which case you will have to give the space group explicitly.

Normally this must be a right-handed operator (ie correspond to a matrix with positive determinant), and the program will fail if it is not, but the keyword LEFTHANDED allows a negative-determinant transformation to be applied. Be VERY sure that you really want to do this! It is only valid if the hand of data has been inverted by some previous mistake in the integration program, ie VERY rarely. Do not use this option unless you really know what you are doing.

TOLERANCE <LatticeTolerance>

Tolerance in degrees for determination of lattice symmetry [default 2 degrees]. Tolerance is the maximum deviation from the expected angle between two-fold axes in the lattice group, eg for a putative tetragonal lattice where a~=b, the expected angle between the diagonals is 90 degrees, and the deviation   delta = 2tan^-1(a/b) - 90
When testing alternative indexing schemes to match HKLIN to HKLREF datasets, this angular tolerance is converted to a "average length" tolerance by multiplying by the average cell edge.

ACCEPT <AcceptanceLimit>

Parameters for acceptance criterion. A group is accepted if its likelihood score is greater than (AcceptanceLimit * Maximum likelihood score)  where AcceptanceLimit is set by the ACCEPT command [default 0.2] 

PARTIALS [[NO]CHECK] [TEST <lower_limit> <upper_limit>] [CORRECT <minimum_fraction>]

Set criteria for accepting complete or incomplete partials. After all parts have been assembled, the total observation is accepted if:-

  1. the CHECK flag is set [default] and the MPART flags are all consistent (these flags indicate that a set of parts is eg 1 of 3, 2 of 3, 3 of 3)

  2. if CHECK fails, then the total fraction is checked to lie between lower_limit & upper_limit [default 0.95, 1.05]

  3. if this fails, then the incomplete partial is scaled up by the total fraction if  it is > minimum_fraction [default 0.6]

COPY

Just copy input to output: useful for converting an XDS (XDSIN) file or SCAIN to MTZ format  (HKLOUT), or just to sort HKLIN file(s). This is equivalent to the command line option "-copy" or "-c"

CENTRE [<hgrid> <kgrid> <lgrid>]

Search for centre of symmetry in the lattice. This should be at index 0,0,0, but occasionally the pattern has been misindexed, eg by +-1 along the rotation axis. This option performs an R-factor (Rmeas) search around 0,0,0, by default for +-2 grid points in each direction  (the size of the search grid may be reset to hgrid, kgrid, lgrid here). The lowest R should be at 0,0,0. Note that if the Laue group symmetry is wrong, this search may be less reliable: a LAUEGROUP command may be given to reset the symmetry. If an HKLOUT file is specified, the reindexed data is written (even if the reindexing operator is the identity).

NAME [PROJECT <project_name>] [CRYSTAL <crystal_name>] [DATASET <dataset_name>]

Assign or reassign project/crystal/dataset names, for output file. XDS files (XDSIN) and SCALEPACK (SCAIN) do not contain this information, so the NAME command may be used to supply this, for writing to the output HKLOUT file. In the case of MTZ input, the names given here supersede those in the input file: if there are multiple files, NAME commands may be interspersed with HKLIN | XDSIN | SCAIN commands to (re)assign datasets. Note that for files defined with input commands, this command must precede the relevant HKLIN | XDSIN | SCAIN command: a single NAME command may be given anywhere to (re)name datasets for files defined on the command line.

SYSTEMATICABSENCE  ON | OFF

Usually after checking for Laue group, reflections in special zones (axes) are checked for systematic absences to try to determine the space group. If this flag is OFF, the systematic absence check is skipped, and the "best"  Laue group is accepted.

NEIGHBOUR <neighbour_fraction>

For the systematic absence analysis along axes, intensities are “corrected” for possible corruption by neighbouring strong reflections which may inflate the value of reflections which should be absent. This is done by subtracting a small fraction of the neighbouring intensities from each intensity, not allowing the result to be negative: this corrected I' is used in the Fourier analysis of I'/sigma(I). The default value of neighbour_fraction is 0.02,

CELL <a> <b> <c> <alpha> <beta> <gamma>

Read cell dimensions. This is required for SCALEPACK unmerged and SHELX files,  as the cell is missing from the file. For other cases, this may be used to override the cell from the input file, but note that no check is made that the input values are appropriate (though a warning is given if they are very different from those in the file). It is also recommended for SAINT files, as the cell inferred from the data may not be very accurate. Note that to take effect, this command must precede the relevant HKLIN | XDSIN | SCAIN command, and the file names must be given as input commands, not on the command line.

WAVELENGTH <lambda>

Read wavelength to set the value for SCALEPACK or SHELX files, and to override the value from SAINT files. At present this is not used for MTZ or XDS files

ASSUMESAMEINDEXING [ON | OFF]

If there are multiple input files for the test data, it may be necessary to test that the 2nd and subsequent files are on the same indexing system as the earlier files.  By default, this test is done if separate file names are given (on the command line or as HKLIN commands), but not done within a “file series” ie a set of multiple input files are generated from wild-cards (eg "hklin foo_*.mtz"). This flag can be used to stop the index test ("ON"), which saves time, or to force the program to do the test ("OFF" or alternatively the command DONOTASSUMESAMEINDEXING).

HKLIN   <test data file name>

The filename for the input test data (MTZ format or others) may be given with this command, as an alternative to putting it on the command line (see HKLIN). Wild-cards "*" and "?" (as on the command line) may be used to specify multiple files (a “file series”). HKLIN may be used for non-MTZ files and the program will auto-detect the type.

WILDFILE [<template>]

Treat the filenames given on  the HKLIN command as if they contain wild cards  to define a file series. If this command is given without an argument, then the default template is "*_###.*, suitable for dealing with the output from MOSFLM with the "Multiple MTZ files" option. [This option is here mainly to get round limitations in ccp4i]

XDSIN   <test data file name>

The filename for the input test data (XDS format, XDS_ASCII or INTEGRATE) may be given with this command, as an alternative to putting it on the command line, or when there are multiple input files (see XDSIN). HKLIN may also be used and the program will auto-detect the type.

POLARIZATION [XDS | MOSFLM] [<polarization_factor>] [NORMAL <Pnx> <Pny> <Pnz>]

INTEGRATE.HKL files from XDS have been corrected for the polarization from an unpolarised incident beam, but not for the additional polarization correction from a synchrotron source, so this additional correction needs to be applied, and will be applied by default for this file type. This command has no effect on other types of input files.

There seem to be two definitions for the "polarization ratio" <polarization_factor>:
Here the value given is assumed to be in the XDS convention, unless the subkeyword MOSFLM is given, and it is later converted to the Kahn/Mosflm convention for internal use. Set polarization_factor <= 0.0 for an unpolarised beam.

The default value if not set explicitly = 0.99, = XDS 0.98, unless the wavelength corresponds to a likely in-house source, in which case the unpolarised value is left unchanged (recognised wavelengths are CuKalpha 1.5418 +- 0.0019, Mo 0.7107 +- 0.0002, Cr 2.29 +- 0.01)

The optional keyword NORMAL is followed by 3 numbers defining the polarization plane normal in the XDS convention (XDS keyword  POLARIZATION_PLANE_NORMAL). The default vector is (0,1,0): it is unlikely that you will want to change this.

 (Reference: Kahn, Fourme, Gadet, Janin, Dumas, & André,  J. Appl. Cryst. (1982). 15, 330-337)

SCAIN   <test data file name>

A file in the formats from SCALEPACK, SHELX or SAINT, or in free-format (see below). HKLIN may also be used and the program will auto-detect the type. The file should be unmerged ("unmerged original indices"): a merged file can be read, but probably is not much use. Project, Crystal and Dataset names may be given with the NAME command. A CELL command must be given except for SAINT files. Note that SCALEPACK may already have removed systematic absences, invalidating the analysis here: a warning is printed if this seems to have been done. Note that SCALEPACK files do not contain all the geometrical information from the data collection, and SHELX files contain no geometrical information, so output MTZ files (HKLOUT) are not really suitable for further scaling in AIMLESS. SAINT files have all necessary information. For merged ShelX files, a MERGED command should be given to indicate the space group (or point group) used for merging.

MERGED <space group name>

Only required for merged ShelX files, to tell the program that the file is merged, and in what space group

 HKLREF   <reference data file name>

The filename for the reference data may be given with this command, as an alternative to putting it on the command line

XYZIN <reference coordinate file name>

The filename for a reference coordinate set. Structure factors will be calculated to use as a reference, in the same way as HKLREF. The file should contain a valid space group name (full name with spaces, eg "P 21 21 21", "P 1 21 1" etc) and unit cell parameters (ie a CRYST1 line in PDB format). These values from the file may be overridden (or supplied) by SPACEGROUP and CELL commands if necessary.

HKLOUT   <output file name>

The filename for the output data may be given with this command, as an alternative to putting it on the command line

XMLOUT   <xml output file name>

The filename for the XML output data may be given with this command, as an alternative to putting it on the command line

ALLOW OUTOFSEQUENCEFILES

In a "file series" defined with a wild-card (see HKLIN), by default the modification times of files are examined to check that they in ascending time order. If a file is found to be out of sequence, that file and the rest of the series are rejected. This command ignores this check: all files in the series will be used. Modification times may also be reset with the system command "touch". [No time check is made on Windows systems]. 

MULTIPLY <input_scale>

Scale all input intensities by this factor (default 1.0)

BLANK [NONE] [RESO[FRACTION] <resofraction>] [NEGATIVE <negativefraction>]

Define criteria for recognising "blank" images which be eliminated from consideration and from the output file. A batch is considered to be blank if the proportion of negative intensities out to a resolution of <resofraction>*maximum resolution is greater than <negativefraction>. By default this test is not done (ie BLANK NONE): the keyword BLANK on it's own sets default values of resofraction = 0.5, negativefraction = 0.3. The value of <negativefraction> may be automatically increased above this value in cases of very weak data.

USE [ALL | EXCEPT]  <run list>

Select which runs to use in symmetry determination. This is largely intended for data from multiple lattices (file from FECKLESS): in that case, by default only Run 1 is used as this will (for now) correspond to the first (primary) lattice (ie default is USE 1). This command then allows override of this default. USE EXCEPT 2 for example uses all runs except run 2. USE ALL uses all. All runs are written to the output HKLOUT file, even if not used for symmetry determination.
NB, beware: this command will likely change in future to apply to Lattice rather than Run (when the Lattice class is implemented)

Input and output files

Filenames may be specified either on the command line or as commands: keywords are HKLIN, XDSIN, SCAIN, HKLREF, XYZIN, HKLOUT, XMLOUT. Note that HKLIN may always be used in place of XDSIN or SCAIN

HKLIN

The file or files containing the test dataset. Files may be specified either on the command line with or without the optional keyword HKLIN, or as command input, keyword HKLIN.  HKLIN may be used for other filetypes, as alternative to XDSIN and SCAIN: the program will automatically detect the filetype. Multiple files may be specified with "wild-cards" "*" or "?".

Lauegroup Mode. This should normally be an unmerged file of intensities eg from Mosflm

Compulsory columns are H, K, L, M/ISYM, BATCH, I, SIGI
Optional columns are IPR, SIGIPR, TIME, XDET, YDET, ROT, WIDTH, MPART, FRACTIONCALC, LP, FLAG, BGPKRATIOS, SCALE, SIGSCALE

If a SCALE column is present it will be applied on input.

A merged file can only be checked for under-merging ie for the true Laue group being a higher symmetry than that used for merging. A LABIN command may be given to specify the column labels for a merged file, otherwise the first column of type J (intensity) or F (amplitude) will be used.

 Index or Reindex Mode. This may be unmerged (as above) or merged. Unless a column is specified in the control input, the first column of type J (intensity) or F (amplitude) will be used for comparison with the reference dataset. Amplitudes are squared to intensities on input.

XDSIN

An XDS_ASCII file (or INTEGRATE): HKLIN may also be used. The file must be unmerged (MERGE=FALSE). Project, Crystal and Dataset names may be given with the NAME command.

SCAIN

A file in the formats from SCALEPACK, SHELX or SAINT, or in free-format (see below): HKLIN may also be used. The file should be unmerged ("unmerged original indices"): a merged file can be read, but probably is not much use. Project, Crystal and Dataset names may be given with the NAME command. A CELL command must be given except for SAINT files. Note that SCALEPACK may already have removed systematic absences, invalidating the analysis here: a warning is printed if this seems to have been done. Note that Scala versions before 3.3.18 did not deal properly with the secondary beam correction for Phi scans.

Free-format files are recognised by a first line beginning "COLUMNS" followed by a list of allowed item names, in the column order they are in the file. Recognised  names are H, K, L, I, SIGI, and BATCH: the first 4 are compulsory. Following this line, there may be optional lines CELL giving cell dimensions (an alternative to the CELL command: one of these alternatives must be given), and SPACEGROUP giving the space group name. Usually it is only the lattice type (P, C, I, F, R, H) which is important. In the absence of the SPACEGROUP information, it is assumed to be P1 and this is likely to cause failure for centred lattices

Values on each data line must be separated by one or more spaces.

Example:

COLUMNS H K L I SIGI BATCH
CELL  210.710 210.710 67.211 90.00 90.00 120.00
SPACEGROUP R3
-48 -18 -5 -17.53 54.63 1

-47 -18 -5 39.56 42.55 1
-46 -24 -4 42.92 40.79 1

HKLREF, XYZIN

The file containing the reference dataset for Mode 2 (alternative). This may be merged or unmerged. For a merged reference file, unless a column is specified in the control input, the first column of type J (intensity) or F (amplitude) will be used for comparison with the reference dataset. Amplitudes are squared to intensities on input. As an alternative, a coordinate file XYZIN may be given, in which case calculated intensities will be used as the reference.

HKLOUT

In LAUEGROUP mode, the test dataset reindexed in the "best" space group or point group. In ALTERNATIVE mode, the test dataset with the best reindexing, in the spacegroup of the reference dataset.

XMLOUT

An XML version of the logfile is written to a file if XMLOUT is assigned. This includes a BestSolution block
eg
<BestSolution Type="pointgroup">
   <GroupName>P 2 2 2</GroupName>
   <ReindexOperator>h,k,l</ReindexOperator>
   <ReindexMatrix>      1      0      0
                        0      1      0
                        0      0      1
    </ReindexMatrix>
   <Confidence>    1.000</Confidence>
   <TotalProb>  0.567</TotalProb>
</BestSolution>

Type is "spacegroup" if a unique space group is determined, or "pointgroup" if there is an ambiguity. The reindex matrix [H] post-multiplies the row vector h (ie h' = h [H]) 

References

P.R.Evans, Scaling and assessment of data quality, Acta Cryst. D62, 72-82 (2006).

The scoring scheme has changed somewhat since this paper was written, particularly in the calculation of the probabilities

 P.R.Evans, An introduction to data reduction: space-group determination, scaling and intensity statistics, Acta Cryst. D67, 282-292 (2011)

Examples

Simple usage, all defaults (mode Lauegroup):

 pointless [hklin] <filename.mtz>


With reference dataset (mode alternative):

pointless hklref amph_I.mtz hklin amph_scaled.mtz


With reference dataset and control input (mode alternative)
:

pointless hklref n20n6c1c2n6x2x14e10e7.mtz \
      hklin cd3_1_F.mtz << eof
resolution 4.0
labref  F_nat20
labin F_cd3_1
eof


Just reindexing

pointless hklin <infile> hklout <outfile> << eof
reindex k,h,-l
spacegroup P31
eof

Just copying (converting XDS file)

pointless -copy xdsin <infile> hklout <outfile>


Multiple input files

pointless hklout 12287.mtz << eof
name project JCSG crystal 12287 dataset INFL
hklin 12287_1_E1_001.mtz
name project JCSG crystal 12287 dataset LREM
hklin 12287_1_E2_001.mtz
name project JCSG crystal 12287 dataset PEAK
hklin 12287_2_001.mtz
eof


Multiple input files with exclusions

pointless << eof
#  File 1
hklin IID3_r5.mtz
#  File 2
hklin IID3_r6.mtz
exclude  batch 1             # final numbering, from file 1
exclude  file 2 batch  4     # original numbering in file 2
exclude  batch  1007 to 1008   # after re-numbering, from file 2
copy
hklout r5r6
eof


Multiple input files

pointless hklin abc_1.mtz abc_2.mtz  hklout 12287.mtz


Multiple input files with wild-card

pointless << eof

hklin abc_*_001.mtz

hklout abc_1.mtz

eof


Scalepack input

pointless scain gp6_e1_nomerge.sca


Release notes

1.10.25

Big speed-up on SF calculation from XYZIN

1.10.24

Added CELL and SPACEGROUP options to free-format input

1.10.23

Allow combination of multiple merged files into an unmerged file. Fix bug in getting unit cells from unmerged files derived from merged files

1.10.22

When comparing test data to a reference or to an earlier file, estimate resolution from CCs and plot CC vs resolution for each potential operator, etc

1.10.21

Trap case of no overlap between reference and data, print "sensible" error message

1.10.20

Further improvement to resolution limit robustness

1.10.19

Small fix to XML if the program finds lattice centering, remove "<>" characters. Changes to keep source in step with hklfile

1.10.18

Assign XML file from command line early so that it picks up syntax errors

1.10.17

Made resolution limit estimate more robust

1.10.16

If CHOOSing lauegroup or solution, set probability to 1.0. Allow I2 <-> C2 switch with reference for merged files

1.10.15

When comparing merged test data with a reference, assume test data point group is correct and check for compatible reindexing

1.10.14

Fix alternative indexing report when there has been a conversion between C2 <-> I2

1.10.13

Some fixes for merged MTZ files, eg resolution limits for accepted data only. Bug fix for error/warning reports.

1.10.12

Flag XDS "MISFITS" (actually outliers) properly: they are rejected by default in Aimless, but option KEEP MISFIT to accept

1.10.11

Fix cell averaging over batches with multiple files

1.10.9,10

Fix to bug in LAUEGROUP HKLIN option. XML fix for xia2

1.10.8

Refactor error monitoring so that error message are written consistently to XML. Close XDS files after reading to avoid problems with large numbers of files

1.10.7

Allow space group change within Laue group without reindex operator

1.10.6

Small XML fix for non-chiral cases

1.10.5

Tidied up explicit run definition

1.10.4

When estimating the effect of errors on predicted value of E(CC) for a "true" condition, reset if less than the maximum observed value for alternative indexings, ie don't let very poor agreement lead to incorrect choice of reindexing. Bug fix for 3 or more input files,  for data with  sigF missing, and for merged test data with unmerged reference.

1.10.3

Keep and output anomalous data (I+/-) from merged files. Fix memory leak in resolution calculation.

1.10.2

Allow spaces in space group in merged SCA files

1.10.1

Improved resolution estimate for point-group estimation, using both CC(1/2) in P1, and <I/sig(I)>, choosing the CC(1/2) estimate by default. Also plot resolution criteria. Flagged warning messages in XML. Added explicit RUN definition and RUN BYFILE. Centrosymmetric statistics in XML

1.9.31,32,33

Fixed XML typo for non-chiral groups. Fixed problem where reference file is in C2 but test data could be in I2. Evade long filename problem in XML

1.9.30

Interpret real-space symmetry operators correctly, and trap illegal space group operators arising from wrong reindexing

1.9.28, 29

Filter E^2 min at -0.1 to remove silly huge negatives. Add resolution cutoff to XML

1.9.27

Added some extra XML, reporting BestCell snd CellDeviation to reference

1.9.26

Update polarization correction for XDS INTEGRATE files to cope with a vertical spindle (eg Diamond I24)

1.9.25

Some fixes for Saint files. Fix to long-standing but rare bug in hash table. Added MMDB flags to help reading PDB files

1.9.24

Improvements to analysis of 6-fold screws (though still needs further work), including avoiding an occasional crash

1.9.22, 23

Minor changes. Internally classify data from the same dataset but from different crystals into the same Dataset object. Record alternative indexing schemes for final point group in logfile and in XML.

1.9.21

Fixed problem where sometimes output file had symmetry operators in wrong order, leading to problems with generating original hkl

1.9.19, 20

Fill in missing IPR (profile-fitted) with I (main intensity): this shouldn't normally happen

1.9.18

Fixed bug in "choose spacegroup" when "setting symmetry-based" is chosen

1.9.17

COPY skips twinning test. Print SG number and CCP4 number if different, to log file & XML

1.9.16

Fixed bug when combining files in C2 and I2 or when I2 is favoured over C2

1.9.14

Correct automatic increment of batch numbers to ensure that batch numbers in runs are not adjacent, eg not 1-1000 and 1001-2000

1.9.13

Allow filenames to include spaces if in quotes. In quoted project|crystal|dataset names, replace space by "_"

1.9.12

Sorted out options for combining multilattice data with single lattice files, renumbering lattices as appropriate

1.9.11

Free-format data file input option, automatically recognised. Simplify code recognising partials: only MTZ files from Mosflm have partials.

1.9.10

Fix to long-standing bug in reindexing files with different (permuted) unit cells. Small change to twinning output.

1.9.9

Bug fixes: for merged files, set CCP4 SG Numbers eg 3018 for P 2 21 21; make EXCLUDE BATCH work for Scalepack & Saint files; copy spacegroup from HKLREF for merged Scalepack files

1.9.8

Changed "<X>" in logfile to "< X >" to keep qtrview happy

1.9.7

More corrections to multilattice handling

1.9.6

Bug fix in writing merged output after comparison to reference

1.9.4, 5

Multilattice output fix  with "copy" flag. Echo command line arguments

1.9.2, 3

Another multilattice bug fix. For XYZIN reference, strip zero occupancy atoms to keep clipper happy

1.9.1

Updates to deal better with multilattice data

1.8.20

Prevent extreme truncation of resolution, try to accept at least 1/4 of resolution bins by reducing I/sigma cutoff

1.8.19

Activated merged MTZ output for merged Scalepack files with reference file (HKLREF), ie reindexing

1.8.18

Added warning message if space group not fully determined

1.8.17

Bug fixes: crash when using hklref with no alternative indexing; with multiple files, include columns present in any file

1.8.16

Added crash message for running out of memory std::bad_alloc

1.8.15

Fix bug in XML graphs (maybe only affects Aimless)

1.8.14

Fixed (rare) crash if only 2 axial reflections

1.8.13

Update MTZ history correctly, appending to anything from input MTZ file

1.8.12

Fix multilattice mtz read bug

1.8.10

Minor changes in XML for ccp4i2

1.8.9

Test for systematic lattice centering absences in primitive lattice, also rhombohedral obverse/inverse twinning test

1.8.8

More bug fixes

1.8.7

Bug fix for multilattice data: when expanded to higher symmetry lattice, overlaps were not excluded from calculations. If a reference file is given, always score the test data against this, even if no alternative indexing is present, and write score to XML file. Changed Likelihood calculation for alternative indexing to a straight comparison between CCs, rather than allowing (wrongly) for pseudo-symmetry, which is only relevant for actual Laue group determination.

1.8.6

Correct rationalisation of input of multiple MTZ files with different space groups in the same crystal system (eg R3 and R32), etc
In output of analysis of individual symmetry operators for primitive orthorhombic, print axis directions in original frame

1.8.5

Fix obscure bug in NAME command ("crystal dataXXX" misinterpreted as DATASET keyword). Rationalise input of multiple MTZ files with different space groups in the same crystal system (eg R3 and R32)

1.8.4

Fix for new version of XDS May 2013

1.8.3

write "proper" CCP4 standard symmetry information to MTZ file, pointgroup name "PGxxx" and strange spacegroup numbers eg 3018 for P 2 21 21

1.8.2

Some fix-ups when checking merged files for under-merging. May set unknown initial spacegroup to eg C1 which wasn't allowed previously

1.8.1

Dataset handling extended to handle better the case of a dataset coming from multiple crystals (really code for Aimless)
Options for handling multilattice data, using singletons only, by default from Run 1 only (see USE keyword)
Additional XML for ccp4i2

1.7.5

XDS "batch" set to int(ZD)+1 instead of Nint(ZD)+1

1.7.0,1,2,4

Major reorganisation: add Dataset class which may contain multiple Xdatasets. Additional XML output for ccp4i2.
Automatically use input MTZ space group if compatible with solutions found, eg P212121 instead of P222

1.6.21

Bug fix in reading XDS files from Windows on non-Windows. Fix "choose spacegroup" for eg P21 2 21 etc (didn't work)

1.6.19

Fix in pointgroup.cpp to cope with new cctbx

1.6.18

Allow "*" in XDS headers (but not "***" which indicates format overflow)

1.6.15, 16, 17

Set "spg_confidence" flag in output MTZ file  to indicate whether space group, enantiomorphic pair or point group have been determined
(17) bug fix from Aimless in Batch class

1.6.14

Changed ice ring rejection tolerance from 4 to 3 sd

1.6.13

Better scoring for 6-fold screw axes

1.6.11

XDS input guarded against illegal batch = 0 (reset to 1)

1.6.10

Minor infelicity in MeanSD class (could give sqrt(negative)) scala_util.cpp

1.6.9

Remove non-fatal error with some Saint files, worked but stopped ccp4i pipeline

1.6.8

Always do twinning test, even when there is a reference file

1.6.7

Fixed bugs in H <-> R lattice reindexing

1.6.6

Bug fix for very small number of overlaps with reference file leading to SD = 0 in probability calculation

1.6.5

Fixed a nasty bug in accessing a potentially deleted object in probability calculation (thanks to Charles Ballard) which crashed in Windows, also some  "uninitialised variable" errors found by Clemens Vonrhein

1.6.4

Reindexing merged files now deals properly with phase and ABCD columns, eg for I2<->C2 or  P2 21 21 <-> P21 21 2 conversions
Improved/fixed detection of new space group after reindexing

1.6.3

Trap "*****" fields from format overflow in XDS headers, just to give a sensible error message

1.6.2

XDS XSCALE now puts [U] = [I] to evade bug in Reindex, used in Xia2. Also restored the History record in the MTZ header (lost at some previous point)

1.6.1

fixed serious bug present in all 1.5.x versions. If an MTZ file input is in the highest point group of the higher symmetry group, specifically 422, 622, R32 or 432, and the final output file is in the same point group, then the symmetry operators were scrambled relative to the ISYM flags on the observations, leading to wrong original hkl being generated in Scala/Aimless: this then affects the secondary beam calculation and hence the absorption correction   

1.5.22

WILDFILE option
For enantiomorpic space groups, prefer to pick the one that matches the input file
Minor fix to reading XSCALE output files
Fixed long-standing bug messing up the phi start, phi end when there are a large number of images (eg > 1638), leading to unwanted breaks in a run in Aimless (this bug was also present in Scala, fixed in version 3.3.20))

1.5.21

Fixed nasty bug introduced in 1.5.18 or 19, data accepted in lower symmetry than the lattice group were output unsorted, leading to completeness in Scala/Aimless of > 100%. 1.5.17 and earlier are OK
Added polarization correction for XDS/INTEGRATE files

1.5.20

Recognise scalepack unmerged files from Scala/Aimless (space group names with spaces in them)

1.5.19

Twinning analysis, by L-test
Improved analysis of 6-fold screw axes

1.5.18

Likelihood score for alternative indexing

1.5.17

If ROT column is missing or all zero (or all the same), use BATCH number instead of ROT

1.5.16

Add more ice rings, from Harry Powell

1.5.15

Check and monitor multiple inconsistent unit cells within a dataset

1.5.14

Force "setting C2" if C2 explicitly chosen "choose spacegroup C2" or "choose lauegroup C2/m"

1.5.11, 12

BLANK test off by default
Moved my modified Rotation class (rotation.hh/.cpp)  from namespace clipper to namespace scala to avoids clashes.

1.5.10

Reads XDS CORRECT output, but puts batch number in the ROT column (since OSCILLATION_RANGE is unlnown)

1.5.8

Change order of output symmetry elements so that 2-folds for point groups 321 & 312 are grouped together
bug fix in string_util,  std::string changed to char*

1.5.5, 6, 7

More checks for merged files to handle some data weirdnesses

1.5.4

Improved (corrected) Laue group scoring for merged files, added explicit warning message for undermerging of merged files 

1.5.3

Fix symmetry handling to : (a) handle the unconventional space group I 1 21 1 (and any other that are in the CCP4 library file syminfo.lib); (b) improve handling of rhombohedral lattice centering (R as opposed to H), though this is still incomplete (reference file cannot be in the R-setting); (c) improved robustness to weird lattices in merged files

1.5.2

Bug fix for obscure pseudo-R3 case
Output MTZ file spacegroup will be labelled as eg "H 3" for rhombohedral lattices in the hexagonal setting, as in versions before 1.5.0 

1.5.1

More refactoring
The program will auto-detect the file type (XDS, SCA etc) for recognised types, for files given on the command line or with the HKLIN keyword.

1.5.0

XYZIN option to generate reference data from coordinate file
Refactored some symmetry classes (hkl_symmetry) to use Clipper classes instead of CCP4 classes: functionally equivalent
Some rationalisation of the treatment of merged files.

1.4.9

Made geometry calculations, and batch header information from SAINT correct for phi scans on 3-axis goniostats, and with rotations going backwards ("time" then = -phi) 

1.4.8

Fixed bug introduced in 1.4.6 when there is a 1-image gap (reset phi to wrong value)

1.4.7

Fixed bug in writing unmerged file read from a merged file

1.4.6

Detects and eliminates blank images (see BLANK command). Doesn't now crash on incomplete or missing files (this worked from 1.3.7, but was broken in 1.4.1)

1.4.5

Fixed bug in reference file reading without a sigma (eg Fcalc)

1.4.4

Fixed long-standing but usually harmless bug in intensity normalisation (could lead to nans in bad cases)

1.4.3

SAINT file input option (SCAIN)

1.4.1

Many changes to file reading to allow input of merged MTZ files (also Scalepack & ShelX files). Merged files may be checked for under-merging & for systematic absences. Allow an F to be used without a corresponding sigma (eg Fcalc).
SHELX format input, automatic detection from SCAIN command.
Fixed bug in detection of 4(1) axes in I-centred cells introduced in 1.3.13
Fixed long-standing bug in pointgroup class, which fortunately only affected rare cases (& probably only non-chiral ones)

1.3.14

An explicit RESOLUTION setting now properly overrides the automatic cutoff for Laue group determination on min<I/sigI>

1.3.13

Detect systematically missing axial data (eg odd-ordered indices previously deleted), & print warning.
Changed (improved?) probability estimates for systematic absences

1.3.11

Fixed bug in assigning partials in files missing an MPART column

1.3.10

Changed scoring of systematic absences to make it more robust to negative intensities

1.3.9

Check times in file series defined on command line as well as on HKLIN command. Allow input files specified both on the command line and on commands. 

1.3.8

Fixed bug in reindexing a merged HKLIN file against a reference HKLREF: it now picks the correct reindexing instead of always [h,k,l]

1.3.7

In a file series HKLIN: (a) incomplete files are ignored (an error message is printed by the CCP4 library); (b) files are checked for being in time sequence, and out-of-sequence files are rejected: this can be overridden with the ALLOW OUTOFSEQUENCEFILES command (not done on Windows)

1.3.6

The output file always takes the spacegroup from the reference file, if present.

1.3.5

Corrected calculation of Rmeas: numbers are larger than before, but are now correct!

1.3.3

Profile-fitted overloads are now actually used (they always were supposed to be!)

1.3.2 

Fixed occasional bug in systematic absence test, if there are only a few reflections ("results requested with SD unset")

1.3.0, 1 

Major rewrite of input file (hklin) reading, to speed up reading of large number of files
Fixed bug in systematic absence checks to select on reduced hkl instead of original hkl.  Previous versions missed eg 0k0 reflections  in point group 422: these are now treated as equivalent to h00.
Fixed some infelicities with non-chiral space groups (introduced by other changes), including gettting the dreaded space group 205 right (ie in the International Tables setting, rather than in the non-cyclic permutation of axes)

1.2.26

MTZ batch header was wrong if 1st batch was excluded: now fixed

1.2.25

Use full resolution range for systematic absence check

1.2.24

bug fix - EXCLUDE BATCH now works for single file

1.2.19

Returns STATUS = +1 for all fatal errors (I hope). Fixed XDS bug with STARTING_FRAME != 1

1.2.16

NAME command for files defined on command line. REINDEX MATRIX option. Now recognises NAME for XDSIN.

1.2.15

Fixed bug in SETTING SYMMETRY-BASED settings in primitive orthorhombic cases. SETTING C2 option

1.2.14

For XDS input, flag as bad PKratio (= 99.99)  & omit MISFIT observations (flagged in XDS file with SD < 0)

1.2.13

Multiple file series, no index checking within a series. Bug fix in XDS reading if STARTING_FRAME != 1

1.2.12

SETTING option, SPACEGROUP without REINDEX, fixed bug in CELL command for MTZ files.
Corrected intensities for systematic absence analysis (NEIGHBOUR option)

1.2.9,10,11

Faster XDS reading, various bug fixes, EXCLUDE BATCH option, CHOOSE option

1.2.7

Multiple HKLIN files from HKLIN command

1.2.6

Read XDS INTEGRATE.HKL files

1.2.5

Multiple input files with wild-cards on command line ("pointless hklin abc*.mtz"), ASSUMESAMEINDEXING  option

1.2.4

Testing alternative indexing to reference (either HKLREF or first HKLIN) now uses unmerged observations instead of the reflection average, to allow for misindexing in confusing cases, where observations which are actually different may have been assigned to the same reduced reflection. Added TESTFIRSTFILE option.

1.2.3

Fix problems with reindexing relative to reference file in non-standard setting or accidental alternative indexing.
Also  "reindex" command  will change space group  if required  (eg P21 2 21 reindexed  [l,h,k] to P21 21 2)

1.2.0-2

Multiple input files, put on to the same indexing scheme with unique batch numbers.
Time-dependent B-factor normalisation (a simple-minded radiation damage correction)
SCALEPACK input (SCAIN), CELL command

1.1.5

Added XDS input (XDSIN), NAME command, COPY (& -copy) options

1.1.3, 4

Alternative indexing option now lists correct cell deviation from reference cell
Fixed bug in reading eg "reindex -h,l,k" where leading "-" were stripped

1.1.2

Fixed up systematic absence scoring so that it works properly for 6-fold screws (allow for non-independence of Fourier terms): previous versions were wrong  (to use a technical term) for 6(1) axes.
Refactor reflection reading to allow for input other than from MTZ files in future.

1.1.1...

New (and better) scoring scheme: P(CC) based on Lorentzian distribution rather than Gaussian; probabilities combined for all symmetry elements in putative Laue group in different & simpler way

1.1.0.6

Fixed bug,  hklout file was written in Laue group instead of space group under some circumstances

1.1.0.5

Tolerance in alternative indexing now based on RMS difference of base vectors, rather mean square, tolerance converted from angular value using mean cell edge. Fixed omission of resolution limits from merged HKLOUT file in reindexing.

1.1.0

New likelihood scoring scheme for Laue groups, seems to work better than Net-Zc, modified acceptance scheme to go with this.

1.0.8

XML output from Index & Centre modes

1.0.7

Reindex mode tidied up (this is when either REINDEX or SPACEGROUP commands given). Anomalous columns are swapped if necessary (probably only if LEFTHANDED option is used, not recommended). Discard fractional indices after reindexing.
CENTRE option added. Uses scaled partials if incomplete & fraction > [0.6] (PARTIALS command)

1.0.6

Use profile-fitted overloads in systematic absence test

1.0.5

XML output if xmlout is assigned

1.0.0, 1

Systematic absence analysis to choose spacegroup
Correction of various bugs

0.5.0,1,2

Alternative ways of combining Z+ & Z- (Za, Zc), remove combined RMSD printing, more input controls (ORIGINALLATTICE, LAUEGROUP, REINDEX, TOLERANCE, ACCEPT)

0.4.0

HKLOUT output added

0.3.0

Mode Alternative, labin, labref

0.2.0

User input, resolution, Isiglimit, Nonchiral