Proof
- New functionality
- TProofMgr
- Add support for the following functionality:
- TProofDraw
- Allow to set a color, size, size, width for lines,
area, markers; the attributes are transmitted via the input list and
automatically derived from the ones of the chain
- Support automatic creation of a dataset out of files
created on the worker nodes by worker processes. The implementation is
an extension of the functionality of the class TProofOutputFile used
for merging via file. See
http://root.cern.ch/drupal/content/handling-large-outputs-root-files
- Add the possibility to enable/disable the tree cache and
to change its size on per-query base; two new parameters are available:
- PROOF_UseTreeCache
Int_t
Enable (0) or Disable (1) the tree cache (default 1)
- PROOF_CacheSize
Long64_t Cache size in bytes
(default 10000000)
Examples:
a) to disable the cache for the next run enter:
proof->SetParameter("PROOF_UseTreeCache", 0)
b) to set the cache size to 20M
proof->SetParameter("PROOF_CacheSize", 20000000)
- Add the parameter
PROOF_UseParallelUnzip to toggle the use of the parallel unzip
(default off for now); to enable it add the following call
proof->SetParameter("PROOF_UseParallelUnzip", 1)
- Add the possibility to give indications about
the number of workers at startup.
E.g.
1. To
start max 5 workers
TProof::Open("<master>","workers=5")
2. To
start max 2 workers per physical machine
TProof::Open("<master>","workers=2x")
This is useful in general when
running tests (equivalent but quicker then full startup
followed by
TProof::SetParallel(n) or TProof::DeactivateWorker(...)).
- Add support for the worker SysInfo_t in TSlaveInfo
(obtained via TProof::GetListOfSlaveInfos())
- Add new submerger functionality to speed up the merging
phase. At the end of the query, a set of workers are promoted
submergers and assigned a sub-set of workers to merge. Once each
sub-merger has merged its sub-set of workers, it sends its result to
the master, which merges the partial results into the final
set of results.
The determination of the sub-mergers is always done dynamically, based
on the recent performance of workers. An optimal (i.e. giving the
highest speed-up) number can be calculated analytically under simple
assumptions.
Merging via submergers is by default disabled. To enable it, with the
optimal number of sub-mergers, one should set the integer parameter
'PROOF_UseMergers' to 0, i.e.
proof->SetParameter("PROOF_UseMergers", 0)
To force S sub-mergers (regardless of the optimal number) do
proof->SetParameter("PROOF_UseMergers", S)
The new functionality can be tested in tutorials by adding the argument
'submergers' to runProof, e.g.
root [0] .L
tutorials/proof/runProof.C+
root [1]
runProof("simple(nhist=10000,submergers)")
(see the top of tutorials/proof/runProof.C for additional options).
A test for the submerger functionality has also been added to
test/stressProof.cxx .
- In PROOF-Lite, add the possibility for the administrator
to control the number of workers. This is done using
the rootrc variable ProofLite.MaxWorkers, which is read out of
/etc/system.rootrc and cannot be overwritten by users. Setting the
value to 0 disables PROOF-Lite.
- Improvements
- TFileMerger
- A few improvements on the way to make TFileMerger and
hadd totally equivalent:
- import from hadd an optimization of key hashing
- import from hadd a better way to invoke Merge for
generic objects
- add option to merge histograms in one go, instead of
one-by-one as for generic objects (this option is not yet supported by
hadd).
- TProofOutputFile
- Add support for the placeholder <file>
the definition of the outputfile. This allows to have complete URL and
to pass options to TFile::Open.
- XrdProofd plugin
- Add automatically the line 'Path.ForceRemote 1' to the
session rootrc file if the ROOT version is < 5.24/00 ; this acts
as a workaround for the wrong TTreeCache initialization at the
transition between local and remote files fixed in 5.24/00 .
- Enable mass storage domain settings when working with
TChain's
in multi-master mode. The Mass Storage Domain must be specified as
option in the URL
chain.AddFile("root:// .....?msd=CERN")
and the string must match the value specified in defining the
submaster node.
- Improved performance monitoring: the 'Rate plot' button
in the dialog box has been renamed 'Performance Plot' and now shows up
to 4 plots as a function of the processing time:
- Instantaneous processing rate, which is now better
estimated by a better estimation of the normalizing times
- Average read chunck size, defined as
TFile::GetFileBytesRead() / TFile::GetFileReadCalls() during the last
unit of time; this allows to monitor the usage of the cache; this plot
is present only if some I/O is done, i.e. not for pure CPU tasks.
- The number of active workers
- The number of total and effecive sessions running
concurrently on the cluster (started by the same daemon); this plot is
present only is the number is at least onec different from 1.
See also http://root.cern.ch/drupal/content/progress-dialog .
- If enabled, send monitoring information from the master
at each GetNextPacket (at each call of TPerfStat::PacketEvent) to allow
extrnal real-time progress monitoring.
- Save the status of a 'proofserv' session into a new file
in the 'activesessions' area. The full path of the new file is
<admin_path>/.xproofd.<port>/activesessions/<user>.<group>.<pid>.status
The status indicates whether the session is idle, running or queued.
The status is updated every 'checkfq' secs (see xpd.proofservmgr;
default 30 s). The status is dumped by the reader thread of TXProofServ
and therefore its r/w access is protected.
- Enable the use of the tree cache also for local files,
adapting the default settings for the cache to the recent changes
- In the XrdProofd plug-in
- Improve synchronization between parent and child during
fork
- Optimize loops over directory entries
- Improve error and notification messages
- Improved handling of Ctrl-C; this follows from a fix in
TMonitor and an improved handling of non-finished query state in the
workers (results are not send to master if the query was aborted)
- Fixes
- TFileMerger
- Fix a problem preventing correct transmission of all
non-mergeable objects (fixes bug #52886)
- Remove the argument isdir from the function
MergeRecursive
- Do not remove the first file in the list when returning
from MergeRecursive (fixes bug #54591)
- Fix a major leak when merging files with collections
written using kSingleKey option. The merger was reading each
key in memory and deleted the object at the end, but the container is
not owner by default, so all objects inside leaked.
- PROOF-Lite
- Fix a couple of memory leaks showing up when running
repeated queries
- Fix a problem in TProofServ::CopyFromCache affecting
the case where the sandbox dir has a '.' and the macro name has no '.',
e.g. compiled selectors in PROOF-Lite.
- TProofOutputFile
- Fix a problem with the determination of the fDir member
affecting mostly PROOF-Lite
- Fix a serious issue whose net effect was to delete the
outputfile just after having open it
- XrdProofd plugin
- Make sure that the limit on the number of old
sessions is applied whenever a new session is started and not only when
the daemon is started.
- Fix the behaviour of the xpd.allowedusers directive: if
at least one of these directives is present, users in the password file
are not allowed by default but must be explicitly appear in one
xpd.allowedusers directive
- Fix a source for memory leak in
XrdProofdProtocol::SendMsg
- Optimize the usage of strings in a few places
- DataSet manager
- Correctly classify as TTree all TTree derived classes
(e.g. TNtuple's)
- Fix a problem in saving the end-point URL for local
files
- Improve realtime notification during 'verify'
- TProofDraw
- Fix a problem with the axis ranges of the underlying
histogram in PolyMarker3D
- Allow to use the default pad instead of forcing
creation of one pad per object
- Add wrapper to handle the feedback default canvas
- TEventIter
- Fix a problem with changing the tree cache size: the
size was reset to the default value after the first file.
- TDataSetManagerFile
- Solve a consistency problem in checking URLs for
duplication when adding them to the relevant TFileInfo
- During dataset validation, do not fail on duplications
but notify and add them to the bad file list
- TPacketizerAdaptive, TPacketizer
- Improve data node / worker matching by always using the
host FQDN
- TPacketizerUnit, TEventIter
- Make sure that the entry
number passed to TSelector::Process is unique and in increasing order
for non-data driven processing (packetizer TPacketizerUnit). This
allows to give a meaning to this variable, for example to related it to
one dimension of an integration.
- Fixes in PROOF-Lite:
- Make sure that with envs settings via TProof::AddEnvVar
are effective; this enables, for example, the automatic valgrind setup
introduced in 5.24/00 or the experiment specific settings via the
script defined by the env PROOF_INIT
- Fix a problem with TProof::Load so that now it can be
also be used for PROOF-Lite
- TProofPlayerRemote
- In SendSelector, add misisng
option kCpBin when sending the selector source; the binary files were
never retrieved, even if present and valid
- TProofPlayerSlave
- In
Process, fix a problem with cache directory locking while building
the selector; the net effect was that each worker process was
re-buidling its own selector binary.
- TProofServ
- Fix
the order in which the log file is sent in asynchronous processing; the
wrong order was screwing up an immediate synchronous query submission
after an asynchronous run; this case occured, for example, in
'stressProof' .