Proof

• New functionality
  • Add support for processing many datasets in one go in TProof::Process(const char *dataset, ...).
    Two options are provided:
     - 'grand dataset':   the datasets are added up and considered as a single dataset (syntax: "dataset1|dataset2|...")
     - 'keep separated': the datasets are processed one after the other; the user is notified in the selector of the change of dataset so she/he has the opportunity to separate the results. A new packetizer, TPacketizerMulti, has been developed for this case: it basically contains a list of standard packetizers (one for each dataset) and loops over them (syntax: "dataset1,dataset2,..." or dataset1 dataset2 ...").
    In both cases, entry-list can be applied using the syntax "dataset<<entrylist", e.g. "dataset1<<el1|dataset2<<el2|".
    The datasets to be processed can also be specified on one or multiple lines in a text file.
    See http://root.cern.ch/drupal/content/working-data-sets for more details.
  • Add support for automatic download of a package when available on the master but not locally. The downloaded packages are store under <sandbox>/packages/downloaded and automatically checked for updates against the master repository. If a local version of the same package is created (using the UploadPackage) the entry in downloaded is cleared, so that the behaviour is unchanged.
    The new functionality is described in http://root.cern.ch/drupal/content/working-packages-par-files .
  • Add the possibility to remap the server for the files in a dataset. This allows, for example, to reuse the dataset information for the same files stored in a different cluster.
  • Add a local cache for TDataSetManagerFile. This is mainly used to improve the speed of TDataSetManager::ShowDataSets, which is run very often by users and may be very slow if the number of dataset is large. The cache is also used to cache frequently received dataset objects.
  • Add the possibility to audit the activity on the nodes via syslog. See http://root.cern.ch/drupal/content/enabling-query-monitoring .
  • New packetizer TPacketizerFile generating packets which contain a single file path to be used in processing single files. Used, for example, in tasks generating files. The files are specified into a TMap - named 'PROOF_FilesToProcess' - containing the list of files to be generated per host (the key is the host name, the value the TList of TObjString (or TFileInfo) with the files names - or a TFileCollection: the output of TFileCollection::GetFilesPerServer() can be directly passed as files map). Workers are first assigned files belonging to the list with host name matching the worker name. The map is distributed to the master via the input list.
  • Add support for automatic setting of pointer data members to the relevant object in the output list. The use of fOutputList->FindObject("name") in TSelector::Terminate is not needed anymore for pointer data members, e.g. histograms.
  • Add the possibility to define an external list of environment variables to be transmitted to the master and workers. This is done via the environment variable PROOF_ENVVARS. This addition allows to change the variables wthout changing the macro or application running TProof::Open. See http://root.cern.ch/drupal/content/controlling-environment-proof-session .
  • Add the possibility to save the perfomance information shown by the dialog into a small ntuple included in the output list. The ntuple contains 5 floats (processing time, number of active workers, event rate, MBytes read, number of effective sessions on the cluster) and it is filled each time the number of active workers changes or at max 100 regular intervals at least 5 secs apart; in this way the ntuple has at most O(100 entries + number of workers). To enable the saving of the ntuple execute the following:
            proof->SetParameter("PROOF_SaveProgressPerf", "yes");
    before running the query. The ntuple is called 'PROOF_ProgressPerfNtuple'.
  • Add support for worker autodiscovery in PROOF using the Avahi/Bonjour technology. The new functionality is supported on Mac (MacOsX >= 10.4; no need of additional installs) and linux (requires the Avahi framework, available by default on most of the distributions). To use this functionality (instead-of or in-addition-to the the static worker configuration via proof.conf or xpd.worker) the new directive 'xpd.bonjour' must be used (see description at
    http://root.cern.ch/drupal/content/configuration-reference-guide#bonjour).
    
    
• Improvements
  • Improve support for valgrind runs in PROOF-Lite
  • Add the possibility to add files to a dataset. This is achieved with a new option 'U' (for update) to RegisterDataSet.
  • Add methof TProof::GetStatistics to allow the client to retrieve the correct values of fBytesRead, fRealTime and fCpuTime at any moment; this will be used to setup a sort of ROOTmarks in stressProof .
  • Several improvements in the test program 'stressProof' and in the tutorials under 'tutorials/proof'
  • Avoid contacting the DNS when initializing TProofMgr as base class of TProofMgrLite: it is not needed and it may introduce long startup delays.
  • Make TProof::LogViewer("") start the viewer for a Lite session, in parallel to whats happen for TProof::Open("").
  • Several improvements in the handling of wild cards in the dataset manager; for example, issuing a GetDataSet(...) on a dataset URI containign wild cards will return a grand dataset sum of all the datasets matching the URI.
  • Add options to get a list of all dataset registered names from ScanDataSets (option kList; the result is a TMap of {TObjString, TObjString} with the second TObjString empty).
  • Improved version of the PQ2 scripts; the scripts now invoke a dedicated ROOT application (named pq2) available under $ROOTSYS/bin .
  • Add support for recursive reading of group config files via the 'include sub-file' directive. This allows to have a common part and, for example, customize differently the quotas.
  • Fix an issue with TTreeFriends. New tutorial showing how to use friends in PROOF.
  • Package management: add support for arguments in the SETUP function: it is possible now to pass a string or a list of objects. The TProof::EnablePackage interface has been extended to support this.
  • Optimize the validation step in the case not all the entries are required. The validation step is stopped as soon as the requested number of events is reached. If the parameter "PROOF_ValidateByFile" is set to 1, the number of files is exactly what needed; otherwise the number of files may exceed the number of files needed by (Number_Of_Workers - 1) .
  • New directive 'xpd.datadir' to better control the user data directories and their permission settings. 
  • In TPacketizerUnit, add the possibility to exactly share the number of cycles between the workers. See the parameter PROOF_PacketizerFixedNum.
  • Implement a timer to terminate idle sessions. The timeout value is controlled by the variable ProofServ.IdleTimeout (value in seconds). This variable can be set for all sessions in the xproofd config file via the 'xpd.putrc' directive.
  • Add the possibility to control the use of sub-mergers with the ROOTrc variable Proof.SubMergers. It has the same meaning of the parameter 'PROOF_UseMergers'. The capabilities of the latter have been extended: now -1 means disable the use of submergers (before  negative values were ignored and there was no way for the user to disable the use of submergers).
    
  • Packetizer optimizations: improved worked distribution when the number of files left to be processed is smaller than the number of workers and at least one file has a number of events significantly larger than the average; better apply the upper/lower limits on the expected packet processing time.
  • Add the possibility to single-out disk partitions in the packetizer; this works adding the beginning of a path in the name defining a new TFileNode (e.g. 'host://disk1' instead of 'host' only as it was so far). These feature can be enabled by defining the rootrc variable 'Packetizer.Partitions', e.g.
               Packetizer.Partitions  /disk1,/disk2,/disk3
  • Add to the output list the parameters used by the active packetizer.
    
  • In the PrintProgress function used to display a text progress bar, show also the average reading rate in [k,M,G}bytes/s in addition to the event processing rate. This is useful to have a feeling of the rate when running of a remote machine in batch mode.
  • Add the possibility to control the resident and virtual memory of a proofserv using 'ulimit', which has less limitations and more flexibility than setrlimit.
  • Deactivate workers when the requested packages could not be enabled properly.
  • Add support for reconfiguring the group manager and the {env,rootrc} settings. The related configuration files are checked for changes during the regular checks done by the XrdProofdManager.
  • Add support for selective definition of env and rootrc variables. Different values can be set for different users, groups, SVN versions or ROOT versions.
  • Improve the diagnostic in case of exceptions. Information about the event and file being processed at the moment the exception was raised is sent to the client, e.g.
       0.5: caught exception triggered by signal '1' while processing dset:'EventTree', file:'http://root.cern.ch/files/data/event_3.root', event:1 - check logs for possible stacktrace
    The patch also fixes a problem with submergers observed when a worker was stopped because above the memory limits: this worker was established as merger but could not do the work, for obvious reasons, freezing the session.
  • Add two new methods to TProof: ShowMissingFiles() to facilitate the display of the list of missing files; and GetMissingFiles() to get a TFileCollection (dataset) with the missing files for further processing.
    
    
• Fixes
  • Fix a bug in error status transmission which avoid session freezing in some cases
  • FIx a few issues in libXrdProofd.so with handling of connection used for admin operation: this should solve some cases where the daemon was not responding.
  • Fix a few memory leaks showing up when running several queries in the same session
  • Fix a few issues affecting the new sub-merging option
  • Fix an issue preventing proper real-time notification during VerifyDataSet
  • Fix an issue with TQueryResult ordering (was causing random 'stressProof' failures)
  • Fix an issue with TProof::AskStatistics (fBytesRead, fRealTime and fCpuTime were not correctly filled on the client; the values on the master, displayed by TProof::Print were correct).
  • Fix several small issues affecting the handling of global package directories
  • Fix an issue with socket handling in the main event-loop while sendign or receiving files via TProofMgr.
  • Fix a problem counting valid nodes in sequential or 'masteronly' mode, generating the fake error message "GoParallel: attaching to candidate!"
  • Fix a few issues with the length of Unix socket paths affecting PROOF-Lite and xproofd on MacOsX
  • Fix an issue with the release of file descriptors when recovering sessions .
  • Fix an issue with a fake error message ("Error in <TROOT::cd>: No such file root:/") in PROOF-Lite when issuing TProof::SetParallel().
  • Fix a problem with negative values for 'workers still sending' in PROOF-Lite .
  • Fix locking issue while building packages locally.
  • Fix issue setting permission and ownership of the dataset user directories.
  • Fix a subtle bug affecting the (possibly rare) case when not all entries are required and # entries does not correspond to an complete subset of files (e.g. # entries = 1001000 with files of 100000 entries each). The effect was uncomplete processing (skipped events, magenta bar) or a session freeze.
  • Fix problem with packet re-assignment in case of a worker death (some packets were processed twice or more times).
  • Fix problem with the transmission of non-default file attributes (e.g. the number of entries) from TChainElement to TDSetElement during TChain processing in PROOF
  • Fix problem in the default packetizer with validating the exact number of needed files when the information about the entries is already available.
  • Fix problem with 'xpd.putenv' and 'xpd.putrc' occuring when the variables themselves contain commas.
  • Avoid resolving the workers FQDN when running in PROOF-Lite, creating unnecessary delays when running PROOF-Lite within virtual machines.
  • Fix problem with the permissions of the user data directory.
  • Add files to the list of files to process only when finally validated.
  • Fix problem with canvases when the feedback canvas and the final canvas are the same (do not delete the feedback canvas at the end of processing)
  • Make sure that TProof::Load, TProofPlayer::SendSelector and TSelector::GetSelector treat consistently the extensions of the implementation files.
  • Unlock the cache after failure to load a selector; prevents session freezing
  • Correctly update the number of submergers when workers die
  • Add missing protection causing a crash in submergers when the output list contained TProofOutputFile objects.
  • Move the creation and start of the idle timeout from the end of SetupCommon to the end of CreateServer, so that the timeout is not active during worker setup.
  • Make sure that the TProof instance on the client is invalidated after an idle timeout.
  • Fix an old issue with DeactivateWorker("*") (the session is was terminated because no worker was active; this call coudl not be used as intermediate step to select a small number of workers).
  • Consistently check both Proof.Sandbox and ProofLite.Sandbox for sandbox non-default location as done in TProofLite
  • Fix a problem with the registration of missing files in the 'MissingFiles' list (files which could not be open on the workers were not always added to the list).