Database Utilities

This file contains the various commands for interacting with the underlying job database within CCP4i. The actual database commands are in the projectdb.tcl file. These functions are concerned with the provision of tools and interfaces for viewing and modifying the database contents.

Argument list: <project> <db> <statusVar> <userVar> <timeVar>

Returns the status in statusVar - "unlocked" or "locked"

project Alias for project

db Directory for corresponding database

statusVar Name of variable to return status in

userVar Name of variable to return user in

timeVar Name of variable to return time in

APIs to Access the Database

APIs to Access the Database

These are CCP4i-specific commands to interact with the database. They are built on top of the core database commands.

Argument list: <arrayname> <taskname> <status> <STARTING>

This procedure will automatically extract data from the array associated with a task and load it into the database. The data in the task array must have the appropriate parameter names:

TITLE A title for the job

INPUT_FILES A list of the *parameter* names in the task array for input files (NB not the file names themselves)

OUTPUT_FILES A list of the *parameter* names in the task array for input files (NB not the file names themselves)

The procedure returns the job id.

arrayname The name of the task array

taskname The name of the task

status Optional status - default is 'STARTING'

Argument list: <job_id>

This should only be used if job_id is the last job in the project. This procedure is usually called if the user has opted to abort running a job.

job_id Job id

Argument list: <job_id> <dir>

There are standard file names for log file of the form jobid_taskname

This does not return the file extension and only includes the full directory path if the dir parameter is set. This will generally be used when you want to create the file.

job_id Job id

dir Optional project/directory alias or the word 'PROJECT' which will use the current project directory.

Argument list: <job_id>

If the file does not exists then return null. This will generally be used when you want to read the file.

job_id Job id

Argument list: <job_id> <logfileVar> <formatVar>

If the file does not exist or can not be read then the format is returned as 'NOFILE'. The format will otherwise be 'TEXT' or 'HTML'.

This procedure reads the first few lines of the file to determine its format.

job_id Job id

logfileVar Returned full path name for log file

formatVar Returned file format

Argument list: <job_id> <mode> <filelistVar> <args>

By default only files are returned in the lists. If the -all argument is specified then any directories are also included in the list that is returned.

job_id Job id

mode either 'INPUT' or 'OUTPUT'

filelistVar Returned list of files with full path names

-all

Include all files and directories

Database GUI

Database GUI

These are procedures to draw the job list from the database in the ccp4i main window.

Argument list: None

The sort/search window consist in choice between a search and a sort and between simple or advanced one. Depending on the user's choice the parameters that can be tuned are different and can help the user to refine a search step by step or to do many different search and sorts. The results of the sorting and searching are displayed in this window, not in the main task window.

Argument list: <mode> <ssdb_w> <arrayname>

This procedure deals with the execution or refinement of a search or sort request when the user clicks on the action button in the window.

mode 0 (execute a new search) or 1 (refine existing results),2 resetting

ssdb_w The path of the toplevel searching window

arrayname The name of the array at global level holding the searching parameters

Argument list: None

db_sort_handler Handle a simple or advanced sorting triggered by the search/sort tool

Argument list: <want_refine> <win_path> <arrayname>

This method deal both with the simple and advanced sorting, and enables to sort on the result of previous searches through its parameters.

want_refine is a boolean value used to see if it is a new sorting or if the user want to sort on previous results. Set to 1 for refining, 0 otherwise.

win_path is the pathname of the search/sort windows used to feed the result

Argument list: <value_list> <unordered_val> <unorder_ids> <backup_val> <bound> <start>

This procedure take the value from a list to compare them to the entries of an array, and reorder all the arrays in the parameters to match the order of the list

value_list is a list containing the ordered values.

unordered_val is the name of the array containing the currently unordered values

unordered_ids is the name of the array containing the currently unordered job ids

backup_val is the name of an array containing values for advanced sort

bound is the upperbound for the index in the array to match the values of the list

start is the index in the array where the search need to start to find matching values

Argument list: <want_refine> <win_path> <arrayname>

This procedure deal with both simple and advanced search, fetching the jobs corresponding to the different values and retaining only those which match all criteria. It deals with an unfructuous search by displaying a message

want_refine is a boolean value used to see if it is a new search or if the user want to sort on previous results. Set to 1 for refining, 0 otherwise.

win_path is the pathname of the search/sort windows used to feed the result

Argument list: None

When this command is invoked initially it sets up an invocation of DbUpdateList (which updates the list of jobs in the main window) to happen a second later. Any additional calls in this time are essentially ignored. Once the list display has been updated, new calls will set up the next update to happen.

This mechanism is intended to prevent a sequence of distracting updates to the job display occurring each time a set of update requests are recieved in rapid succession.

Argument list: <args>

This procedure defines the parameters and format to be listed and calls db_update_list which actually draws the list.

Note that the -nosave option no longer does anything, as the save operations are handled internally by the database functions.

-nosave

Does nothing but is retained for backwards compatibility.

Argument list: <project> <db_display> <db_display_format> <\>

project The alias for the project to access

db_display A list of the database parameters to be displayed

db_display_format A list of the field widths (as integers) to use for displaying the items in the db_display list.

no_jobs_message The message to display if there are currently no jobs in the database.

Argument list: <mode> <w> <args>

save current selection in archive(CURRENT_SELECTION)

mode Usually 'select' for if user selected job or 'clear' if user clicked F2

w The widget identifier for the listbox which displays the job list

-ssdb

Indicates that the selection is coming from the search/sort tool

Argument list: None

This procedure updates the CURRENT_SELECTION element in the archive global array by checking that each item in the selection actually exists.

Argument list: <w> <x> <y> <ysel>

This procedure generates a "context menu" presenting appropriate actions available to the user to manipulate the currently selected set of jobs in the database job list.

The available operations depend upon the number of selected jobs.

w The name of the window that the menu was invoked from

x The screen X position of the mouseclick

y The screen Y position of the mouseclick

ysel The y position of the mouseclick within the window

Argument list: None

This procedure attempts to display the logfile for the job, if it has one.

Argument list: None

This procedure attempts to rerun the job that has been double clicked by the user in the job database list, provided that the job has a status other than REPORTED.

Argument list: <command>

All of the utility menu items invoke this procedure when picked. This procedure checks that the appropriate number of jobs for the utility have been selected.

command Keyword to identify the utility that the user has selected.

Argument list: None

This procedure defines the menu content and then call db_draw_utilities which does the drawing.

Argument list: <optionlist> <commandlist> <messagelist> <\>

All information for drawing the menus is input as lists with one list item per menu item. This approach makes it easy to change the menu but ccp4i currently does not do this (it's probably a bad thing anyway). It ought to be possible to write a similar procedure which draws the utilities as pull down menus and the user can then have the option on how to lay out the main window.

optionlist A list of the text to appear in menu. If the menu item has a submenu then the item is a list.

commandlist A list of the command keywords to be used by db_command_handler to invoke the right utility procedure.

messagelist A list of the messages associated with each menu item. Note that there are no distinct messages for each submenu item.

activelist A list of 0/1 to indicate if a menu item is active. Currently all input has these set to 1.

helpfile Name of a help file.

helplist List of the targets in the help file for each menu.

-ssdb

Indicates that the menu is drawn from search/sort tool

Argument list: <args>

This procedure is called when the user selects a job. The id of the menu is saved as SystemVar VIEW_JOB_JOBID.

The procedure tries to give appropriate options dependent on format of log file (? is it html) but does not look for tables in log file as this can be slow for big log files.

-ssdb

Indicates that the view is in the search/sort tool

Argument list: <mm> <directories>

This procedure is used from within DbViewJobMenu, to add a cascading menu to the end of the "View Files From Job" menu for each output subdirectory that is associated with the job.

The cascading menu lists the files contained within the subdirectory, and further cascading menus for each of the subdirectories within the current subdirectory. The procedure is called recursively to add further layers of nesting as required.

mm The window name for the Tk menu to add the cascades to

directories A list of directories to be added

Initialising, Loading & Saving the Database

DbGetCurrentProject Return the project alias and database directory for the current project.

Argument list: <projectVar>

Just a wrapper for GetCurrentProject.

projectVar Name of variable in which the project alias will be returned.

Argument list: None

Called from taskbrowser or whatever is main program

Argument list: <args>

This is essentially a wrapper to DbOpenDatabase which also performs some CCP4i-specific admin tasks, and invokes graphical components if prompting is required from the user.

-init

Should be specified the first time the procedure is invoked. It forces drawing of the utility menu if CCP4i is running in taskbrowser mode.

Argument list: None

This is essentially a wrapper to DbCloseDatabase.

Argument list: <project>

The user has changed the project so update the current project directory and database.

Handling Input from Running Jobs

Handling Input from Running Jobs

Jobs which are run external to the main graphical ccp4i process return status updates using socket connections.

The information returned, the job status, finish time, and output files, is entered into the database using these procedures.

Argument list: <job_id> <project> <args>

If the update information is that STATUS is FINISHED then this procedure will update the job time to the finish time and call the task review procedures. Any task can optionally define a review procedure which is called after the job has finished (only a handful of tasks do have review procedures). Finally redraw the job list.

job_id The job number

project The database project of the job - beware if user has changed project then cannot update database

parameter_1,data_1,parameter_2,data_2.. List of pairs of parameter name and value.

Argument list: <job_id> <project> <args>

job_id Job id

project The database project of the job - beware if user has changed project then cannot update database

file_1,dir_1,file_2,dir_2.. A list of pairs of parameters, the first is the file name and the second the project/directory alias. The file name should ordinarily be just the file name and not the full pathname. The dir is a project or directory alias. If a full path is given then only the trailing filename is used unless the directory alias is {"Full} path..\" or an empty string (in which case the directory is set to {"Full} path..\". If the filename is an empty string then this is still stored along with the directory name.

Argument list: <job_id> <start_time> <0>

If the job start time is defined then also check that any file with the appropriate name was created between the job start and finish time. If the file does not exist or has wrong creation time then remove it from the list of output files.

Nb The output 'files' can also include directories. In this case the filename should be an empty string.

job_id Job id

start_time Optional job start time in machine time.

Argument list: <filename> <args>

Return a list of job ids which have output files with the given name

This procedure is most useful in checking that a currently running job not create a file of the same name.

-running

Only search through currently running jobs

Argument list: None

This procedure is usually called when ccp4i is started

Checks jobs with status RUNNING REMOTE ON_HOLD

Argument list: <logfile> <statusVar> <timeVar> <output_filesVar>

ccp4i scripts put a short section of termination status on end of log file which is usually not needed but may be checked for jobs which have RUNNING status when ccp4i graphical process is started.

logfile Name of log file

statusVar Returned status for job

timeVar Returned finish time for job

output_filesVar Returned list of output files (this will be list of file/dir alias pairs)

Argument list: <filename>

When specified output file already exists and user has opted to overwrite it then check thru all jobs in database to see if the overwritten file is a recognised output file. If it is then delete it from the output_file list of the previous job.

filename File name - NOT the full path name

Database Utility GUIs

db_archive_window Draw the window with archive/delete options

Argument list: <job_id_list>

call db_cleanup_handler and db_archive_handler to do the business

NB code from this procedure has been hacked together in the db_removeoutput_handler (which is used to delete files when the "Kill&Remove" option is selected). The common code should probably be farmed out to separate procedures in the future.

job_id_list List of job ids

Argument list: <job_id> <job_id_list>

job_id Job id for job with changed default options or 'ALL' if overall default is changed

job_id_list List of ids for all jobs currently in archive/delete window

Argument list: <job_id_list>

job_id_list List of the ids for jobs in archive/delete window

Argument list: <job_id>

This procedure is called when the user selects to Kill a running job and remove all output. It prepares input for and then calls db_archive_handler, which does the removal.

The procedure uses code hacked from db_archive_window. The common code should probably be farmed out to separate procedures in the future.

job_id The id of the job being removed

Argument list: <job_id_list> <action> <>

This procedure will delete temporary files and directories, def files, com files (the input scripts for the programs) and remove the job from the database. If action is not set then apply the cleanup defined by archive(DELETE_OPTIONS) parameter

job_id_list List of the ids for jobs in archive/delete window

action Action to apply to the jobs in list (optional)

Argument list: <job_id_list>

This command should be invoked when the used clicks Apply&Exit in the Delete/Archive Files window - the interface is locked while the operations are in progress, to prevent the user exiting partway through.

job_id_list List of job ids

Argument list: <job_id_list>

Calls db_restore_handler to do the business

job_id_list List of the ids for jobs in archive/delete window

Argument list: <nselect> <text>

nselect Text defining number of jobs that must be selected

text Text describing utility that user picked

Argument list: <job_id>

job_id Job id

Argument list: <job_id>

Use a conventional task interface called taskinfo

job_id Id for job to edit. If it is 0 then enter info for a 'REPORTED' job

Argument list: <arrayname> <job_id>

This procedure used by db_edit_info

arrayname Array name for task array

job_id Job id

Argument list: <job_id>

job_id Job id

Argument list: <filename> <job_id>

An archive file is one that has been gzipped and move to the project database directory. Return 0 (no archive file) or 1 (archive file exists)

filename File name (NOT full path name)

job_id Job id.

Argument list: <filename> <job_id> <action>

Either archive or restore filename dependent on action

return a status for the file:

"null" can't find copy of file

"project" file in project directory (i.e. file with name filename exists)

"archive" archived version of file exists

"both" both project and archive version of file exist

filename Full path file name for file

job_id Job id

action Action required : 'archive' 'restore' or '' (do nothing)

Argument list: <filename> <job_id> <args>

filename File name (can be full path name)

job_id Job id

-ext

Append appropriate extension for compressed file

Argument list: <job_id>

Return 1 if a notebook entry exists and 0 otherwise

job_id Job id

Argument list: <job_id>

job_id Job id

Argument list: <job_id>

The com files are the scripts which are input to the programs

job_id Job id

Argument list: <f1> <f2>

See the Tcl documentation on lsort.

f1,f2 The com file names

Argument list: <job_id>

Use the KillScript or KillRemoteScript prcedures in the local.tcl

job_id Job id

Argument list: <job_id>

db_view_session_log Display the session log file

Argument list: None

db_config_window Present a window for the user to configure the connection to the db

Argument list: None

This generates a window which gives the user information about the current configuation of CCP4i with respect to accessing the project database.

In the case that CCP4i is connected to the database handler, it also monitors the status of the connection.

Argument list: <arrayname> <startupVar>

This sets the USE_DBCCP4I_ON_STARTUP parameter in configure to a new value specified by element startupVar in the array supplied by the calling procedure.

The configure parameters are then saved to file and reloaded into the interface. The mode of database access isn't changed until the next time CCP4i is started.

arrayname Name of the array used in the database configuration window

startupVar Name of the parameter storing the user's startup preference

Argument list: <arrayname> <statusVar> <infoVar>

This is initially called from the database configuration window. Subsequently it calls itself periodically to update the status and information parameters, essentially monitoring the connection to the database server in real time.

arrayname Name of the array used in the database configuration window

statusVar Name of the parameter displaying the connection status

infoVar Name of the parameter displaying the server information

Autotest Utilities

DbAutotest Handle command line running of autotest

Argument list: <args>

db_autotest Interface to automatic regression testing - rerun jobs in project

Argument list: None

the real work is done by db_autotest_handler

Argument list: <args>

Create a new project directory which is a sub-directory of the current project directory. Create a new database directory and database.def. Run all tasks in the project putting output files into the new project subdirectory.

-nographics

Running in command line (nographics) mode

Argument list: <mode> <job_id> <last_job> <old_project> <project> <args>

While the auto tests are running this procedure calls itself every couple of seconds and test if the current job is finished and if so then starts the next job

mode Mode is either 'init' for first time procedure is called or 'watch' thereafter

job_id The job id of the currently running job

last_job The id of the final job that the procedure is required to run.

old_project The project alias of the master project that is being rerun

project The project alias for the working project

-exit

Exit from ccp4i after completely the tests in this project

-remote machine

Run jobs on remote machine

Argument list: <old_project> <project> <job_id> <run_mode>

Change the directory aliases to that of the working project

old_project The project alias of the master project that is being rerun

project The project alias for the working project

job_id The job id

run_mode Set to 'remote' or 'background' - required to set the def file header

Argument list: <project> <user> <time>

project Alias of project

user Name of user who currently owns the lock

time Time when the lock was created

Argument list: <project>

Socket commands e.g. from running scripts which could not be executed immediately due to the main process being inaccessible, may be stored in a buffer file. DbFlushCommandBuffer flushes the buffer file and executes the commands using the "ccp4ish -socket" mechanism.

project Project alias

Handling Delayed Execution of Buffered Commands

DbGetBufferFile Return the name of the database command buffer file

Argument list: <project>

Clone of GetBufferFile in execute.tcl

project Alias for project

Starting Jobs (src/runjob.tcl)

Starting Jobs

The procedures in runjob.tcl will initiate the running of a job after the user has clicked the 'Run' button on a task interface.

Argument list: <mode> <taskname> <arrayname0>

This procedure checks that a script file script/taskname.tcl exists and will fail if it is not found unless running in interactive mode. In future it is feasible that scripts could be written in other languages and run_command would need to be intelligent about this to ensure the script was started appropriately. There is currently some unused code to handle the case of csh script.

mode can be: background: run in background as separate process on same machine, edit: as background but show user program scripts to edit, remote: Give user options to choose to run remote/batch, interactive: create the database entry but return control to the task interface to run the job interactively, #d_arg taskname The name of the task

arrayname Task interface array

Argument list: <mode> <format> <def_file> <script_file> <job_id>

The command to run start a ccp4i process is taken from the configure array - inappropriate setting of this value is a likely cause for jobs not running. The RunRemote procedure is invoked to run jobs remotely.

Note regarding the "mode" argument: this is a list that can contain between one and three elements. The first element indicates the mode of use (either "background" or "remote"). If the mode of use is "remote" then there must be a second element which is the name of the remote machine, and there may also be an optional third element that indicates the protocol to use to connect to that machine (either "rsh" or "ssh").

mode = background or remote

def_file Name of def file containing parameters for job

script_file Script file - usually only required for running Python

job_id The project database id for the job

Argument list: <arrayname> <input_files> <output_files>

A list of array elements for the names of input and output files are keep in the array elements INPUT_FILES and OUTPUT_FILES respectively.

arrayname Task interface array

input_files List of array elements containing the names of input files

output_files List of array elements containing the names of output files

Argument list: <arrayname> <file>

arrayname Task interface array

filename Name of one input file

Argument list: <arrayname> <file>

arrayname Task interface array

filename Name of one output file

Argument list: <arrayname>

arrayname Task interface array

Argument list: <job_id> <project>

When a job is aborted by the user, this command will be invoked by a message received by ServerAcceptInput in order to perform clean up of the job by invoking the cleanup handler.

job_id Number of the aborted job

project Name of the project that the job was running in

Argument list: <job_id> <project> <status>

When a job finishes running, this command will be invoked by a message received by ServerAcceptInput in order to perform any post-run operations - for example, running the task review procedures defined in the task file and finalising the status of the output files.

Completion in this context includes both successful and failed jobs.

job_id Number of the aborted job

project Name of the project that the job was running in

status Status of the job on completion e.g. FINISHED or FAILED.

Argument list: <arrayname> <text>

only used in few tasks so far

arrayname Task interface array

text text to insert in task title

Argument list: <script_file> <job_id> <args>

script_file Name of def file containing task parameters

job_id The project database id for the job

-machine machine

Set a default remote machine

Argument list: <w> <arrayname> <action> <def_file> <job_id> <args>

There is a fragment of code to run in 'SERVER' mode - this would be an implementation to allow running 'remotely' on the Windows platform.

w The id of the window

arrayname 'Remote' window array

action User selection: 'abort' or 'run'

def_file Name of def file containing task parameters

job_id The project database id for the job

Project Directory Utilities (src/projectdirs.tcl)

Project Directory Utilities

Within CCP4i the user's work is divided into 'project directories' with each directory being referenced by a 'project alias'. A project also has an associated job database.

Core Project Directory Functions

Core Project Directory Functions

These functions handle initialisation and saving of the overall project directory information. The data is stored in the global 'directories' array.

Argument list: <args>

This is a wrapper for InitialisePreferences directories, with some additional functionality to initialise the user's projects if necessary.

If the database handler is being used as the source of directories data then a connection is established with the handler, and status 1 is returned if there are no projects currently defined or 2 if there is at least one project already defined.

If a connection to the handler is not requested, or if a connection cannot be made, then InitialisePreferences is called and the status from that call is returned.

InitialiseDirectories accepts the same arguments as InitialisePreferences, although in the case of the database handler being used these are essentially ignored.

-lock

Check whether directories.def file is locked

-host hostname

Connect to a handler running on the specified host

-port port_number

Connect via a specific port number

Accessing Project Directory Data

Accessing Project Directory Data

These functions handle interactions with the information about the project directories.

Argument list: None

This returns a list of all the available project names (aka aliases) that are currently defined. If there are no projects currently defined then an empty list is returned.

Note that the projects are sorted into alphabetical (dictionary) order regardless of the order that they are retrieved from the database.

Argument list: <alias> <dir>

This adds a new project to the user's list, if necessary also creating a new directory and a database subdirectory.

It returns 1 on success, 0 on failure.

alias The one-word alias for the project

dir The full path for the project directory

Argument list: <alias>

Only the reference to the project is deleted - the project directory and contents are left intact.

alias The one-word alias for the project

Argument list: <project>

Returns the directory path associated with the project name or an empty string if the project is not defined.

project Alias/name of the project

Argument list: <project>

Returns the directory path for the database associated with the project name, or an empty string if the project is not defined.

project Alias/name of the project

Argument list: <project> <dir>

The project alias must already be defined.

This command should be used with caution as it could have side effects for reviewing and rerunning jobs in this and other projects.

project Alias/name of the project being updated

dir The full path for the new directory

Argument list: <project>

This is an internal function that does a lookup of the array index value for a project name in the directories array.

If the project name isn't defined then an empty string is returned.

project Alias/name of the project being indexed

Accessing DefDir Data

Accessing DefDir Data

These functions handle interactions with the information about the 'default directories' (defdirs).

'Default directories' are directories with associated aliases which are not project directories - that is, that have no associated job database.

Argument list: None

This returns a list of all the available defdir names (aka aliases) that are currently defined. If there are no defdirs currently defined then an empty list is returned.

The returned list will have TEMPORARY as the first def dir, if it exists.

A 'defdir' is a data directory i.e. a directory that has a shortcut name/alias but doesn't have an associated project database.

Note that the def dirs are sorted such that TEMPORARY will always be listed first (if it exists) and subsequent aliases will be in alphabetical (i.e. dictionary) order.

Argument list: <alias> <dir>

This adds a new defdir to the user's list. The directory doesn't have to exist when it is added.

It returns 1 on success, 0 on failure.

alias The one-word alias for the defdir

dir The full path for the directory

Argument list: <alias>

Only the reference to the def dir is deleted - the directory and contents are left intact.

alias The one-word alias for the def dir

Argument list: <defdir>

Returns the directory path associated with the defdir name or an empty string if the defdir is not defined.

defdir Alias/name of the defdir

Argument list: <defdir> <dir>

The defdir alias must already be defined.

This command is currently deprecated as it could have unpredictable results.

defdir Alias/name of the defdir being updated

dir The full path for the new directory

Argument list: <defdir>

This is an internal function that does a lookup of the array index value for a defdir name in the directories array.

If the defdir name isn't defined then an empty string is returned.

defdir Alias/name of the defdir being indexed

Project and DefDir Utility Functions

Project and DefDir Utility Functions

These functions provide simple interrogations of the project and default directory data.

Argument list: <project>

ProjectIsDefined Return 1 if the project name is already defined, 0 if not.

Argument list: <project>

project Alias/name of the project

Argument list: <dir>

Returns 1 if dir is a project or default directory, 0 if not.

dir Full path of the directory to check

Functions for handing default dir menus

Functions for handing default dir menus

These functions deal with the internal details of setting and getting the DEFDIR_MENU parameter.

Argument list: <menu>

menu A list containing the items in the menu

Argument list: None

Functions for handling database specifics

Functions for handling database specifics

These functions handle the specifics of the job database implementation.

Argument list: <dbfile> <utility_name> <identifier>

This is a copy of the DbCreateNewFile command in database.tcl, and creates a new database.def file.

dbfile Name of database file

utility_name type of database - expect it to be 'database'

identifier Identifier to write to file header - usually 'CCP4_Project_Database' - this is equivalent to the taskname in the usual task def files. A more descriptive identifier is used here in attempt to explain the file to anyone who reads the file itself.

Argument list: <callbackProc>

This allows a callback procedure to be specified, so that whenever the directories data is saved to persistent storage the callback will be invoked immediately afterwards.

See also the SaveDirectories procedure.

callbackProc Name of the command to be invoked.

Commands for Handling Automatic Directories Saves

Commands for Handling Automatic Directories Saves

These commands are used internally by the directories functions to manage scheduling automatic saves of the directories data to persistent storage as required.

DirectoriesModified is called by 'write' operations (e.g. adding or deleting projects). This schedules calls to SaveDirectories, which manages the actual saving operation.

Argument list: <broadcast>

This command is called by the directories 'write' operations. It sets a flag to indicate that the directories data needs saving to file, and schedules a call to SaveDirectories to perform the operation (if one is not currently scheduled).

This is an internal command and should not normally be called directly.

broadcast (Optional) The broadcast data received from the database handler

Argument list: <args>

This is a wrapper for SavePreferences directories and takes the same arguments.

Internal functions for dealing with the state of CCP4i

Internal functions for dealing with the state of CCP4i

These are internal functions used to keep information on the current project that CCP4i is operating with.

Argument list: <project_database>

project_database New value for PROJECT_DATABASE

Argument list: None

Commands for switching the database access mode

Commands for switching the database access mode

These commands can be used to change the method that the interface is using to access the database. The choice of commands is dependent only on the target mode, and not on the current access mode.

Argument list: None

Direct access is when CCP4i reads and writes the project database files directly.

Argument list: <host> <port>

Resets the database connection mode to connect to a handler running on a remote machine. The handler must already be running on the remote machine.

host The hostname of the remote machine with the handler instance

port The port number used by the remote handler

Argument list: None

This is the default mode for CCP4i when using the database handler to access the database.

Argument list: None

This command should be called after startup in graphical mode. It checks whether the current configuration for database access is consistent with the expected configuration when the database handler is being used.

In the event that the interface is configured to use the handler but the actual operating mode has automatically switched to direct access it offers the user the option of updating the configuration to use this mode on startup in future.

Internal functions for interacting with the database handler

Internal functions for interacting with the database handler

These are internal functions used to keep information on the current project that CCP4i is operating with.

Argument list: None

Returns 1 if the database handler is being used as the database source and 0 if CCP4i is accessing the database files directly (i.e. via the directories and database global arrays).

Initially the preferred access mode is set according to the setting in configure.def.

Argument list: <args>

Establishes a connection to the database handler using the client API commands. If the connection is successfully made then the internal parameter USE_DBCCP4I is set to one, otherwise it is set to zero (meaning that no connection is available). The return value of this command is the final setting of the USE_DBCCP4I parameter.

By default the connection is made to a local handler instance i.e. one on the current machine. If the -host and -port arguments are specified then the connection will instead be made to a remote handler instance. Note that the -host and -port arguments must be used together.

In either case if a connection has already been established then no action will be taken.

-host host_name

Connect to a handler running on hostname, rather than locally

-port port_number

Connect to port_number

Argument list: <broadcast>

This is invoked by the dbccp4i client API when the database changes. It shouldn't be invoked directly from within CCP4i.

Argument list: None

Returns 1 if the connection to the database handler is still active, and zero otherwise.

Argument list: None

Returns a list of the project names that can be accessed using the database handler connection. Note that it is possible not to have any projects defined, in which case an empty list will be returned.

Argument list: <alias> <dir>

This command will first ask the database handler to create a new project with the supplied name and directory. If this fails it then attempts to import the project instead. If either of these steps succeeds then 1 is returned, otherwise zero is returned.

alias Name of the project

dir Full path of the project directory

Argument list: <alias>

This command attempts to remove the reference to a specified project from the list of all the projects that the handler currently knows about. It returns 1 on successful removal and zero on failure.

Note that deleting the project reference doesn't remove the project directory or any of the files from the system. Deleted projects can therefore be added back later using the dbccp4i_add_project command.

alias Name of the project

Argument list: <project>

Returns an empty string if the specified project isn't found.

project Name of the project

Argument list: <project>

Returns the path to the database directory associated with a project (typically /path/to/project/CCP4_DATABASE/), or an empty string if the project isn't found.

project Name of the project

Argument list: None

Returns a list of the data directory names that can be accessed using the database handler connection. Note that it is possible not to have any data directories defined, in which case an empty list will be returned.

Argument list: <alias> <dir>

This command associates an alias with a directory where data is normally stored. If the add operation succeeds then 1 is returned, otherwise zero is returned.

alias Name of the data dir

dir Full path of the data directory

Argument list: <alias>

This command attempts to remove the reference to a specified data directory from the list of all data directories that the handler currently knows about. It returns 1 on successful removal and zero on failure.

Note that deleting the directory reference doesn't remove the actual directory or any of the files from the system. Deleted data dirs can therefore be added back later using the dbccp4i_add_defdir command.

alias Name of the data directory

Argument list: <defdir>

Returns the path to the data directory associated with the specified name, or an empty string if the data dir isn't found.

defdir Name of the data directory

Argument list: <dir>

Returns 1 if the specified directory path corresponds to a known project directory or data directory, and zero if it doesn't.

dir The directory path to check

Argument list: <project>

If the specified project name is already in the list of projects available to the handler then returns 1, otherwise returns 0.

project Name of the project

Argument list: <level> <text>

level A messaging level (with 0 being most serious)

text Text to report

Database cache for database handler

Database cache for database handler

The commands in this section are used to set up and maintain a cache of data for the current database, to speed up interactions with the job database (particularly for updating the job list).

Argument list: None

This clears any current contents of the cache and repopulates it with data for the specified project from the handler.

Once the cache is populated, data about individual projects and def dirs should be kept up-to-date automatically in response to broadcasts received from the handler.

Based on the dbccp4i_populate_cache command in projectdb.tcl.

Argument list: None

This returns 1 if the directories caches has been loaded indicating that the data is cached, and zero otherwise.

Argument list: <broadcast>

Given a broadcast message from the handler, this command determines which project or defdir has been updated and the operation that has been performed. It then calls the appropriate command to update the cached data.

broadcast Broadcast message received from the database handler

Argument list: <project>

This command retrieves the data for the specified project from the handler and inserts it into the cache.

project The name of the project being cached

Argument list: <project>

This command deletes the specified project from the cache.

project The name of the project to be removed

Argument list: <defdir>

This command retrieves the data for the specified def dir from the handler and inserts it into the cache.

defdir The name of the def dir being cached

Argument list: <defdir>

This command deletes the specified def dir from the cache.

defdir The name of the def dir to be removed

Core Database Functions (src/projectdb.tcl)

Core Database Functions

The database of user's jobs is stored in the array called 'database' and this saved to a file project_directory/CCP4_DATABASE/database.def in a standard def format file. It is strongly discouraged to access the database array directly from any procedure outside of this file. An empty database is defined in the file $CCP4I_TOP/etc/database.def. Each job in a project is given a job id which is an integer which is incremented, from 1, for every new job. The currently supported database parameters are NJOBS (the number of the highest job id in the database) and then parameters *,$n where n is the job id and the required values of * are given below. These parameters are written to the project database file, there are additional parameters may be saved in the database array but not be saved to file.

TASKNAME The name of the type of task

DATE The date that the job started (if still running) or finished saved as the machine time.

STATUS A one-word job status

LOGFILE The file name of the log file (NOT the full path name)

TITLE A text string of the job title provided by the user

INPUT_FILES A list of the input file names (not paths UNLESS the user has specified Full path rather than used a directory alias)

INPUT_FILES_DIR A list of the project/directory aliases for the input files

INPUT_FILES_STATUS Currently unused

OUTPUT_FILES A list of the output file names (not paths UNLESS the user has specified Full path rather than used a directory alias)

INPUT_FILES_DIR A list of the project/directory aliases for the output files

OUTPUT_FILES_STATUS Currently unused

Each project that the user creates will have its own database.def file and the appropriate file is loaded whenever the user changes between projects. The database.def file is updated from memory after every significant change. The time taken to read a database.def file is significant for projects with many jobs.

Commands for Handling Database Files

Commands for Handling Database Files

These commands deal with loading and saving the job database information from the persistent storage on disk i.e. the database.def file.

They are specific to CCP4i handling the database.def files directly by itself.

Argument list: <project> <args>

Returns 1 for success, 0 for failure, -1 if project is already locked by another process.

In the event of a lock not being overriden, the user and time information can be retrieved ysing the DbGetMessage command.

project Alias of project to load

-grablock

Override and grab the lock from another process (if necessary)

Argument list: <project> <args>

This procedure contains a messy bit of code to set the status of jobs in the output file to 'ON_HOLD' without changing the status in the loaded database. This is used for creating regression test directories and databases in db_autotest_handler

Return 1 if save was successful, 0 if not

project Alias of the project for which the database is to be saved

-hold

Set job status in output file to ON_HOLD

-range first last

if -hold also set, set job status of jobs in range first to last to ON_HOLD

Commands for handling messages and callbacks

Commands for handling messages and callbacks

These functions provide a mechanism for getting additional information back from the underlying database functions, particularly in the event of a failure or other problem.

Functions call DbSetMessage to record details in addition to the status or other values that they might return. Higher level functions can then fetch the message using DbGetMessage in the event of an error.

An example of this is when a database fails to load because of a lock, DbLoadFile records the user and time from the lock file using DbSetMessage. The calling function can retrieve this data using DbGetMessage in order to report it to the end user.

This isn't a particularly pretty way of doing things but it should work.

The DbRegisterCallback function provides a way for the top-level application to bind one or more commands which are invoked each time the database is updated by a write operation.

Argument list: <message>

message The message to be reported

Argument list: None

This returns the last message that was recorded by a call to DbSetMessage, and then clears the stored message.

Argument list: <callbackProc>

This allows a callback procedure to be specified, so that whenever the database is saved to persistent storage the callback will be invoked immediately afterwards.

See also the DbSave procedure.

callbackProc Name of the command to be invoked.

Commands for Handling Automatic Database Saves

Commands for Handling Automatic Database Saves

These commands are used internally by the database functions to manage scheduling automatic saves of the job database to persistent storage as required.

DbModified is called by 'write' operations (e.g. adding or deleting jobs, and modifying the value of data items). This schedules calls to DbSave, which manages the actual saving operation.

Argument list: <broadcast>

This command is called by the database 'write' operations. It sets a flag to indicate that the database needs saving to file, and schedules a call to DbSave to perform the operation (if one is not currently scheduled).

This is an internal command and should not normally be called directly.

broadcast (Optional) The broadcast data received from the database handler

Argument list: None

This command is used internally and should not normally be called directly. Instead a save of the data to persistent storage is scheduled by a call to the DbModified command - typically this is done automatically by the 'write' database operations.

The actual save of the data is done by a call to DbSaveFile. The save operation will only be performed if the DB_NEEDS_SAVE flag is set, to reduce the overhead of redundant saves that might be incurred otherwise.

Each time DbSave is called it checks for and invokes any callback procedures that were previously specified using the DbRegisterCallback command. The callbacks are invoked regardless of whether the database was saved.

Commands for Handling Database Locking

Commands for Handling Database Locking

These commands handle creating, interrogating and removing lock files associated with CCP4i project database.def files.

Argument list: <project>

Lock file name is database.LOCK

project Alias of project

Argument list: <project> <userVar> <timeVar> <pidVar> <>

Return 0 is lock file exists and 1 (i.e. OK to open database file) if it does not.

project Alias of project

userVar Return name of user who created lock file

timeVar Return time of creation of lock file

pidVar (Optional) Return id of process which owns the lock file

Argument list: <project>

Returns 1 if lock was created, 0 otherwise

project Alias of project

Argument list: <project> <args>

-force

Force deletion of the lock even if not owned by the current process

Commands for Opening and Closing the Job Database

Commands for Opening and Closing the Job Database

These commands handle opening and closing of the job database for a project.

Argument list: <project> <args>

Returns status from DbLoadFile: 1 for success, 0 for failure, -1 if project is already locked by another process.

project Project name (i.e. alias) for project

-grablock

Grab the lock from another process

Argument list: <project>

If there is a current 'open' database then make sure it is up to date and then unset the database array

project Alias for project

Core Database Functions

Core Database Functions

These functions handle direct interactions with the database information stored in the global database array.

Argument list: <taskname> <status> <STARTING>

DbJobExists Check whether a job exists in the database

Argument list: <job_id>

Checks for the existence of the specified job and returns 1 if found, 0 if not.

Currently just a wrapper for DbItemExists job_id STATUS.

job_id Job id

Argument list: <job_id> <type>

job_id Job id (Ignored if type = NJOBS)

type The type of data to be checked - see

Argument list: <job_id> <args>

There is no content checking in this procedure - the user is responsible for sensible input. There can be one or more pairs of input param_name/param_value arguments.

job_id Job id

param_name Name of database parameter - see

Argument list: None

NJOBS is an integer value corresponding to the highest assigned job id in the current project. If zero is returned then there are no jobs currently assigned to the project.

Argument list: <job_id> <args>

job_id Job id (Ignored if type = NJOBS)

type The type of data to be returned - see

Argument list: <job_id>

Returns 1 if the specified job has a subjob database associated with it, and zero otherwise.

Subjob databases can only be accessed if the database handler is being used. Otherwise subjob databases cannot be accessed, even if they exist.

project Name of the project

job_id Job number

Argument list: <job_id>

This command returns the full path and filename for a notebook file associated with the specified job in the current project.

Note that the file may not actually exist.

job_id Job for which the notebook is requested

Argument list: <type> <pattern>

By default the jobs are sorted on job number in descending order.

type Name of data item to select on

pattern Regular expression pattern to match against

Argument list: <project> <job_id> <db_display> <db_display_format>

project The alias for the project to access

job_id Job id

db_display A list of the database parameters to be displayed

db_display_format A list of the field widths (as integers) to use for displaying the items in the db_display list.

Argument list: <job_id>

job_id id for the job record to be removed

Argument list: None

Returns 1 if the lockfile has the same process id (pid) as the current process (in which case it is assumed that this process owns the database lock) and 0 if the lockfile has a different pid (in which case it is assumed that a different process owns the lock).

If no lock is found then assume that the current process cannot access the project i.e. it doesn't own the lock.

Internal functions for interacting with the database handler

Internal functions for interacting with the database handler

These are internal functions used to keep information on the current project that CCP4i is operating with.

Argument list: <project> <args>

Projects must exist and must be opened before they can be accessed. This command is a wrapper for the OpenProject client API command and so accepts the same arguments as that command, see below.

Return values are 1 (the project was opened successfully), 0 (the open operation failed for some reason), or -1 (the project wasn't opened because another process has a lock on it and the -force, -grablock or -readonly operations weren't specified).

This command will always try to open the project with both read and write access privileges. If the project is opened successfully but does not have write access, and the -readonly option was not specified, then an attempt will be made to acquire write privileges automatically.

project Name of the project to open

-grablock

Forces the handler to open a locked database

-force

An alias for -grablock

-readonly

Open project with read only mode

Argument list: <project>

Once the project has been closed, read/write operations will not work. The project must already be open in order to close it.

project Name of the project to close

Argument list: <project>

Returns 1 if write operations can be performed (i.e. creating or deleting jobs and setting job data) and 0 if not.

project Name of the project to check

Argument list: <project> <taskname> <status>

If successful, this command creates a new job record in the specified project and returns the job id number for that job. Otherwise a job id of zero is returned, indicating that the operation failed.

project Name of the project

taskname Task being run

status Status to assign to the job

Argument list: <project> <job_id> <item>

Returns 1 if both the job and the specified data item are found in the specified project, or 0 otherwise.

project Name of the project

job_id Job number

item The data item of interest

Argument list: <project> <job_id> <args>

Given a set of one or more item-value pairs, this attempts to set the value of each item.

project Name of the project

job_id Job number

item

The name of a data item in the project database e.g. TASKNAME, TITLE, STATUS etc

value

The new value to set the data item to.

Argument list: <project>

This returns the highest job id number, not the number of jobs in the project.

project Name of the project

Argument list: <project> <job_id> <args>

Given a list of one or more data items, this attempts to fetch the value of each.

If a single data item is requested then the return value will also be a single item, otherwise the command returns a list with each list item corresponding to the value of the requested data item.

project Name of the project

job_id Job number

item

The name of a data item in the project database e.g. TASKNAME, TITLE, STATUS etc

Argument list: <project> <job_id>

Returns 1 if the specified job has a subjob database associated with it, and zero otherwise.

project Name of the project

job_id Job number

Argument list: <project> <item> <pattern>

Given a regular expression pattern and the name of a data item this command returns a list of jobs where the value of the data item matches that pattern.

Note that this command expects Python regular expression syntax which may differ in details from Tcl for more complex expressions.

project Name of the project

job_id Job number

pattern A regular expression pattern

Argument list: <project> <job_list> <db_display> <db_display_format>

Given a list of jobs, a set of data items and a format descriptor this command returns a list where each item corresponds to one of the jobs in the original list, and consists of a string with values of each of the data items specified in db_display formatted using field widths given in db_display_format.

project Name of the project

job_list List of one or more job ids to get the description for

db_display List of database items

db_display_format List of field widths corresponding to the items in db_display

Argument list: <project> <job_id>

This removes a job from the project database. Note that any associated input or output files are NOT removed by this command - only the record of the job.

project Name of the project

job_id Job record to be deleted

Database cache for database handler

Database cache for database handler

The commands in this section are used to set up and maintain a cache of data for the current database, to speed up interactions with the job database (particularly for updating the job list).

Argument list: <project>

This clears any current contents of the cache and repopulates it with data for the specified project from the handler.

Once the cache is populated, individual job records should be kept up-to-date automatically in response to broadcasts received from the handler.

project Name of the project to cache

Argument list: <broadcast>

Given a broadcast message from the handler, this command determines which job has been updated and the operation that has been performed. It then calls the appropriate command to perform the update on the cache data.

broadcast Broadcast message received from the database handler

Argument list: <project> <job>

This command retrieves the latest data for the specified job from the handler and inserts it into the cache. The job doesn't need to exist before it is updated - it will exist afterwards (if it also exists in the persistent database).

project The name of the project which has been changed

job The id of the job to be updated

Argument list: <project> <job>

This is a wrapper to dbccp4i_cache_updatejob. It is possible that in future additional data may need to be associated with new jobs, in which case it should be done here.

project The name of the project which has been changed

job The id of the job to be added

Argument list: <project> <job>

Remove the data stored for the specified job, essentially removing the record from the cache.

project The name of the project which has been changed

job The id of the job to be deleted