This document gives an overview of test procedures for the CCP4i project database handler "dbccp4i" and the underlying code. The tests are divided into different types:
There are a number of test scripts in the test directory of the dbccp4i distribution.
A Tcl wrapper script called run_tests.tcl is provided which runs all the test scripts, and reports any failures. It is invoked from the command line with:
% tclsh run_tests.tcl
however it is possible to invoke it from any location provided that the appropriate path to the script is provided. The wrapper script uses helper scripts located in the test/helper subdirectory.
The tests should run on both Unix and Windows platforms. Note that the wrapper script will refuse to run if it detects a dbccp4i process already running. This is to prevent the tests from interfering with the operation of other existing client applications, and vice versa.
Note also that the tests currently use the existing directories.def file of the user that runs them - so if the tests fail, in the worst cases references to the test directories may be left behind. This should not be a problem however the user will at some point need to remove the references manually.
The following sections outline the test scripts that are included in the test directory and which are run by the wrapper script. They can also be run standalone.
| test_ccp4i.py | Test functions for the classes and methods in the ccp4i.py module, built using the Python unittest module. Usage: % python test_ccp4i.py |
| test_dbsocket.py | Unit tests that check the classes and functions in dbsocket.py directly (specifically, the XML wrapping and unwrapping functions), built using the Python unittest module. Usage: % python test_dbsocket.py |
| dbccp4i_test_client.py | A simple client test application that tries basic operations via the Python client API. |
| test_dbxml.tcl | Unit tests that check the procedures in dbxml.tcl directly. Loads the functions from dbxml.tcl in the ClientAPI directory and attempts various operations to test the functions. Usage: tclsh test_dbxml.tcl The script attempts to perform the following actions:
Note that this script tests the functions directly without invoking the handler or the client API. |
| dbccp4i_test_client.tcl | A client application that tests various operations via the Tcl client API. |
| broadcast_test_client.tcl | A client application that tests the broadcast messages sent by the handler in response to various operations. |
| dirlock_client.tcl | A client application that tests how dbccp4i handles a lock on the user's directories.def file when client attempts to register. Usage: tclsh dirlock_client.tcl If the test succeeds then the handler should deal correctly with the lock it encounters. |
| rude_client.tcl | A client application that tests for handler failure on unexpected client disconnection. Usage: tclsh rude_client.tcl If the test succeeds then the handler should still be running correctly after the client disconnects. |
| test_directories_client.tcl | A client application that tests for the consistency between two different methods of acquiring data about projects and data dirs. |
These tests attempt to check used by the version of MrBUMP that uses the handler with a custom directories.def.
| dbccp4i_test_client.tcl -directories directories_mr | Run the test client application using the custom directories.def for MrBUMP, rather than the default directories.def file. |
| test_directories_mr.py | Simple client application that uses the custom directories.def file for MrBUMP via the Python client API. |
These tests require that a pre-existing project be supplied as an argument. Within the run_tests.tcl script, helper scripts setup_test_project.tcl and teardown_test_project.tcl are used to create a temporary project with a unique name that the tests are subsequently run using.
| bigrequest_client.tcl project -limit 10000 | Operates in one of two modes. In the first (default) mode, it opens the specified project and then submits requests which are longer each time, until the handler fails to return consistent output, giving a rough estimate of the maximum size of the request that the system is able to process through the Tcl client API. In the second mode the user specifies a maximum number of jobs that the system must process, and the script reports whether this number can be successfully processed. Usage: tclsh bigrequest_client.tcl <project_name> [-limit <n> ] [ -adjust_timeout ] In the default mode, the script attempts to perform the following actions:
The script reports the size (in terms of numbers of jobs) of the failing request. If -limit is specified then the script tries a single request with a list of <n> jobs, and reports whether or not this was successfully processed. Notes: This test is of limited use in ascertaining the maximum size of requests that the system can deal with.
For reference: with the default timeout in the iterative mode, the maximum request size on one particular machine. is around 16300 jobs. In the same mode with the adjusted timeout option, the maximum request size obtained is significantly higher (41400 jobs). A value of "10000" is used in the wrapper script as a baseline limit. |
| ccp4icompat_client.tcl project | Client application that attempts to simulate the situation when a second process (e.g. CCP4i) interfering with the project lock and database files while the handler is also accessing the project. Usage: tclsh ccp4icompat_client.tcl <project_name> The script attempts to perform the following actions:
At each step the script attempts to get data from the project to verify that the behaviour is as expected. |
| openclose_client.tcl project | Client application that attempts to close unopened or previously closed projects, and to open projects that have already been opened. Usage: tclsh openclose_client.tcl <project_name> The script attempts to perform the following actions:
The script reports the success or failure for each of these actions, which should all fail if the tests are successful. |
| opendb_client.tcl project | Client application that tests for dealing with existing lock on project database when client attempts to open it. Usage: tclsh opendb_client.tcl <project_name> If the test succeeds then the script should report the list of jobs found in the project. |
| demanding_client.tcl project | Client application that attempts to simulate the effect of a client that generates large numbers of new jobs and other changes to a project in a short space of time (for example, an automated script). Usage: tclsh demanding_client.tcl <project_name> <task_name> The script attempts to perform the following actions:
The script can be edited to adjust the time interval between the creation of new jobs. Outside of the wrapper script, multiple instances of the script can be run simultaneously to test the effect of multiple demanding clients, e.g.: #!/bin/sh -e # # Run many instances of the demanding client tclsh demanding_client.tcl JUNK fake1 & tclsh demanding_client.tcl JUNK fake2 & tclsh demanding_client.tcl JUNK fake3 & tclsh demanding_client.tcl JUNK fake4 & tclsh demanding_client.tcl JUNK fake5 & tclsh demanding_client.tcl JUNK fake6 & Warning this script is very messy - it will leave large numbers of "fake" jobs in the project that it is run in. Either use a temporary project, or else open the project in CCP4i and use the "Delete/Archive..." options to remove this jobs in bulk. |
| dbitems_client.tcl project | Client application tests that the data items available from the handler matches the list expected from looking at the default database.def file. |
| test_addfiles.py project 1 filename | Client application to simulate the addition of the same filename
to a job multiple times.
Usage:
% python test_addfiles.py <project> <job_id> <filename>
|
These are not true automated tests, as they do report success or failure in a way that can be checked from the wrapper script.
| dbsocket_crash.py project 1 | Python client application that is designed to crash. In this situation the
Python client API should also terminate.
Usage: % python dbsocket_crash.py <project> <job_id> |
The following procedures outline manual tests that can be performed to check that the software exhibits the correct behaviours.
| Identifier | #1 |
| Name | Only one handler runs per user |
| Description | Only one instance of the handler should run for each user of the system. Starting a subsequent instance of the handler should result in control being passed to the existing one, and the subsequent instance should then stop. |
Procedure & Expected Outcomes
On UNIX/Linux:
Start a handler in one terminal window using the "dbccp4i debug" command
You should see the startup message from the handler:
================================================================ DB Handler started at 2006-10-30 15:06:12.258985 Version: $Revision: 1.9 $ ================================================================
Start a second handler in another terminal in the same way
You should see the following message from the second handler:
There is another handler running on port 4090.
Do "ps -ef | grep python"
Only one python handler process should be running:
pjx 22602 15780 2 15:09 pts/11 00:00:00
python /home/pjx/BIOXHIT/Bioxhit_db/dbccp4i/dbccp4i/dbccp4i.py
| Identifier | #2 |
| Name | Multiple clients |
| Description | Two clients operating on behalf of the same user should be able to connect to the same handler, and modify and read the same data. |
Procedure & Expected Outcomes
On UNIX/Linux:
Start a handler process using the "dbccp4i debug" command
You should see the startup message from the handler:
================================================================ DB Handler started at 2006-11-02 13:54:58.231302 Version: $Revision: 1.9 $ ================================================================
Start a console process in another window
You should see the console start up message and list of commands.
Start a second console process in yet another window
You should see the console start up message and list of commands.
Open the same project in both consoles by doing OpenDatabase in each
The project should be opened, e.g.:
% OpenProject JUNK OpenProject JUNK Result: OK
Create a new job in the first console
A new job should be created and the id number returned, e.g.:
% NewJob JUNK NewJob JUNK Result: 3
Broadcast messages should be received in each console:
% ****************************************************** Received broadcast message: Update JUNK 3 ******************************************************
List the jobs in the second console
The new job id number should also appear in the list, e.g.:
% ListJobs JUNK ListJobs JUNK Result: 1 2 3
In the second console, set the TASKNAME, TITLE and STATUS data items for the new job
% SetData JUNK 3 TASKNAME test TITLE "This is a test" STATUS REPORTED SetData JUNK 3 TASKNAME test TITLE "This is a test" STATUS REPORTED Result: 1
Broadcast messages should be received in each console:
% ****************************************************** Received broadcast message: Update JUNK 3 ******************************************************
In the first console, check the values of the TASKNAME, TITLE and STATUS data items for the new job
% GetData JUNK 3 TASKNAME STATUS TITLE
GetData JUNK 3 TASKNAME STATUS TITLE
Result:
test REPORTED {This is a test}
| Identifier | #3 |
| Name | ccp4i/dbhandler compatibility |
| Description | When running both dbhandler (including its clients) and ccp4i, ccp4i should be able to take control of the database while the handler should be able to read database and reflect the database update which made by ccp4i, but handler shouldn't be able to write to the database. |
Procedure & Expected Outcomes
On UNIX/Linux:
Start a dbconsole
You should see the console start up message and list of commands.
open a project in console by doing OpenDatabase projectname
The project should be opened, e.g.:
% OpenProject TEST Added item INPUT_FILES_STATUS to the db Added item OUTPUT_FILES_STATUS to the db Warning: directory in file is /home/wy45/projects/TEST/CCP4_DATABASE OpenProject TEST Result: 1
Start a ccp4i and open the same project.
You should see a warning lock message, click 'Override Lock', the jobs should appear in the job window in ccp4i.
Attempt to update data from console.
In console, do SetData TEST 1 STATUS RUNNING
This operation should failed. e.g.:
% SetData TEST 1 STATUS RUNNING SetData TEST 1 STATUS RUNNING Result: 0
however, the console should be able to read the data. do GetData TEST 1 STATUS
you should see the following:
% GetData TEST 1 STATUS GetData TEST 1 STATUS Result: FINISHED
Update database from ccp4i
In ccp4i, edit the job 1, change status to FAILED
console should receive broadcast message.
% Warning: directory in file is /home/wy45/projects/TEST/CCP4_DATABASE ****************************************************** Received broadcast message: TEST is changed by other process, the database is reloaded with readonly mode. ******************************************************
in console, do GetData TEST 1 STATUS
% GetData TEST 1 STATUS GetData TEST 1 STATUS Result: FAILED
Exit console by doing exit
% exit Disconnecting before exit, please wait... ========================================================== Finished ========================================================== wy45@WY45PORT:~/work/Bioxhit_db/dbccp4i/application> ====================================================================== DB Handler stopped at 2007-03-09 14:57:11.859556 ======================================================================
Exit ccp4i by clicking 'Exit' button.
Start a dbviewer.
You should see the display of one of the existing project.
Start a ccp4i and open the same project.
You should see a warning lock message, click 'Override Lock', the jobs should appear in the job window in ccp4i.
Update database from ccp4i
In ccp4i, edit the job 1, change status to FINISHED
In dbviewer, you should see it automatically refresh the display and the job number 1 status change to green.
Exit both ccp4i and dbviewer.
Start ccp4i.
e.g OpenProject TEST.
% OpenProject TEST Added item INPUT_FILES_STATUS to the db Added item OUTPUT_FILES_STATUS to the db OpenProject TEST Result: 0Open the project with readonly mode by doing OpenDatabase TEST -readonly.
% OpenProject TEST -readonly Added item INPUT_FILES_STATUS to the db Added item OUTPUT_FILES_STATUS to the db OpenProject TEST -readonly Result: 1
Repeat step 3 & 4.
Start a dbviewer, repeat step 8.
Exit ccp4i, dbconsole & dbviewer.
These are documented here in case they are useful in special circumstances, when they may help as debugging tools.
| dbccp4i_diagnose.tcl | A basic diagnostic tool. Usage: tclsh dbccp4i_diagnose.tcl This script reports basic data about the environment, versions of Tcl, availability of SQLite and so on, and runs a set of diagnostics on each of the projects that it finds. This can be a very lengthy procedure which is not particularly informative. |
| check_saves.tcl | Independently monitors saves of the database.def file for a project Usage: tclsh check_saves.tcl <project_name> This monitoring script is intended to be used in conjunction with the console application. It checks periodically on the last modification time of the project in question, and reports when the file has been updated (i.e. rewritten). Ideally writes should only happen when the database content has been updated by the console (or other) application. |
| test_cache.tcl | Intended to help in manally testing the internal caching mechanism in the Tcl client API. |
| test_nocache.tcl | Intended to help in manally testing the internal caching mechanism in the Tcl client API. |
CVS Id: $Id: tests.html,v 1.9 2008/08/12 10:48:16 pjx Exp $