Documentation

Identify the Decode Function

The client’s submit function and the worker’s decode function work together as a pair. Therefore, the submit function must identify its corresponding decode function. The submit function does this by setting the environment variable MDCE_DECODE_FUNCTION. The value of this variable is a character vector identifying the name of the decode function on the path of the MATLAB worker. Neither the decode function itself nor its name can be passed to the worker in a job or task property; the file must already exist before the worker starts. For more information on the decode function, see MATLAB Worker Decode Function. Standard decode functions for independent and communicating jobs are provided with the product. If your submit functions make use of the definitions in these decode functions, you do not have to provide your own decode functions. For example, to use the standard decode function for independent jobs, in your submit function set MDCE_DECODE_FUNCTION to 'parallel.cluster.generic.independentDecodeFcn'.

Pass Job and Task Data

The third input argument (after cluster and job) to the submit function is the object with the properties listed in the following table.

You do not set the values of any of these properties. They are automatically set by the toolbox so that you can program your submit function to forward them to the worker nodes.

With these values passed into your submit function, the function can pass them to the worker nodes by any of several means. However, because the name of the decode function must be passed as an environment variable, the examples that follow pass all the other necessary property values also as environment variables.

The submit function writes the values of these object properties out to environment variables with the setenv function.

Define Scheduler Command to Run MATLAB Workers

The submit function must define the command necessary for your scheduler to start MATLAB workers. The actual command is specific to your scheduler and network configuration. The commands for some popular schedulers are listed in the following table. This table also indicates whether or not the scheduler automatically passes environment variables with its submission. If not, your command to the scheduler must accommodate these variables.

Your submit function might also use some of these properties and others when constructing and invoking your scheduler command. cluster, job, and props (so named only for this example) refer to the three arguments to the submit function.

Identify File Name and Location

The client’s submit function and the worker’s decode function work together as a pair. For more information on the submit function, see MATLAB Client Submit Function. The decode function on the worker is identified by the submit function as the value of the environment variable MDCE_DECODE_FUNCTION. The environment variable must be copied from the client node to the worker node. Your scheduler might perform this task for you automatically; if it does not, you must arrange for this copying.

The value of the environment variable MDCE_DECODE_FUNCTION defines the filename of the decode function, but not its location. The file cannot be passed as part of the job AdditionalPaths or AttachedFiles property, because the function runs in the MATLAB worker before that session has access to the job. Therefore, the file location must be available to the MATLAB worker as that worker starts.

You can get the decode function on the worker’s path by either moving the file into a folder on the path (for example, matlabroot/toolbox/local), or by having the scheduler use cd in its command so that it starts the MATLAB worker from within the folder that contains the decode function.

In practice, the decode function might be identical for all workers on the cluster. In this case, all workers can use the same decode function file if it is accessible on a shared drive.

When a MATLAB worker starts, it automatically runs the file identified by the MDCE_DECODE_FUNCTION environment variable. This decode function runs before the worker does any processing of its task.

Read the Job and Task Information

When the environment variables have been transferred from the client to the worker nodes (either by the scheduler or some other means), the decode function of the MATLAB worker can read them with the getenv function.

With those values from the environment variables, the decode function must set the appropriate property values of the object that is its argument. The property values that must be set are the same as those in the corresponding submit function, except that instead of the cell array TaskLocations, each worker has only the individual character vector TaskLocation, which is one element of the TaskLocations cell array. Therefore, the properties you must set within the decode function on its argument object are as follows:

1. Create a Scheduler Object

You use the parcluster function to create an object representing the cluster in your local MATLAB client session. Use a profile based on the generic type of cluster

c = parcluster('MyGenericProfile')

If your cluster uses a shared file system for workers to access job and task data, set the JobStorageLocation and HasSharedFilesystem properties to specify where the job data is stored and that the workers should access job data directly in a shared file system.

c.JobStorageLocation = '\\share\scratch\jobdata'
c.HasSharedFilesystem = true

If JobStorageLocation is not set, the default location for job data is the current working directory of the MATLAB client the first time you use parcluster to create an object for this type of cluster, which might not be accessible to the worker nodes.

If MATLAB is not on the worker’s system path, set the ClusterMatlabRoot property to specify where the workers are to find the MATLAB installation.

c.ClusterMatlabRoot = '\\apps\matlab\'

You can look at all the property settings on the scheduler object. If no jobs are in the JobStorageLocation folder, the Jobs property is a 0-by-1 array. All settable property values on a scheduler object are local to the MATLAB client, and are lost when you close the client session or when you remove the object from the client workspace with delete or clear all.

c

You must set the IntegrationScriptsLocation property to specify the folder containing the submit functions for this cluster.

c.IntegrationScriptsLocation = '\\scripts\myIntegrationScripts';

You can now define the scheduler object, and the user-defined submit and decode functions. Programming and running a job is now similar to doing so with any other type of supported scheduler.

2. Create a Job

You create a job with the createJob function, which creates a job object in the client session. The job data is stored in the folder specified by the cluster object's JobStorageLocation property.

j = createJob(c)

This statement creates the job object j in the client session.

This generic scheduler job has somewhat different properties than a job that uses an MJS. For example, this job has no callback functions.

The job’s State property is pending. This state means the job has not been queued for running yet. This new job has no tasks, so its Tasks property is a 0-by-1 array.

The cluster’s Jobs property is now a 1-by-1 array of job objects, indicating the existence of your job.

c

3. Create Tasks

After you have created your job, you can create tasks for the job. Tasks define the functions to be evaluated by the workers during the running of the job. Often, the tasks of a job are identical except for different arguments or data. In this example, each task generates a 3-by-3 matrix of random numbers.

createTask(j, @rand, 1, {3,3});
createTask(j, @rand, 1, {3,3});
createTask(j, @rand, 1, {3,3});
createTask(j, @rand, 1, {3,3});
createTask(j, @rand, 1, {3,3});

The Tasks property of j is now a 5-by-1 matrix of task objects.

j.Tasks

Alternatively, you can create the five tasks with one call to createTask by providing a cell array of five cell arrays defining the input arguments to each task.

T = createTask(job1, @rand, 1, {{3,3} {3,3} {3,3} {3,3} {3,3}});

In this case, T is a 5-by-1 matrix of task objects.

4. Submit a Job to the Job Queue

To run your job and have its tasks evaluated, you submit the job to the scheduler’s job queue.

submit(j)

The scheduler distributes the tasks of j to MATLAB workers for evaluation.

The job runs asynchronously. If you need to wait for it to complete before you continue in your MATLAB client session, you can use the wait function.

wait(j)

This function pauses MATLAB until the State property of j is 'finished' or 'failed'.

5. Retrieve the Job's Results

The results of each task’s evaluation are stored in that task object’s OutputArguments property as a cell array. Use fetchOutputs to retrieve the results from all the tasks in the job.

results = fetchOutputs(j);

Display the results from each task.

results{1:5}

    0.9501    0.4860    0.4565
    0.2311    0.8913    0.0185
    0.6068    0.7621    0.8214

    0.4447    0.9218    0.4057
    0.6154    0.7382    0.9355
    0.7919    0.1763    0.9169

    0.4103    0.3529    0.1389
    0.8936    0.8132    0.2028
    0.0579    0.0099    0.1987

    0.6038    0.0153    0.9318
    0.2722    0.7468    0.4660
    0.1988    0.4451    0.4186

    0.8462    0.6721    0.6813
    0.5252    0.8381    0.3795
    0.2026    0.0196    0.8318

Save Job Scheduler Data

The first requirement for job management is to identify the job from the cluster’s perspective. When you submit a job to the cluster, the command to do the submission in your submit function can return from the scheduler some data about the job. This data typically includes a job ID. By storing that job ID with the job, you can later refer to the job by this ID when you send management commands to the scheduler. Similarly, you can store information, such as an ID, for each task. The toolbox function that stores this cluster data is setJobClusterData.

If your scheduler accommodates submission of entire jobs (collection of tasks) in a single command, you might get back data for the whole job and/or for each task. Part of your submit function might be structured like this:

for ii = 1:props.NumberOfTasks
    define scheduler command per task
  end
  submit job to scheduler
  data_array = parse data returned from scheduler %possibly NumberOfTasks-by-2 matrix
  setJobClusterData(cluster, job, data_array)

If your scheduler accepts only submissions of individual tasks, you might get return data pertaining to only each individual tasks. In this case, your submit function might have code structured like this:

for ii = 1:props.NumberOfTasks
    submit task to scheduler
    %Per-task settings:
    data_array(1,ii) = ... parse character vector returned from scheduler
    data_array(2,ii) = ... save ID returned from scheduler
    etc
  end
  setJobClusterData(scheduler, job, data_array)

Define Scheduler Commands in User Functions

You now have the scheduler data (such as the scheduler’s ID for the job or task) stored on disk along with the rest of the job data. You can then write code to control what the scheduler should do when that particular job or task is canceled or destroyed.

To do so, create the following functions:

Your cancelJobFcn.m function defines what you want to communicate to your scheduler in the event that you use the cancel function on your job from the MATLAB client. The toolbox takes care of the job state and any data management with the job data on disk. Therefore your cancelJobFcn.m function needs to deal only with the part of the job currently running or queued with the scheduler. The toolbox function that retrieves scheduler data from the job is getJobClusterData. Your cancel function must have the following signature:

function cancelJobFcn(sched, job)

    array_data = getJobClusterData(clust, job)
    job_id = array_data(...) % Extract the ID from the data, depending on how 
                             % it was stored in the submit function above.
    command to scheduler canceling job job_id

In a similar way, you can define what you want your scheduler to do when deleting a job, or when canceling and deleting tasks.

Delete or Cancel a Running Job

To use your functions with your cluster, you must locate the functions in the folder identified by the IntegrationScriptsLocation property of your cluster object.

Continue with job creation and submission as usual.

j1 = createJob(c);
for ii = 1:n
    t(ii) = createTask(j1,...)
end
submit(j1)

While the job is running or queued, you can cancel or delete the job or a task.

This command cancels the task and moves it to the finished state. The cancelJobFcn function is executed, and sends the appropriate commands to the scheduler:

cancel(t(4))

This command deletes job data for j1. The deleteJobFcn function is executed, and sends the appropriate commands to the scheduler:

delete(j1)

Get State Information About a Job or Task

When using a third-party scheduler, it is possible that the scheduler itself can have more up-to-date information about your jobs than what is available to the toolbox from the job storage location. To retrieve that information from the scheduler, you can write a function called getJobStateFcn.m. You must place this function in the folder identified by the IntegrationScriptsLocation property of your cluster object. The function must have the following signature:

function getJobStateFcn (cluster, job, state)

Suppose that you use a toolbox function such as wait, etc., that accesses the state of a job on the generic scheduler. After retrieving the state from storage, the toolbox runs the getJobStateFcn.m function located in IntegrationScriptsLocation. The result is returned in place of the stored state. The function you write for this purpose must return a valid value for the State of a job object.

When using the generic scheduler interface in a nonshared file system environment, the remote file system might be slow in propagating large data files back to your local data location. Therefore, a job’s State property might indicate that the job is finished some time before all its data is available to you.