Configure Advanced Options for MATLAB Job Scheduler Integration

Follow these instructions to configure advanced options for your MATLAB^® Job Scheduler cluster.

In the following instructions, matlabroot refers to the location of your installed MATLAB Parallel Server™ software. Where you see this term used in the instructions that follow, substitute the path to your location.

Set Up Windows Cluster Hosts

If this is the first installation of MATLAB Parallel Server on a cluster of Windows^® machines, you need to configure these hosts for job communications.

Note

If you do not have a Windows cluster, or if you have already installed a previous version of MATLAB Parallel Server on your Windows cluster, you can skip this step.

Configure Windows Firewalls on the Cluster Nodes

If you are using Windows firewalls on your cluster nodes,

Log in as a user with administrator privileges.
Execute the following in a Windows command prompt.
```
matlabroot\toolbox\parallel\bin\addMatlabToWindowsFirewall.bat
```
This command adds MATLAB as an allowed program. If you are using other firewalls, you must configure them for similar accommodation.

Open Required Ports on Server

The mjs service uses as many ports as required, starting with the value defined by the BASE_PORT parameter in the mjs_def file. By default, BASE_PORT is 27350.

If you use a host that runs a total of nJ job managers and nW workers, the mjs service reserves a total of 7+2*nJ+4*nW consecutive ports for its own use. All job managers and workers, even those on different hosts, that are going to work together must use the same base port. Otherwise the job managers and workers are unable to contact each other. In addition, MPI communication occurs on ports starting at BASE_PORT+1000 and use 2*nW consecutive ports.

For example, if you use a host with 1 job manager and 16 workers, then you need the following ranges of ports to be open:

27350 – 27422 for the mjs service.
28350 – 28382 for MPI communication.

You can change the default value of the BASE_PORT parameter in the mjs_def file. You must use the same value for the BASE_PORT parameter in the mjs_def file on all hosts in the cluster. To change the value of the BASE_PORT parameter, edit the mjs_def file on your cluster hosts before installing the mjs service and starting MATLAB Job Scheduler. You can find the mjs_def file in these locations:

matlabroot\toolbox\parallel\bin\mjs_def.bat on Windows operating systems
matlabroot/toolbox/parallel/bin/mjs_def.sh on Linux^® operating systems

To learn more about the parameters in the mjs_def file, see Define MATLAB Job Scheduler Startup Parameters.

To connect from MATLAB to a cluster with a non-default BASE_PORT value, you must append the value of BASE_PORT to the Host property in the MATLAB Job Scheduler cluster profile. You must do this in the form Hostname:BASE_PORT, for example myMJSHost:44001.

Increase Heap Memory

MATLAB Parallel Server supports MATLAB Job Scheduler clusters with up to 10,000 workers. When you scale up the number of workers or tasks in your cluster, you must increase the heap memory available to the job manager.

To do so, you must edit the mjs_def file on your cluster nodes before installing the mjs service and starting MATLAB Job Scheduler. You can find this file in these locations:

matlabroot\toolbox\parallel\bin\mjs_def.bat on Windows operating systems
matlabroot/toolbox/parallel/bin/mjs_def.sh on Linux operating systems

To learn more about the parameters in the mjs_def file, see Define MATLAB Job Scheduler Startup Parameters.

In the mjs_def file, set the JOB_MANAGER_MAXIMUM_MEMORY parameter in the mjs_def file based on the following recommendations.

Use a minimum value of 1000m (1000 MiB).
Use 1000m for every 1,000 workers.
Add 1000m for every 100,000 tasks expected to be queued at peak load.

For example, for a cluster with 10,000 workers and a peak queue size of 400,000 tasks, set JOB_MANAGER_MAXIMUM_MEMORY to 14000m.

After you have updated the mjs_def file, you must restart the mjs service, job manager and workers.