How can I call an external function (executable on a high performance computing cluster) for fminsearch?

I would like to use fminsearch to find the minimum of a black box function that is the result of a complex simulation. The simulation is written in C++ and I have a linux HPC cluster where the function is evaluated (it takes between 2 and 25 hours to complete the simulation). Is it possible to create a wrapper function in MATLAB that calls the function with the input parameters and then returns the results when it is complete? Thank you for your help.

I have another optimization routine that works just like fminsearch (meaning that it accepts a function as one of its arguments) developed external to MathWorks that I would also like to try out to see how well it does on this optimization problem.

[Walter Roberson]

See https://www.mathworks.com/help/matlab/call-cpp-library-functions.html

Answer 1

[Deepak]

Hi @Troy Rockwood,

I understand that you want to use "fminsearch" in MATLAB to optimize a black-box C++ simulation that runs on an HPC cluster. An approach is to create a MATLAB wrapper function that takes input parameters, writes them to a file, submits the simulation as a batch job using a scheduler like SLURM (sbatch), waits for the output file to be generated, and then reads the result. This function acts like any regular MATLAB function and can be passed to "fminsearch" or other optimizers that accept function handles.

Below is a simple structure for the wrapper:

function fval = simWrapper(x)
    writematrix(x, 'input.txt');         % Write input
    system('sbatch run_sim.sh');         % Submit job
    while ~isfile('output.txt'), pause(60); end  % Wait
    fval = load('output.txt');           % Read output
end

We can then call "fminsearch(@simWrapper, x0)" as usual. This lets MATLAB control the optimization while the actual evaluation runs externally on the cluster.

Please find attached the relevant documentations for reference:

fminsearch: www.mathworks.com/help/matlab/ref/fminsearch.html

system: www.mathworks.com/help/matlab/ref/system.html

I hope this helps.