Clarification regarding 2GB transfer limit between MATLAB and Python

Could someone please clarify what exactly is meant by 'The size of variables passed between Python and MATLAB is limited to 2 GB when you call a Python function out-of-process. This limit applies to the data plus supporting information passed between the processes.' on https://uk.mathworks.com/help/matlab/matlab_external/limitations-to-python-support.html?

I want to run a sensitivity study using the MATLAB-based UQLab library (https://www.uqlab.com/), which comprises 12 evaluations of a custom Python model spanning many files. The Python model (Python 3.9) is treated as a black box function, which is executed from MATLAB as follows (NOTE: pyenv(ExecutionMode = "OutOfProcess")):

pyrunfile(strcat("python_file.py ", input_variables), output_variable);

where

"python_file.py " is the master script managing my Python model

input_variables is a JSON string of model inputs

output_variable is a single float constituting the model output

Halfway through the 11th model evaluation, MATLAB returns a Python package-specific error ("Python Error: TypeError: Numerical inputs to HAPropsSI must be ints, floats, lists, or 1D numpy arrays."). Now, when I ONLY run the 11th model evaluation I don't get an error, and the results of the individual evaluation are identical to those from the full set of evaluations up to the point where the latter failed. Moreover, when I run all 12 evaluations on a subset of my model (less computationally/memory intensive), I don't get an error either. After repeating the aforementioned steps multiple times, the error message always occurred in the exact same location, halfway through the 11th model evaluation.

All this suggests to me that I am exceeding the 2GB transfer limit between MATLAB and Python, but I don't understand what is meant by 'size of variables passed between Python and MATLAB'? The inputs and outputs described above are only a few byte, but I wouldn't know what falls under 'supporting information'.

Answer 1

[Sarthak]

Hi Nils,

I think you are facing this error because you are transferring full simulation data for each run to MATLAB which is exceeding the transfer limit. Since you want to run a sensitivity analysis, methods such as Sobol’ indices are usually applied to a limited set of relevant quantities of interest post-processed from your simulation. I believe only this information should be transferred and not the full simulation data for each run which should resolve this error.

I hope this helps!

[Nils Barner]

Dear Sarthak,

Thank you for your answer. I am pretty certain that I am only transferring the relevant quantities required to calculate Sobol' indices, which should amount to a few kilobytes at most.

Best wishes,

Nils