Servicing Python Matlab API queries

Hello All,

I am dabbling into the python matlab api. I'd like to use the API to enable a python program that is spawned indirectly as a result of a system call from matlab, and allow that python program to interact with the parent matlab session using the API. Something like this:

function call_external_program
    matlab.engine.shareEngine; % share this engine so that we can connect to it from python script.
    
    %launch the program in the background
    if ispc
        %windows
        [status , result] = system('START wrapper_program.exe') %<-- note that wrapper_program.exe spawns a python program that connects back to this session...
    else
        %linux
        [status , result] = system('wrapper_program &') %<-- note that wrapper_program.exe spawns a python program that connects back to this session...
    end
    
    % wait until we detect termination of the wrapper_program.  during this time, we expect to be servicing calls to python_api_entrypoint coming from 
    % a python program that was spawned by wrapper_program...
    while 1
        %Hmm.. how can I pause here in such a way that Matlab can service the Python API?
        % It only works if I put a breakpoint here!
       pause(0.1)
       
       if detected_wrapper_program_completion || timeout %don't ask how we really do this...
           break;
       end
    end
    
end

function dataOut = python_api_entrypoint(dataIn)
    %example algorithm that we use matlab for...
    dataOut = sum(dataIn);
end

Unfortunately, the above scheme only works if I put a breakpoint at the 'pause(0.1)' line, presumably because matlab control returns to the command line. Otherwise, matlab will not service a call from the python program unless we quickly breakout of the parent function and return matlab control to the command line. This smells like something drawnow() could help with, but no such luck. Any ideas?

[Dario Ringach]

Hi Jesse,

Have you had any luck working this out? I am running into a similar situation, but pause() does not help either.

Thanks for any tips...

--

Dario

[Jesse Hopkins]

Hi Dario -

We do have quite a nasty hack in place that gives some functionality. In a nutshell, use a timer object to do some monitoring/processing of the external program while matlab control remains at the command prompt. Some more detail:

a. Data needed for the matlab function called from python is stored in a singleton matlab handle object. It is retrieved like:

function dataOut = python_api_entrypoint(dataIn)
    % get important state data from matlab
    stateDataObj = MyStateDataClass.getSingleton();
    
    %example algorithm that we use matlab for...
    dataOut = stateDataObj.dosomething(dataIn);
end

b. In order for the matlab/python API to service queries, seems that Matlab must be at an interactive command prompt. Never could get anything like pause/etc.. to work. So the matlab function that launches the external program must return back to the command prompt.

c. Note that (b) imposes a severe limitation. It is not possible in this scheme to call said matlab function from any other matlab script/function, as it must return back to the matlab command prompt in order for matlab to service the upcoming calls to the matlab/python API [ Have some ideas on addressing this limitation later..]

d. Needed to split the top-level matlab function mentioned in (b) into some parts:

1. Prepare to call the external program (i.e. create/update the singleton state object)
2. Setup a timer object to monitor the external program. This timer object will be started just before the top level function returns back to command prompt
3. Spawn the external program via system
4. Start timer object initialized by (2)
5. Top-level function returns to command line
6. External program continues to run
7. Timer object monitors execution of external program, deletes itself upon termination of external program.

Clear as mud?

I think that the limitation mentioned in (c) could be worked-around by spawning a completely seperate instance of matlab, which if setup with necessary state information could perhaps service the python API queries. It would need to be spawned and just sit and wait at the command line. Haven't gotten around to trying that.

[Joseph Owen]

I have this same issue. The sharedEngine calls in Python will hang unless you're at a prompt in Matlab (i.e. debugger breakpoint). It makes the sharedEngine useless for any sort of automatic python/Matlab interaction.

Answer 1

[Mohammad Sami]

I would suggest use .NET or Java to run the external process. This would return you a .NET or java object which have functions to check the external process you started has completed. You can then add an event listener to the process object which then calls your code to do stuff once the wrapper has exited. Ideally .NET will raise the "Exited" event which Matlab can capture using event listener and run the rest of your code.

Alternatively you can combine this what Jesse suggest on using either timers or starting a second matlab session in shared mode using startup arguments. So you create two process i.e shared matlab instance and your wrapper and you can monitor these two processes on your intial session.

https://www.mathworks.com/help/matlab/matlab_external/connect-python-to-running-matlab-session.html

https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.process?view=net-8.0

% .net example
process = System.Diagnostics.Process();
process.StartInfo.FileName = 'matlab.exe';
process.StartInfo.Arguments = '-r "matlab.engine.shareEngine"';
% additional parameter you may want to specify
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardInput = false;
process.StartInfo.CreateNoWindow = true;

% start the process
process.Start();

% listen for the exit event and run some other code
% you may need to try if the event listeners work for you
addlistener(process,'Exited',@do_exit_callback);

% check process has exited
process.HasExited

% if you want to force quit the process
process.Kill();

Answer 2

[Joseph Owen]

In my case I have a Matlab application and a Python application and they need to pass data back and forth synchronously in a single-threaded loop. The sharedEngine process seemed ideal for this, but it turns out it doesn't work in a loop (which is silly - it should work, but it doesn't). Instead, I implemented a simple tcp link based on this suggestion:

https://www.mathworks.com/matlabcentral/answers/1783635-sending-information-to-matlab-from-python-via-tcp

This worked like a charm, and is really simple if all you're doing is passing semaphore flags back and forth.