Run serial code on parallel cluster interactively

Hi, I need help from the experts.

Suppose I have a slow machine, and I have an access to a fast parallel cluster. Let say my code is fully serial - no parallel language constructs like parfor or spmd. If I want to run the code on the cluster, of course I can submit a batch job using batch function. But, what if I want to run interactively? So far, interactive processing only allows parallel programming, but if my machine is extremely slow, using even one worker on the cluster is still benificial.

What I do currently is using this workaround:

parpool('MyCluster', 1);

spmd (1)

MyVariable = myFunction.

end

MyVariable = MyVariable{:}

I am looking for a more elegant (official?) solution. Using an spmd as a workaround is far from elegant: (i) codes need to be enclosed by spmd blocks, (ii) variables created in spmd blocks are Composite objects that must be indexed by curly braces first before they can be used/displayed normally.

Please help and suggest a better way. Thanks!

[Damian Pietrus]

Hey Khairul, I'll start with one potential option. Do you have a way of starting MATLAB directly on the parallel cluster? Often time cluster schedulers have a way of requesting an interactive session directly on a compute node, where you can then launch MATLAB and run your serial code. If you're unfamiliar with this process, let me know what scheduler the cluster is using and I can hopefully point you in the right direction.

[Khairul Adib Yusof]

Hi Damian. I can physically interact with the cluster machine, so I can open MATLAB directly on the cluster machine. It uses MJS. Not sure if this is what you're asking.

Thank you.

[Damian Pietrus]

Hey Khairul,

I spoke with some colleagues to make sure I wasn't overlooking anything, and there isn't a great option for running serial code interactively on the cluster in the way you're looking for. The main options are the ones you've already mentioned -- using batch jobs or putting your code in an spmd block.

As I mentioned in my last comment, one potential option would be to launch MATLAB directly on one of the compute nodes on the cluster. The only caution I have with this approach is making sure that you don't cause any resourse contention on the cluster. MJS is only aware of jobs in the Parallel Server queue and nothing else running on the systems. If the queue is busy and every machine is busy running jobs, starting MATLAB manually on a compute node outside of the scheduler queue would slow both the running job and your MATLAB session. You could potentially talk to the person who manages the cluster (if that's not you) to set aside an entire machine or perhaps a handful of cores for you.

[Khairul Adib Yusof]

Dear Damian,

Thanks for the effort. Can you elaborate further the approach that you describe? How do I launch MATLAB on compute node from a client machine? And if I can do that, how do I interact with the opened MATLAB session from the client machnie? Is it like remote desktop where I control the mouse and keyboard? Your help regarding this is much appreciated.

Regardless, I really think that a native implementation of interactive serial code execution on a parallel cluster to be a helpful feature that people in MathWorks could consider working on.

Thank you!

[Damian Pietrus]

Yes, if you wanted to launch MATLAB directly on a compute node, you'd have to connect to it via RDP or SSH depending on it's operating system. You'd then treat it the same as any other MATLAB session, though I'll warn again that you should be cautious of resource contention if you were to try this.

I've put in an internal enhancement request for this type of interactive workflow from a remote client with MATLAB Job Scheduler so our dev team can take it under consideration.

[Khairul Adib Yusof]

Thank you Damian for the tremendous help! Looking forward to a better implementation of parallel computing in this manner.