For loop pauses and restarts randomly

I have a simple code (below) that contains two for loops. This code computes the power spectrum of a signal and fits a slope to the log-log spectrum.

m=zeros(100,100);
for x=1:100
for y=1:100
curr=squeeze(data(y,x,:));
[pcurr,f]=pwelch(curr,[],[],[],1);
pp=polyfit(log10(f),log10(pcurr),1);
m(y,x)=pp(1);
end
end

The code will run and use up about 60% of the CPU for some amount of time, 30 seconds is probably about the max, and then will pause and do nothing for a seemingly random amount of time, sometimes 5-10 minutes or longer before it starts up again and runs for a few seconds. The total run time should only be a few minutes, but it ends up taking hours.

This doesn't happen with only this code, but with other functions inside a for-loop that will utilize more then one processor at a time such as 'imfilter'. It will also happens when I run the code as a function or in the command window.

I am running Matlab r2011b on a 12-core Intel system.

If anyone has any idea what I could do so that the program does not pause please let me know.

[Sean de Wolski]

Could you wrap a tic and toc around the innards of the for-loop and have it print the time for each iteration and x/y. I'm curious if there are certain iterations that repeatedly take longer.

Answer 1

[the cyclist]

Without looking at your code in detail, I can say that that behavior is typical of what happens when the memory needed by your code transitions from RAM into virtual memory. The bottleneck becomes the memory access, so the CPU usage drops radically. You might want to look at memory usage.

(I don't have the signal processing toolbox, so I can't investigate more.)

[Jason Ross]

Note that if you are using Windows 7, the "Resource Monitor" aggregates all the cpu, memory, disk I/O and netwotk I/O very nicely. You can launch it right from the Task Manager and see if the cyclist's theory lines up with what's happening.

[Christopher]

Yes it was a problem with the virtual memory, the maximum paging file size was set to only 16MB. I increased this to align the physical ram, and the program ran continuously.

Thanks!

[Jason Ross]

Unsolicited advice: If you are running a 64-bit OS, investigate if you can get more RAM. 16 GB can be had for under $100 for some types of RAM, and not going into swap will definitely increase the performance of your code, as you've already demonstrated that you are running out of it.

[Christopher]

I actually don't think it is an issue just with physical ram. I have 52GB in this system, and that isn't anywhere near used up when this happens. It seems to be an issue when Matlab uses multiple cores and ends up using the virtual cores for certain functions. It doesn't happen when I use matlab pool, only when functions automatically use multiple cores.

Do you know if there is a way to keep matlab on physical cores only without completely shutting off intel hyperthreading?

[Jason Ross]

I am not aware of one without turning off the hyperthreading in the BIOS. I know for matlabpool the default pool size is the number of physical cores, so that's likely why you don't see it there.

Good to know you have a decent amount of RAM in your system.

Answer 2

[Ed Callway]

Fascinating - I have the same issue in 2021! Programs just stall randomly.

The setup:

• Win 10 up to date, 4TB SSD with a ton of space, 32 GB DRAM, 6 core 12 thread Ryzen 3600X, no overclocking or BSOD, monitoring speed and temps and fans fine, multi-day borrowed license for offline work, internet disconnected to avoid distractions.
• Running 6 copies of a single script in 6 instances of matlab R2020b from a batch file like this:
• matlab /r "someScriptName (parameterSet1a, parameterSet1b)"
• matlab /r "someScriptName (parameterSet2a, parameterSet2b)" etc etc to set 6a&b
• Matlab app writes nothing to disk till it finishes, so disk idle
• Those 6 copies of everything using just 7GB of 32 GB, no memory thrashing
• Total CPU load ~66%, each matlab process taking about 10%, command windows printing out progress, all well and good
• The script contains loops doing fairly normal math, and one of the loops does use parfor (gives about a 10% speed boost)
• At the end of one of the bigger loops (about once a second) it does a tik tok to print out time elapsed and estimated finish time

The problem:

Runs fine for about 5 hours out of the 24 it needs, then every 10 minutes or so the 6 instances stop one at a time about 10 seconds apart, stop printing to command window, and their CPU usage each drops to 0%.

• After a minute or 2 they start up again one at a time and run fine for about another 10 minutes.
• Note from Task Manager I know the order they were started in. They ALWAYS stop running starting from the last one started (hours ago), progressing to the 1st one started. When they start up again it's also always in that same order.
• I can start CPU-Z and run an all core stress that pushes the CPU to 100% and drops the matlab load to 50%. Temps remain fine, doesn't affect the matlab copies, running or stalled.
• I can manually start and run another matlab instance when the other 6 are stalled and it's fine
• Any other program I start when the matlabs are idled runs fine too.
• Renabling the network, running a browser etc also makes no difference

So I see nothing on power, voltage, temps, network, cpu load, disk or memory load that looks bad...except matlab randomly sleeps.

The code itself is long and proprietary, can't put it up here. Would love to hear things to avoid, other similar stories with clues, or tiny cut down scripts to try.

As Leeloo Dallas said, plz halp

[Ed Callway]

I think I found a partial answer after a lot of digging.

Normally if code includes a toolbox without an available license (expired, not enough network licences, not online to license server, not borrowed etc etc) the code halts and says so. And that's fine.

BUT if the parallel tool box is not available there may be no message!

I managed on one run to catch the program in one of those "undocumented and reserved for internal use" routines that said something like "if there is no parallel toolbox, keep trying for a long time" or something like that.

Rather than random 5 minute stalls, I would be much happier if:

1. The program halted and said "please get a toolbox license"
2. The program ignored parfor and went back to serial

Sorry I didn't save the routine, but someone familiar with the parallel toolbox internals may know.