Every program in newer versions of Matlab runs slower, Why?

Every program in newer versions of Matlab runs slower and slower. I have measured with external stop watches also. And it does drop frame rate in vision processing which is very obvious from human observation too.

Bench (Same matlab, different computers) produces same results butactual program runs slower measured with external stop watches.

Bench (different Matlab vesrions, same PC) produces rsults which match external stop watches.

Also newer figure Uifigure and widgets are too slow.

I have swicth to old matlab for faster execution.

Why is it happening?

[Adam Danz]

I can't answer your question but it would be helpful to know which old and new versions you're comparing.

[Khalid Mahmood]

I have always compared all version since 1997. Currenlty 2020b runs far slower than 2017b, 2017b and 2018b almost same speed

[Walter Roberson]

R2021b should have faster App Designer than earlier releases; they worked hard to make performance improvements.

Answer 1

[Rik]

Some things get slower, some things get faster. It really matters what exact functions you're calling.

If I write a function where time might be an important factor, I tend to write my own performance tester. Feel free to have a look at one of my functions: ComputeNonCryptHash. If you look at the performance section (the HTML doc is the 'examples' tab), you can compare the relative speed.

[Khalid Mahmood]

Its obvious that somethings must get faster and some must get slower for advanced algorithms, generality and accuracy of results. But Mathworks must bech mark its version of toolboxes and publish results. Its only them who can tell with more authenticity and accuracy. Every newer version provoided more functionality with advancement in research, it's obvious. But speed benchmark is missing or it is not considered important. Anyhow Matlab is best when it comes to Simulation, Scripting and Debugging, interoparability with C++, Python, openCV( though with some limitations in OpenCV interoparability).

[Walter Roberson]

@Khalid Mahmood

Could you give some examples of companies working in comparable fields that publish official benchmarks?

[Rik]

There isn't really much point in publishing benchmarks for all functions, because then you would have the tedious job of putting everything together. And even then, the performance changes based on how often the function has been called with a comparable syntax (or the exact same data). The only logical thing is to let you write your own benchmark.

Mathworks does provide a benchmarking tool to compare between different computers: bench.

Matlab is such a multi-purpose tool that I doubt publishing benchmarks with enough detail for each function would be feasible, while anything less would be meaningless.

[Khalid Mahmood]

I don't think its always a tedious job. Who ever writes code for newer version/updates can run bench on older version vs newer version and fastest of all versions--only 3 bench marks .. as a guide or pathway for curiuos fellows.. ;leave the rest for users . Matlab will get a huge boost ( don't fear of fall, its out of question). Speed boost is what is always required when resources are limited and time is short; but at what compromise, this part is a bit more challenging. Matlab is already doing wonders. Why not make a new leap forward if other companies are failing to provoide benchmarks.

Matlab already proides a lot of examples for speeding up, tuning, etc. Only extra thing will be carefully selecting test/validation data and running bench. This will definitely open up a huge era of most suitable use cases of versions, functions, alternative approaches... research will get another booster.

I don't underestimte Matlab developer team, they are far better than ordinary developers. They know better, and I might be more valid in logic than just annoying developers.

[Khalid Mahmood]

One more thing, thanks to mex functions, I use C and compile it to mex code for speed up. This makes it more independent of Matlab versions. I had to write my own version of HOG which is 30x+ faster than matlab's extractHogFeatures, with my realtime tracker. I also had to change many installed scripts related to imaq hardware to make video capture and processing at least 6 times faster. I can submit these to developers, they can review and add to Matlab. I prefer Matlab over Python, C++ when huge boost in results is required with fast paced research and silumations. That's why I use C inside Matlab to benefit from speed and Simulation and debugging at the same time.

[Rik]

I think you underestimate how hard it is to write a good benchmark function. What kind of data are you going to use? What sort of procedural generation are you going to use? How many edge cases are you going to include?

I don't doubt they have internal tools to assess performance, but I can understand a hesitancy to share such things. They also depend on hardware, not just code. Most things in Matlab run on a single thread, but some things can run in parallel, in which case the speed up will depend on your core count. There are just a huge amount of variables.

It is not that I think it is tedious to run the benchmarks, because I don't. They developped a testing framework. Do you really think there would be any human interaction when running the test suite? I just code for myself, and even I have created tools that avoid manual interaction.

Just a final note: the things bench does is not guaranteed to stay the same between versions. Mathworks changes it every now and then to reflect changes in expected hardware and workloads. (like how pacman is not used to benchmark the latest generation of GPUs)

[Michelle Hirsch]

I can confirm that we do extensive performance testing throughout our product family. We run unit-level performance tests and application-level tests.

It's not that hard to write a performance test - the hard part is making sense out of it, so I agree with Rik and others that it's not that useful to publish all performance tests. I'm the head of product for MATLAB and I find most unit level tests impenetrable. They are essential for locking down performance, but aren't helpful for figuring out the impact of changes on the code that I care about. Application-level tests can be much more interesting, but aren't easily generalizable to other applications.

We have been increasingly publishing performance release notes, but these focus on just the good news - things that we've made faster.

@Khalid Mahmood, I encourage you to reach out to MathWorks support with the specific issues you are encountering with slowdowns from release to release. They may be able to suggest a workaround and they may pass along helpful feedback to development to address the issues in future releases.

You've also identified many functions you've made appreciably faster. This could prove to be helpful input to development, too. Though I'll caution that I often see that there's a tradeoff between generality and performance. Once I make the limiting assumptions that are valid for my code, I have the potential to bake these assumptions into the code I call to get better performance. But the changes I make might not be appopriate for other users of the same functions - they might slow down different use cases, stop supporting other use cases altogether, reduce numerical stability, etc ...

[Steven Lord]

I agree 100% with what Michelle wrote. I've written more than a few unit-level performance tests over the nearly 19 years I've been in the MathWorks Quality Engineering department. Based on my experience publishing performance data would not be as simple as writing a simple function or class, publishing a one or two page report tabulating the data, and calling it a day.

As Rik mentioned (There are just a huge amount of variables.) even a simple function can require a large collection of unit-level performance test cases that sweep over several dimensions including but not limited to:

• problem sizes (small, medium, and large, so we can evaluate how a function's performance scales)
• data types (if there are different code paths used for double precision, single precision, integer data, logical data, etc.)
• sparsity (sparse matrices are stored differently in memory than full matrices and require or can benefit from processing by algorithms that can take advantage of the storage format)
• and sometimes properties of the data itself. See the Algorithms section of the documentation page for the mldivide function aka the \ operator for how it handles matrices with different shapes and properties.

And that's just for one function. Looking at the function list for MATLAB in the documentation the Mathematics category alone has 558 functions. [That's the one with which I'm most familiar.] Granted, not all of those 558 functions would require the same amount of performance testing. At the simplest extreme, something like the pi function doesn't need to have a performance test. :) But you'd get positively buried in data if we reported on decent sized subsets of the 558 Mathematics functions and the 534 Data Import and Analysis functions and the 539 Language Fundamentals functions and the ...

And that's not even touching upon the many toolboxes we have.

[Khalid Mahmood]

You are true and I respect what you want to say! Actually every new release has to add extra functionality, more parameters.. that optimised still becuase of more polished but general means more clock ticks and powerful. That adds complexity and ease too and every new generation needs faster hardware!