simple copy task much slower with high memory use, workaround possible?
9 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
All memory usage is as reported by the function "memory".
I found that my algorithm is much slower after running for some time, and the reason is Matlab takes longer for a quite simple copy task, when it already uses a lot of memory. In my case Matlab uses 9.9 GB RAM of 16 GB. When copying matrices like
tic; for i = 1:10000; t = a(1:n); end; toc
and plotting the resulting time over n, the result is:
On an Matlab instance with 5.5 GB RAM usage, the same leads to this instead:
So some kind of very time-intensive memory handling sets in with the 9.9GB-Matlab when copying more than 2000 entries. Why? What can I do to work around that? Using less memory is an option but adds time for file handling, so I'd like to keep using ~10GB.
Edit: I tried generating enough random data from scratch to reach 9.9GB usage, the problem doesn't arise in that case! This seems to be a reproducale bug in memory handling with my specific 9.9GB of data!
2nd edit: saved to disk, the "real" 9.9GB data, make up 15.2 GB data. The "fake" 9.9 GB data are only 7.5GB on disk. Maybe that is important somehow?
10 comentarios
Walter Roberson
el 15 de Mzo. de 2021
-v7.3 files are stored in HDF 5, and there is a notable amount of overhead for container datatypes such as cell arrays or struct; storing pure numeric arrays is not nearly as bad.
Respuestas (2)
Jochen Schuettler
el 11 de Mzo. de 2021
Editada: Jochen Schuettler
el 11 de Mzo. de 2021
4 comentarios
Jan
el 15 de Mzo. de 2021
Why does task manager show so much less memory use than matlabs "memory"?
The term "memory usage" is not uniquely defined.
x = rand(1, 1e6);
x = 5
Now the large memory block reserved by the first command is free'd, but maybe the memory manager of the OS did not overwrite it with zeros already and therefore it is not available for other programs. Now it is a question of taste, if this memory belongs to Matlab or not. In theory the OS can decide by its own when to clear the contents. Usually it does this on demand or on idle only.
In addition the OS stores a file cache for each program. Does this belong to the application or to the operating system?
Would it help to declare the intermediate data as global/persistent, so we don't need for-loops?
Maybe. You did not post a section of your code, which would clarify exactly what you are doing. So I could speculate only. It is more reliable, if you try this. Remember, that the behavior can change with the Matlab version, the OS and the avalaible free RAM.
Jochen Schuettler
el 16 de Mzo. de 2021
Editada: Jochen Schuettler
el 16 de Mzo. de 2021
7 comentarios
Jan
el 19 de Mzo. de 2021
In this forum "closing" means, that a question is removed soon. So we close questions only, if they contain too few information to be answered.
Ver también
Categorías
Más información sobre Logical en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!