how to store large data during the execution of a program without getting out of memory

1 visualización (últimos 30 días)
I am working on partitions. while i run program that generates partition it get out of memory. What i want to do is to store these partitions in a data store and use tall arrays to work on chunks to filter required partitions. partitions are stored in cell arrays. Thanks
  4 comentarios
Walter Roberson
Walter Roberson el 14 de Oct. de 2019
If I were to ask you for partition #107 would you be able to calculate it without calculating all the ones before it in the same call?
asim nadeem
asim nadeem el 14 de Oct. de 2019
No the function I have creates a cell array of size stirling no of 2nd kind and stores partitions in it. I can restrict the size to be less than the above size say like 10 partitions but then remaining will not be calculated. Thanks

Iniciar sesión para comentar.

Respuesta aceptada

Walter Roberson
Walter Roberson el 15 de Oct. de 2019
You will need to go through calculating the all partitions and writing them to a file, but keeping only a limited working set in memory as you do so. After you have written them all to file, you can create a datastore from the data file and create a tall array from that and start working on it.
There are some methods of building partitions that work recursively, creating partitions of size 1, extending to size 2, then to size 3, and so on, with (for example) all the partitions of size 19 needed to calculate the partitions of size 20. You would want to create as large a partition in memory as your memory could afford before writing that to a file and proceeding through tall array methods to addition additional columns, creating a file as you go, going back and making that file a datastore, and so on. It would probably be a bit of a nuisance, but It should be possible... I think.
Matters would be easier to arrange if you had an algorithm that could translate from an index to a particular partition; perhaps Knuth volume 4A; https://math.stackexchange.com/questions/222780/enumeration-of-set-partitions

Más respuestas (0)

Categorías

Más información sobre Large Files and Big Data en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by