why does "save -append" work so slow and increase the size of mat file for simple replacement?

Hi,

I am in this situation for couple of times but have no clue. I can understand that when there are structures in my data, saving may take long time due to decompression. However, what confused me is that, even when I replace the existing structures only, without creating any new elements, the updated mat file increased a lot. It looks like the old structures are not erased at all. Could anyone help me out here? Thanks a lot.

Answer 1

[Benjamin Avants]

If you use the -append option, save will not overwrite any of your existing data. It will simply add the new data into the existing file. This can/will duplicate variables and structures already stored in the file.

If you want to change some of the values and not others, you should open the file and load the existing data. Then change the values you want to change in the imported data before overwriting the old file with the modified data. This is faster than appending an existing file and you don't get duplicate entries.

If you have the full data set in memory that you wish to save, simply overwrite the existing file with the data in memory. This would be the fastest way if it is possible in your code.

[Diderot]

Thanks a lot.

Could you please clarify the following? If as you said the -append option only duplicate variables, then next time when I load these variables, which version will be loaded? The old ones or the duplicated new ones?

My problem is that I have 30+ objects in one data set and I only update a few of them. If I use "save data.mat a b", where a and b are the two updated variables, then the new data.mat only contain a and b. All other variables in the old data.mat will be lost.

Is there any way to update a few variables without losing others?

Answer 2

[Diderot]

According to this webpage, -append option seems to overwrite the old variables: http://matlab.izmiran.ru/help/techdoc/matlab_prog/ch10_in7.html#119024

Appending to an Existing File

You can add new variables to those already stored in an existing MAT-file by using save -append. When you append to a MAT-file, MATLAB first looks in the designated file for each variable name specified in the argument list, or for all variables if no specific variable names are specified. Based on that information, MATLAB does both of the following:

For each variable that already exists in the MAT-file, MATLAB overwrites its saved value with the new value taken from the workspace.

For each variable not found in the MAT-file, MATLAB adds that variable to the file and stores its value from the workspace.