File size continuously increase when use h5D.write() to update some data block in original place.

14 visualizaciones (últimos 30 días)
Hi, I am trying to update a data block into a h5 file using h5D.write.
The updated file opens with no problem, but the file size continuously increased about 6MB everytime I update.
The code is below:
% the data block size is [sizeX,sizeY,1], always at the same position.
fid = H5F.open(aPath,'H5F_ACC_RDWR','H5P_DEFAULT');
dset_id = H5D.open(fid,['/Data']);
block = [sizeX,sizeY,1];h5_block = fliplr(block); %the same size as original file
dims = [sizeX,sizeY,1];
mem_space_id = H5S.create_simple(length(dims),h5_block,[]);
space_id = H5D.get_space(dset_id);
aStart = [0,0,0];h5_start = fliplr(aStart);
H5S.select_hyperslab(space_id,'H5S_SELECT_SET',h5_start,[],[],h5_block);
H5D.write(dset_id,'H5ML_DEFAULT',mem_space_id,space_id,'H5P_DEFAULT',aData); %aData is updated data of same size as original file
H5D.close(dset_id);H5F.close(fid);
Since no size expansion is involved, Is there a way to avoid file size increase?

Respuesta aceptada

Walter Roberson
Walter Roberson el 13 de Dic. de 2021
Sorry, no.
When you update, HDF5 writes a new block in, and then marks the old block as no longer in use.
Unfortunately it also has no defined garbage collection, so although the next time around the now-unused block would in theory be large enough to store the data to be written, in practice it may not be.
There is no inherent HDF5 operation to reclaim space.
  3 comentarios
raym
raym el 13 de Dic. de 2021
but another problem is that the increased file size is much larger than original file size.
It may be caused by the different deflate option.
As this is re-write data block instead of create a data set from begining, Is there a way to indicate the deflate option to updated data?
Walter Roberson
Walter Roberson el 13 de Dic. de 2021
If the data is being compressed, then remember that a change in value may result in the compressed data being shorter or longer than the previous version.
h5repack appears to support a command line argument to indicate compression type.
For example -f GZIP=9

Iniciar sesión para comentar.

Más respuestas (0)

Etiquetas

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by