How to read only one potion of HDF5 files in Matlab ?

Dear friend,

I am trying to open a very big (20G) hdf5 file in matlab. But the file is too big, so I hope there is a way I can open just samll part of the matrix. For example, the matric is 2x1024x2555904, I wish to only read or access part of the matrix. it could be just 2x1024x(1:100) something like that.

If I do h5disp('GOLD_XYZ_OSC.0001_1024.hdf5')

HDF5 GOLD_XYZ_OSC.0001_1024.hdf5 
Group '/' 
    Dataset 'X' 
        Size:  2x1024x2555904
        MaxSize:  2x1024x2555904
        Datatype:   H5T_IEEE_F32LE (single)
        ChunkSize:  []
        Filters:  none
        FillValue:  0.000000
    Dataset 'Y' 
        Size:  24x2555904
        MaxSize:  24x2555904
        Datatype:   H5T_STD_I64LE (int64)
        ChunkSize:  []
        Filters:  none
        FillValue:  0
    Dataset 'Z' 
        Size:  1x2555904
        MaxSize:  1x2555904
        Datatype:   H5T_STD_I64LE (int64)
        ChunkSize:  []
        Filters:  none
        FillValue:  0

I do can access Y or Z, because they are small. e.g dset = hdf5read('GOLD_XYZ_OSC.0001_1024.hdf5','/Z');

X here is the 20G data, so,Matlab cannot let me do that, because is says its too big. Is there a way I can just access or read part of data ? E.g, just show 3rd dim from 1:100, so output size is 2x1024x100 in X ?

Btw, I tried dset2 = h5read('GOLD_XYZ_OSC.0001_1024.hdf5','/X',[2 2 1],[2 2 10]); I got error: The index arguments exceed the size of the dataset. Why, it is in the range, right?

If yes, someone can teach me the code to do that? Thank you!

update: the new question is, how to take a chuck of X but not from very beginning? For example, I want the check dset3 = X(:,:,101:200) ? All value in 1st dim, all value in 2nd dim, but 3rd dim is from 101 to 200. Thanks.

[Pavel Inchin]

Good day,

First brackets represent starting position, second ones - number of element to count (from starting position). So expression: [2 2 1],[2 2 10] means that you want to read from 2nd value of 1st dimension to 3rd element of 1st dimension - which doesn't exist.

Lets say you want to read 2x1024x100 of 2x1024x2555904. The solution is:

dset2 = h5read('GOLD_XYZ_OSC.0001_1024.hdf5','/X',[1 1 1],[2 1024 100]), and thus your dset2 will be 2x1024x100 starting from [1 1 1].

Hope this helps

[Sunson29]

My friend, you helped me !!!!

Now, by using

dset2 = h5read('GOLD_XYZ_OSC.0001_1024.hdf5','/X',[1 1 1],[2 1024 100]),

I have first chuck of 2x1024x(1:100) as in, 1dim, 1:2, 2nd dim: 1:1024, 3rd dim: 1:100. Hope my undertstanding is right,

Allow me to ask, what should I do if I need 2nd chuck, but not from the very beginning ?

For example, when I do

dset3 = h5read('GOLD_XYZ_OSC.0001_1024.hdf5','/X',[1 1 101],[2 1024 200]),

the size of dset3 is 2x1024x200! not 2x1024x100 ? It seems that, it take the chuck from very first to 200, not 100 ( in other words, I want the data is 1dim, 1:2, 2nd dim: 1:1024, 3rd dim: 101:200). I feel like my understadning is not very right. Could you please advise me how to take a chuck in the "middle" of this 20G 3d matrix ? Thank you!!!!!!

update: I think I should do [1 1 101] [2 1024 100]. so, start at 1 1 101, and count 2 1024 100 ? this means X(:, :, 101:200) right?

[Pavel Inchin]

yes, you are right. As I wrote earlier, in second brackets you have to set number of elements you want to get startring from 101 (in your case), so yes [1 1 101] [2 1024 100] will give you [1:2,1:1024,101:200]

[Sunson29]

thank you!