What are faster alternatives than fread for costumn import patterns?

Hallo Everyone,

I have to import point cloud 'video' - data from a binary file. Each file contains an unknown amount of frames, while each frame contains a number of points that is provided in the 'header' section of the frame.

Currently, I'm using fread to import the data, but as you all know, it is rather slow in loops.

My code looks currently like this:

    frame_index = 1;

    while ~feof(file_id)

        % -- begin body --
        temp.frame_width = fread(file_id,1,'*int');
        temp.frame_height = fread(file_id,1,'*int');
        temp.nPoint = fread(file_id,1,'*int');

        % -- begin point cloud --
        if (temp.nPoint > 0)
          for i = 1:temp.nPoint
            temp.indexes(i) = fread(file_id,1,'*int');
            temp.user_indexes(i) = fread(file_id,1,'*int8');
            temp.points(i,1) = fread(file_id,1,'*int16');
            temp.points(i,2) = fread(file_id,1,'*int16');
            temp.points(i,3) = fread(file_id,1,'*int16');
          end

        else
            temp.indexes = [];
            temp.user_indexes = [];
            temp.points = [];
        end

        % -- point cloud end --
        % -- body end --

        depthClouds(frame_index) = DepthPointCloud(temp.frame_width, temp.frame_height, ...
            temp.nPoint, temp.indexes, temp.user_indexes, ...
            temp.points);

        frame_index = frame_index +1;

    end

I'm happy for every suggestion that could help to advance the speed of the import. Thanks in Advance,

Karen

Answer 1

[Charu]

Hello,

To improve the speed of importing point cloud data from a binary file, instead of reading each data point individually. You can read larger chunks of the data at once to reduces the overhead of the repeated function.

Additionally, you can utilize the MATLAB's ability to handle operations on entire arrays at once, rather than iterating through elements with loops.

Below is a code snippet which uses larger chunks of data at once and pre-allocates the arrays:-

if (temp.nPoint > 0)
    
   % Preallocate arrays
   temp.indexes = zeros(temp.nPoint, 1, 'int32');
   temp.user_indexes = zeros(temp.nPoint, 1, 'int8');
   temp.points = zeros(temp.nPoint, 3, 'int16');
   
   % Batch read all data at once
   temp.indexes = fread(file_id, temp.nPoint,   '*int32');
   temp.user_indexes = fread(file_id, temp.nPoint, '*int8');
   temp.points = fread(file_id, [3, temp.nPoint], '*int16')';
end

For more information, refer to the following MathWorks Documentation:

“fread” function:

https://www.mathworks.com/help/releases/r2024b/matlab/ref/fread.html

[Walter Roberson]

This is incorrect.

The original byte stream after the header looks like

4 byte int, 1 byte int, 2 byte int, 2 byte int, 2 byte int

4 byte int, 1 byte int, 2 byte int, 2 byte int, 2 byte int

4 byte int, 1 byte int, 2 byte int, 2 byte int, 2 byte int

... N times

Whereas you have coded for

N by 4 byte int, N by 1 byte int, 3*N by 2 byte integer

To have a chance of working properly, you would need to ftell() the current location, frewind() to the saved location, fread() with the "skip" parameter set, frewind to the saved location, move forward 4 bytes, fread() with the "skip" parameter set, frewind to the saved location, move forward by 5 bytes, fread() with the "skip" parameter set, and so on. The three 2 byte int groups would have to be done seperately.

Answer 2

[Walter Roberson]

After the reading of temp.nPoint,

bytes = fread(file_id, 11*nPoint, '*uint8');
indices_uint32 = uint32(bytes(1:11:end))*16^0 + uint32(bytes(2:11:end))*16^1 + uint32(bytes(3:11:end))*16^2 + uint32(bytes(4:11:end))*16^3;
indices = typecast(indices_uint32, 'int32');
user_indices = typecast(bytes(5:11:end), 'int8');
points(:,1) = typecast(uint16(bytes(6:11:end))*16^0 + uint16(bytes(7:11:end))*16^1, 'int16');
points(:,2) = typecast(uint16(bytes(8:11:end))*16^0 + uint16(bytes(9:11:end))*16^1, 'int16');
points(:,3) = typecast(uint16(bytes(10:11:end))*16^0 + uint16(bytes(11:11:end))*16^1, 'int16');