Loading Time difference between files containing timetables and files containing numeric values
Mostrar comentarios más antiguos
TL;DR
The who and load function are very slow when loading timetable data compared to loading numeric data. Why? Is this maybe a bug? How can I speed up these functions for timetable?
Background
I have a huge file (>1GB) containing various sensor logs of different frequencies. Right now the values and times are stored seperatly as numeric double vectors. A value vector can be mapped to a time vector through the variable name of the value vector, e.g.: "velocity_t32" can be mapped to the time vector "t32". The timestamps of the logs are irregular -> They can't be compressed into starttime+timesteps.
Now I want to switch to MATLAB timetables since they seem to be more convenient to work with. I have run some experiments to compare the two formats against each other and I found out that the matlab load and who functions run significant slower when loading a file containing timetables than loading a file containing only numeric values. I attached the live-script.
Generation of sample-files
%% Code to generate sample-files (I can't share original data, this should be close enough)
signals_as_timetables = struct();
signals_as_numeric = struct();
num_signals = 2000;
length_run_in_ms = uint64(1000000); % 1000s ~ 16min
d = uiprogressdlg(uifigure, Title='Loading Files');
drawnow;
for i = 1:num_signals
d.Value = i / num_signals;
% Mock Sample Rate
n = randi(9, 1, 1) + 1;
e = randi(3, 1, 1) - 1; % 1/3 of signals will be of sample rate 1ms
avg_sample_rate = uint64(n * realpow(10, e)); % in ms, e.g. 2ms, 40ms or 90ms
length_signal = uint64(length_run_in_ms / avg_sample_rate);
% Mock Time-Vector
times = randi(length_run_in_ms, length_signal, 1);
times = sort(times);
% Mock signal data
values = rand(length_signal, 1);
% Mock numeric Format
signals_as_numeric.("signal_XY_t"+i) = values; % e.g. signal_XY_t50
signals_as_numeric.("t"+i) = times; % e.g. signal_50(t50)
% Mock Timetable
times = milliseconds(times) + datetime(date); % Convert numeric to datetime
times.Format = 'yyyy-MM-dd HH:mm:ss.SSS Z';
signals_as_timetables.("signal_"+i) = timetable(times, values);
end
close(d);
save("timetables.mat", '-struct', 'signals_as_timetables', '-v7.3'); % file size: ca. 1.5 GB
save("numeric.mat", '-struct', 'signals_as_numeric', '-v7.3'); % file size: ca. 1.4 GB
Preview of both files:


Experiments
% Compare listing all variable names
tic
signal_names = who("-file", "timetables.mat");
toc % Elapsed time is 10.471696 seconds.
tic
signal_names = who("-file", "numeric.mat");
toc % Elapsed time is 0.163475 seconds.
As you can see there is a difference of factor 65 when loading the variable names.
% Compare loading complete file
tic
signals = load("timetables.mat");
toc % Elapsed time is 10.293126 seconds.
tic
signals = load("numeric.mat");
toc % Elapsed time is 8.330326 seconds.
Loading the complete timetable file is slower than numeric data, but the difference seems acceptable.
% Compare loading single signal
tic
signal = load("timetables.mat", "signal_1");
toc % Elapsed time is 10.212967 seconds.
tic
signal = load("numeric.mat", "signal_XY_t1");
toc % Elapsed time is 0.079693 seconds.
However, loading a single variable is super fast for the numeric data, but loading timetable data seems to be equally slow like loading the complete file. For this task the numeric data has 145 times faster loading times than timetable data.
Questions
- Why does this behave like this? Is there something I am missing about timetables?
- Is there a better way of storing this kind of data (without a timeseries-database like InfluxDB)?
- How can I speedup the who and load function for timetables?
I am happy about every Answer, Tip, Advice, Experience-Report etc. Thank you! :)
Environment
- MATLAB R2021b Update 3
- Windows 10
- CPU: Intel i7 9700k
- SSD: SanDisk SDSSDH3 2T00 (Read-Speed: 500MB/s)
Respuesta aceptada
Más respuestas (1)
Walter Roberson
el 4 de Jun. de 2022
1 voto
v7.3 files are considerably slower for composite data types.
v7.3 files are based on HDF5, which does not really have a good way to define a composite type, but has the ability to say "okay, for the value of this element, see this internal variable"... through multiple levels.
1 comentario
Tobias Hölzer
el 9 de Jun. de 2022
Categorías
Más información sobre Programming en Centro de ayuda y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!