Save data during matlab loop

21 views (last 30 days)
Simon Meyer
Simon Meyer on 11 Dec 2020
Edited: Walter Roberson on 18 Dec 2020
Hello community!
Time for another question :)
My current project is a simulation run by matlab. So mainly it is a call of a "SimulateStep" function in which a lot of different things happen.
One of the things is the call of a huge mex function which gives me some internal data. So, during each call I have a relativly big struct that has a lot of data in it.
And my question is now: What is the most efficient way to save this data during each step and save it with the propriate time? Also, it should be easy to access and read it afterwards.
My current attemp is looking like that:
function saveData(this, time)
% input data
namelist = fieldnames(this.structToSave);
for it = 1: length(namelist)
name = char(namelist(it));
if ~isfield(this.timeSeries, name)
this.timeSeries.(name) = timeseries;
this.timeSeries.(name) = addsample(this.timeSeries.(name), 'Data', this.structToSave.(name), 'Time', time);
It works pretty good. At the end of the simulation I save the "timeSeries" struct to a workspace variable and that's it.
But this attemp is way to slow. If I have a list of ~30-40 signals to save you can heavily feel the simulation getting slower.
Does someone has any better alternative to that? Or is it only my programing that sucks here? ;)
Thanks a lot!
Simon Meyer
Simon Meyer on 16 Dec 2020
Edited: Simon Meyer on 17 Dec 2020
Yes, all the data is numeric. Mainly double, some integer.
There might be also boolean, but it's no problem to convert that to an integer.
But! The element of the struct can be a vector containing doubles.

Sign in to comment.

Accepted Answer

Walter Roberson
Walter Roberson on 17 Dec 2020
Edited: Walter Roberson on 18 Dec 2020
Performance during simulation is important, but performance after the simulation is not as important. So save quickly and process later
function saveData(this, time)
if isfield(this, timeSeries)
this.ntimeSeries = this.ntimeSeries + 1;
this.ntimeSeries = 1;
this.timeSeries = cell(10000,2); %pre-allocate
this.timeSeries(this.ntimeSeries,:) = {time, this.structToSave};
The 10000 pre-allocation is a total guess about how many simulation steps you need. It is best to over-estimate: the overhead for unused rows is only 16 bytes per row, whereas the overhead for growing the cell can be high. (The code can be tweaked for lower overhead for growth.)
After the execution, this.ntimeSeries will record the number of entries actually used, and this.timeSeries will be an N x 2 cell array in which the first column is time information and the second is the struct you wanted to save.
In post-processing you can run though and do the saving.
To do the post-processing saving, I would suggest that you fieldnames() of the second column (the structs) and unique(), and then for each unique field name, run through all of the cell rows that saved that field, accumulate the time information (first cell) in the big vector, accumulate the contents of the vectors (making sure to concatenate along an appropriate dimension) into a 2D array for that variable, and addsample() all of it at one go -- only one addsample() for each different variable.
Of course if you are always saving exactly the same list of variables each time, the logic would be easier.
Simon Meyer
Simon Meyer on 17 Dec 2020
Edited: Simon Meyer on 17 Dec 2020
Hello Walter,
thanks for your help, it works perfectly!
Here is my post-processing:
function my_timeseries = postProcessData(this)
time_vec = this.timeSeries(:,1);
input_vec = this.timeSeries(:,2);
internal_vec = this.timeSeries(:,3);
output_vec = this.timeSeries(:,4);
idx = cellfun(@isempty, time_vec);
time_vec = time_vec(~idx);
input_vec = input_vec(~idx);
internal_vec = internal_vec(~idx);
output_vec = output_vec(~idx);
names = fieldnames(input_vec{1,1});
for i = 1:numel(names)
name = char(names(i));
data = cell2mat(cellfun(@(s) s.(name), input_vec, 'UniformOutput',false));
my_timeseries.input.(name) = timeseries(data, time_vec);
names = fieldnames(internal_vec{1,1});
for i = 1:numel(names)
name = char(names(i));
data = cell2mat(cellfun(@(s) s.(name), internal_vec, 'UniformOutput',false));
my_timeseries.internal.(name) = timeseries(data, time_vec);
names = fieldnames(output_vec{1,1});
for i = 1:numel(names)
name = char(names(i));
data = cell2mat(cellfun(@(s) s.(name), output_vec, 'UniformOutput',false));
my_timeseries.output.(name) = timeseries(data, time_vec);
As you can see I saved multiple structs in the same line - I just made an easier example in the question above. Also, I modified the saving a little bit. I already have a counter in the simulation call - so I don't need the own counter. But this counter isn't available in the post processing anymore - that's why I'm filtering all data with the time information.
The output is exactly the same like in my previous attemp, but much faster.
During simulation the saving is not noticable and the post processing is - like you said - no problem because it does not matter. I havent measured, but I guess it is the range of only few seconds.
Thanks a again and have a merry christmas!

Sign in to comment.

More Answers (1)

Nitin Kapgate
Nitin Kapgate on 16 Dec 2020
As the data is numeric/boolean, I would suggest that you to write the data at every step or periodically to a file (say a CSV file with columns for different signals and rows for every step). That would help in increasing the RAM available for simulation and improve the simulation speed. At the simulation progresses, the growing size of logged data can consume huge amount of RAM and slow down simulations.
Nitin Kapgate
Nitin Kapgate on 18 Dec 2020
I am glad that the issue is resolved.

Sign in to comment.




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by