Reading and analysing files in subfolders - indexing

I'm trying to find and open *.tab files containing a string in their filenames, in several sub-folders. I have tried various options online, but always get errors. I think my issues is that I don't understand indexing in for loops. The code below gives the error "Error using fopen First input must be a file name or a file identifier." on line 6 (fname).

I've already tried:

How do I use MatLab to loop through many folders and extract columns of data from .txt files in those folders? - (mathworks.com)

FAQ: How can I process a sequence of files? - (mathworks.com) and similar.

Any help is greatly appreciated!

Closest code:

dt = 0.005;
tt = 0:dt:60;
files = dir(fullfile('...\**\*xacc.tab')); %finds 86 files with xacc string;
FileName = fullfile({files.folder},{files.name});%count = 0;
for k = 1:numel(files)
    fname = FileName(k);
    fid(k) = fopen(fname, 'r'); %I don't know how to open files based on name in files variable
    data_xacc = textscan(fid(k),'%*f%f%f','delimiter',' ','headerlines',2); %I have tried other options such as dlmread, etc. but don't work given .tab extension.
    fclose(fid(k));
    time{k} = data_xacc{1,k}{1,1}; %I want to be able to extract column vectors from each tab file to analyse/interpolate.
    xacc{k} = data_xacc{1,k}{1,2};
    xacc{k} = interp1(time{k},xacc{k},tt,'linear','extrap');
end

Answer 1

[Bjorn Gustavsson]

Why not skip the generation of FileName and index into files directly inside the loop:

dt = 0.005;
tt = 0:dt:60;
files = dir(fullfile('...\**\*xacc.tab')); %finds 86 files with xacc string;
FileName = fullfile({files.folder},{files.name});%count = 0;
for k = 1:numel(files)
  fname = fullfile(files(k).folder,files(k).name); % generate filename for the k-th file
  %fname = FileName(k);
  fid = fopen(fname, 'r'); % Seems unnecessary to keep all fid-s
  data_xacc = textscan(fid,'%*f%f%f','delimiter',' ','headerlines',2); %I have tried other options such as dlmread, etc. but don't work given .tab extension.
  fclose(fid); % since you instantly close them after reading, they will not be useful later afaik
  time{k} = data_xacc{1,k}{1,1}; %I want to be able to extract column vectors from each tab file to analyse/interpolate.
  xacc{k} = data_xacc{1,k}{1,2};
  xacc{k} = interp1(time{k},xacc{k},tt,'linear','extrap');
end

HTH

[A Wol]

Thanks, Bjorn. That works to open the files, but how do I then create the time, xacc variables related to each of the k files? I've tried variations on the below, but get the error: "Unable to perform assignment because the indices on the left side are not compatible with the size of the right side. Error in HistoryAnalysis_disp data_xacc(k) = textscan(fid,'%*f%f%f','delimiter',' ','headerlines',2);"

If I run fopen and textscan without indices, I get the following error for time: "Brace indexing is not supported for variables of this type. Error in HistoryAnalysis_disp time(k) = data_xacc{k}{1,1};".

Thanks again!

for k = 1:numel(files)
    fname(k) = fullfile(files(k).folder,files(k).name);
    fid(k) = fopen(fname(k), 'r');
    data_xacc(k) = textscan(fid(k),'%*f%f%f','delimiter',' ','headerlines',2);
    fclose(fid(k));
    time(k) = data_xacc{k}{1,1};
    xacc(k) = data_xacc{k}{1,2};
    xacc(k) = interp1(time(k),xacc(k),tt,'linear','extrap');
end

[Bjorn Gustavsson]

Since I don't know the contents of your file and how that will be returned in data_xacc I cannot give you specific advice. But a general recommendation is for you to simply set k to an arbitrary number between 1 and numel(files) and then run the first three line in the loop on the command-line. Then you'll have data_xacc as returned by textscan. From there you can doodle around and check what works. Worst come to worst you might have to extract the time and xacc from data_xacc in another inner loop. In that case check and really check the other data-reading functions: importdata, readvars, dlmread, cvsread. And whatnot. In this situation you might even be better off writing your own reader-function that handles this filetype reading line-by-line using fscanf - the reason for doing this is that with such a function returning file-format in nice orderly arrays separates the tasks/steps of this processing into several neatly contained "clean" functions - this might seem to be an extra layer of complexity, but it really is the other way around.

HTH

[A Wol]

Hi, thanks for your advice. I'll give it a go. For reference, an example file is attached. Any further advice would be greatly appreciated.

Thanks!

[Bjorn Gustavsson]

Yeah, once you start extracting time and xacc from the data_xacc variable this shouldn't be done with the k-index, since that would index into ever larger columns data_xacc in sucessive files. This seems peculiar to me. This is what works for the file you sent:

t_time{k} = data_xacc{1}; % avoid time as a variable, it is built-in function
xacc{k}   = data_xacc{2};                                                   
% Find elements that are finite in both variables
idx_ok    = isfinite(t_time{k})&isfinite(xacc{k});
% use only those for interpolation, also not overwriting the xacc-variable
xacc_i{k} = interp1(t_time{k}(idx_ok),xacc{k}(idx_ok),tt,'linear','extrap');

[A Wol]

Thanks so much, Bjorn. Works now!

Cheers

[Bjorn Gustavsson]

Good, happy that it helped.