Transferring all columns in all text files in a folder and graphing data

Hello all! I am trying to develop a matlab code for my lab that will take all the .txt files in a folder with 5 columns of data. I then need to be able to separate the data by the columns and be able to graph certain ones against each other. I have tried so many things at this point. I am currently using readtable for impDAT(ii), I have also tried dlmread, importdata and textscan to no avail. I have also tried many different ways of formatting, I am getting an error with them all that it cannot open file. I will attach my code and if anyone has any advice would mean a lot. Also once I do this I need to figure out a way that every time it does a new text file it is a different color with a legend based on text filename. Thanks so much, I could graph manually but with all medias and trials I would spend days. :)

delimiterIn = ' ';
headlinersIn=1;
D = 'C:\Users\Sarah\Desktop\Sarah_2018\di no tape trial 1';
%S = dir(fullfile(D,'*.txt'));
F = dir(fullfile(D,'*.txt'));
for ii = 1:length(F)
   %fid = fopen(F(ii).name);
   impDAT(ii) = readtable('F(ii)');
    %c(ii)= struct2cell(impDAT(ii))
    %float(impDAT(ii).data)
%cell2mat(struct2cell(impDAT(ii).data))
DAT1(ii)= abs(impDAT(ii).data(:,3));
DAT2(ii)= (impDAT.data(:,2));
DAT3(ii)= (impDAT.data(:,1));
DAT4(ii)= (impDAT.data(:,4));

figure(1);
subplot(2, 2, 1);
plot(DAT2(ii), DAT1(ii), '.', 'Color', 'red');
titstr=sprintf('Nyquist Plot');
title(titstr,'FontSize',13);
xlabel("Z'/ohm");
ylabel('-Z"/ohm');
grid on;
hold on

subplot(2, 2, 2);
loglog(DAT3(ii), DAT4(ii), '.', 'Color', 'blue');
titstr=sprintf('Frequency Vs. Impedance Plot');
title(titstr,'FontSize',13);
xlabel('Freq/Hz');
ylabel('Z/ohm');
grid on;
hold off
end
% N = length(F);
% % loop for each file 
% for i = 1:N
%     %thisfile = files(i).name ;
%    impDAT(i) = importdata(fid, delimiterIn, headerlinesIn);
% DAT1(i)= abs(impDAT.data(:,3));
% DAT2(i)= (impDAT.data(:,2));
% DAT3(i)= (impDAT.data(:,1));
% DAT4(i)= (impDAT.data(:,4));
% 
% figure(1);
% subplot(2, 2, 1);
% plot(DAT2(i), DAT1(i), '.', 'Color', 'red');
% titstr=sprintf('Nyquist Plot');
% title(titstr,'FontSize',13);
% xlabel("Z'/ohm");
% ylabel('-Z"/ohm');
% grid on;
% hold on
% 
% 
% subplot(2, 2, 2);
% loglog(DAT3(i), DAT4(i), '.', 'Color', 'blue');
% titstr=sprintf('Frequency Vs. Impedance Plot');
% title(titstr,'FontSize',13);
% xlabel('Freq/Hz');
% ylabel('Z/ohm');
% grid on;
% hold off
% 
% 
% end

[Stephen23]

@Sarah Brianne: today I fixed your code formatting for you. In future you can do it yourself: select the code text, click the {} Code button, ensure that there is an empty trailing line.

I also got rid of the uselesss # symbols from your tags, and added space characters between the words: you just need to use commas to separate the tags.

Please upload a sample file by clicking the paperclip button. Without a sample file it is very difficult for us to help you.

[Sarah Brianne]

Thank you so much for your time. I am very apologetic and do better in the future. I just started doing this for school and now my lab. I have a lot to learn. Thank you so much for teaching me! I have uploaded a sample file for you if you still need it.

Answer 1

[Stephen23]

The basic problem is that you are not providing the directory when you read the file data, so whatever function you use will only look in the current directory. When you call dir you correctly added the directory path D using fullfile, you just need to do the same for when you read the file data:

D = 'C:\Users\Sarah\Desktop\Sarah_2018\di no tape trial 1';
S = dir(fullfile(D,'*.txt'));
C = cell(1,numel(S));
for ii = 1:numel(S)
    F = fullfile(D,S(ii).name); % <- you forgot to include the directory here!
    C{ii} = whateverFunctionReadsYourFile(F);
end

As you did not provide any sample file I have no idea what function you need to use to read it.

[Sarah Brianne]

I will also upload the type of files I am trying to use.

[Sarah Brianne]

C{k}

Where does this k come from?

[Stephen23]

@Sarah Brianne: I normally use k for the loop index. I changed my answer to use ii, just like your code.

I tried dlmread on your sample file and it worked perfectly. I did not try them, but as your file is a very simple whitespace-delimited with only numeric values I am sure that readtable and importdata won't have any problems either.

Once you have all of your data in cell array C you could loop over the cells and plot the data, or concatenate the data into one numeric array, or whatever you require.

[Sarah Brianne]

I really appreciate your help! I need to improve bad but the people in my lab were starting to look at me. If you would ever like to reach out you can add me on facebook @ Sarah Brianne

[Sarah Brianne]

Error: File: Nyquistandfreqvsimpplots.m Line: 16 Column: 15 ()-indexing must appear last in an index expression.

I am getting this error. It is also telling me that I cannot index a temporary array. I looked at other answers tried some things and they did not work. I have class now but will look more later. Figured I would ask if you had ever experienced this error before.

[Stephen23]

@Sarah Brianne: you need to write tidier code: why do you define D twice, and call dir twice? This just confuses. You should align your code using the default MATLAB editor settings: select the code, press ctrl+i. Badly aligned code is one way the beginners hide bugs in their code.

That error occurs because in MATLAB it is not possible to index into already indexed expressions. So this is not allowed:

abs(temp(ii)(:, 3));
    temp(ii)          % first indexed expression
            (:, 3)    % not allowed immediately after!

It is not clear what you are trying to achieve with that anyway, as you used parentheses to obtain temp from cell array C, which means temp seems to be a scalar cell array. Probably you meant to use curly braces to obtain the contents of the cell array:

• {} curly braces access the cell contents.
• () parenthese access the cells themselves.

https://www.mathworks.com/help/matlab/matlab_prog/access-data-in-a-cell-array.html

Even if temp was a non-scalar (e.g. a matrix), then it is not possible to allocate a non-scalar array to one element of an array (an element of a numeric/char/logical array can only hold one value) like you are trying here:

DAT1(ii) = some non-scalar numeric array

Numbered variables are a sign that you are doing something wrong: instead of numbered variables, you should be using indexing.

You probably want to start with something like this:

D = 'C:\Users\Sarah\Desktop\Sarah_2018\di no tape trial 1';
S = dir(fullfile(D, '*.txt'));
figure();
ax1 = subplot(2,2,1, 'NextPlot','add');
ax2 = subplot(2,2,2, 'NextPlot','add');
for ii = 1:numel(S)
    F = fullfile(D, S(ii).name);
    mat = dlmread(F);
    plot(  ax1, mat(:,2), mat(:,3), '.', 'Color', 'red');
    loglog(ax2, mat(:,1), mat(:,4), '.', 'Color', 'blue');
end

which creates these plots (the test files are attached):

Oh, that looks like the silhouette of a horse!

[Sarah Brianne]

It does look like a horse, in my opinion as a coder you are already an artist but now it is a double whammy. Thanks so much for your help. Could the issue be that I have the 2017 version of matlab?

[Sarah Brianne]

I have fixed it. Loglog is not working on mine though tackling that now if you have any tips.

[Sarah Brianne]

Also do you know a way to make it change colors every iteration and perhaps make a legend with filenames,

[Sarah Brianne]

Fixed it all, now just working on legend and color change

[Stephen23]

"Also do you know a way to make it change colors every iteration..."

For a start remove the 'Color' option when you call any plotting functions. Because of the axes 'add' option, MATLAB should then make each plot its own color. You can change the color palette by setting the axes ColorOrder property.

For the legend try something like this:

legend({S.name})

https://www.mathworks.com/help/matlab/matlab_prog/comma-separated-lists.html

https://www.mathworks.com/matlabcentral/answers/320713-how-to-operate-on-comma-separated-lists

Please accept my answer if it helped you: this is the easiest way to show your appreciation to the volunteers on this forum.

[Sarah Brianne]

Hey Stephen, long time no talk. I missed you :) Okay so here is what is happening. Code works wonderful most of the time, but not always. I cannot figure out why. I attached an image of my issues and files, if you could please take a look and help me make the code more robust and not apt to failure. Also, I am currently simply deleting text headers to allow code to work on files if there is a way to make this work with type of file in last picture without modification please let me know.

[Stephen23]

@Sarah Brianne: so, two new problems... okay, lets have a look.

1. Error: It depends on what causes the error. Please post a comment with the complete error message posted as text. This means all of the red text.
2. Header: it depends on what function you are using to read the data with. Please upload your code by clicking the paperclip button.

[Sarah Brianne]

Those two lines are the only red error lines provided by Matlab. It is I feel like almost the code earlier but I will upload. Thank you so much!

[Stephen23]

@Sarah Brianne: this dlmread usage looks a bit suspicious:

mat = dlmread(F,' ',ii);

It is not clear what purpose the loop iterator is supposed to achieve as the third argument, especially as the dlmread help does not list any syntax like that.

I suspect that you want something like this:

mat = dlmread(F,' ',1,0);

Where the third and fourth arguments specify the row and column to start reading the data from. Adjust them to suit your file, following the information in the help. Depending on your MATLAB version it might not be able to read the file if it contains non-numeric data, which I guess your header contains.

If you upload a sample data file I will take a look at it.

[Sarah Brianne]

Here is file with header I have no use for.

[Sarah Brianne]

I tried doing dlmread in way you spoke of. Still get the following error.

Index exceeds matrix dimensions.

Error in labgraphsimpedancespectrum (line 14) matpos = (abs(mat(:,3)));

[Sarah Brianne]

Any luck?

[Stephen23]

@Sarah Brianne: dlmread does not work on files which contain non-numeric data (like that header section): the entire file should be a matrix of delimited numeric data for it to work properly.

I recommend that you use readtable or textscan, something like this:

fmt = repmat('%f',1,5);
opt = {'HeaderLines',8, 'CollectOutput',true};
... inside the loop:
[fid,msg] = fopen(F,'rt');
assert(fid>=3,msg)
C = textscan(fid,fmt,opt{:});
mat = C{1};
...

Adjust the textscan options to get it to work with your data file.

[Sarah Brianne]

Thank you so much. I messaged you while I was at lab and found a way to just redo all my text files and do them in the future so they are linear. Code works fine ? It was just the damn text files and the conversion process the programming was doing