How to load multiple .dat files into MATLAB?

Hey,

I have 53 .dat (000n_3.dat, 000n_50.dat; 000n+1_3.dat, 000n+1_50.dat; etc. - as you can see, their names are similar but not purely consecutive) files all stored in the same directory, which I want to import into MATLAB all at the same time. Loading each of them works (e.g. load(0002_03.dat)) but what I want is to have all 53 of them loaded into a giant .mat spreadsheet from where I can neatly copy and paste all the rows and columns into an Excel spreadsheet.

How can I do this?

Thanks in advance.

[Jan]

The question is not clear. How do you copy&paste values from a MAT file to an Excel sheet?

[Bianca Elena Ivanof]

When you double click on it in workspace --> a spreadsheet, where you can edit datapoints and copy and paste all rows and columns at your convenience.

[Image Analyst]

I think the script I gave in my answer below would be less tedious for the user.

Answer 1

[Image Analyst]

You can use the FAQ: http://matlab.wikia.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F Use the second example. Then, inside the loop, read the data file with csvread(), readtable(), importdata(), or any of several other functions and append the data to your growing array. After the loop, create a filename and call xlswrite()

% Specify the folder where the files live.
myFolder = 'C:\Users\yourUserName\Documents\My Data'; % Wherever...
% Check to make sure that folder actually exists.  Warn user if it doesn't.
if ~isdir(myFolder)
  errorMessage = sprintf('Error: The following folder does not exist:\n%s', myFolder);
  uiwait(warndlg(errorMessage));
  return;
end
% Get a list of all files in the folder with the desired file name pattern.
filePattern = fullfile(myFolder, '*.dat'); % Change to whatever pattern you need.
theFiles = dir(filePattern);
for k = 1 : length(theFiles)
  baseFileName = theFiles(k).name;
  fullFileName = fullfile(myFolder, baseFileName);
  fprintf(1, 'Now reading %s\n', fullFileName);
  % Now do whatever you want with this file name,
  % such as reading it in, plotting it, using it, or whatever.
  dataArray = csvread(fullFileName); % Or whatever function you want.
  % Append data
  if k == 1
     allDataArray = dataArray;
  else
     allDataArray = [allDataArray; dataArray]; % Must have same number of columns
  end
end
xlswrite(outputFilename, allDataArray, 'All Data', 'A1');

[Jeffrey Maloles]

Hi! This code work perfectly. Thank you.

However, I was going to ask how I can edit this code if

(1) Some of the .dat files in the loop are empty

(2) Some .dat files have missing columns.

Thank you again.

EDIT: I have tried exploring isempty and masking with NaN but I can't seem to make it work.

[Image Analyst]

@Jeffrey Maloles just check for invalid conditions and continue if any are there, like

if isempty(dataArray) || any(isnan(dataArray(:))) 
    continue; % Skip appending if this file is empty or has nans in it.
end
if size(data, 2) ~= 23
    continue; % Skip this file because it does not have the required 23 columns in it.
end

[Jeffrey Maloles]

Thank you for replying.

Error using dlmread (line 147)

Empty format character vector is not supported at the end of a file.

Error in csvread (line 48)

m=dlmread(filename, ',', r, c);

When I open the said .dat file that gives the error, it says "The file is invalid or empty" (see attached screenshot). And stops the loop.

I am actually wondering if isempty is the correct logical operator / condition or something else I haven't encountered yet. I appreciate your time. Thank you.

[Image Analyst]

@Jeffrey Maloles The problem is you're getting a read error before it has a chance to return anything. So to just skip those files, put them in a try catch

try
    m=dlmread(filename, ',', r, c);
catch ME
    continue; % Skip to bottom of the loop if there is an error.
end

If you don't think you should get such an error, attach the file with the paperclip icon in a new question.

[Jeffrey Maloles]

Thank you! Here is the full version of my question.

[asem]

hello @Image Analyst, thank you for your answer above, however, I have similar question but I want to have all the .dat file readout as table or array, when I used the code above, its only read the first file, and compined all the rest of the files in one arrays, any possiable way to do so? I hope its clear as I am new to matlab. thanks in advance.

[Image Analyst]

@asem csvread does give you the data as an array, so not sure why you think it doesn't. If you want the recommended function, use readmatrix. If you want a table, use readtable.

Answer 2

[SnukeN3]

I adapted this script to what I needed. It seems to be adding each file onto the bottom of the last. How do I get it to put new files into new columns?

[Image Analyst]

Replace

allDataArray = [allDataArray; dataArray]; % Must have same number of columns

with this:

allDataArray = [allDataArray, dataArray]; % Must have same number of rows

Answer 3

[SnukeN3]

Yep, that was it. Funny story, I tried the comma originally, but had mis-matched file lengths on the sample files I grabbed. Thanks for the help!