remove duplicate rows from text file

I'd be glad to have some advise on the fastest way to read a text file which contains duplicate rows, remove all the duplicates and write it back.

Matlab version R14 SP3

[Paolo]

What does the text file look like? Could you attach it to your question?

[michael]

couple of hundreds of following rows:

F.field1 = ProtoField.uint16("abc", "ABC", base.DEC)

Answer 1

[Guillaume]

lines = strsplit(fileread(yourtextfile), '\n'));  %read file and split into lines
lines = unique(lines, 'stable');  %remove duplicate lines
fid = fopen(newfilename, 'w');    %open file for writing
fwrite(fid, strjoin(lines, '\n'), 'char'); %merge lines and write
fclose(fid);

[michael]

this one doesn't contains functions to my release

[Paolo]

@Michael, if its not possible for you to share the file, could you show a sample of what the text file looks like?

[Guillaume]

Matlab version R14 SP3

Wow! You're only 25 versions behind the current version, soon to be 26.

While it's probably fairly easy to write a version of strsplit using regexp that would work with your version and strjoin is probably not too complicated either, you'd also have to write your own unique function since the 'stable' option didn't even exist back then. That's a more complicated task particularly as I'd have to pore over the online documentation to find out what was possible or not back then.

I'm sorry you're on your own there. At some point you've got to let go of legacy code.

[Walter Roberson]

lines = strsplit(fileread(yourtextfile), '\n'));  %read file and split into lines

[ulines, ia] = unique(lines);
lines = ulines(sort(ia));

fid = fopen(newfilename, 'w');    %open file for writing
fprintf(fid, '%s\n', lines{:});
fclose(fid)

Together with

function S = fileread(filename)
  fid = fopen(filename, 'rt');
  if fid < 0
    error('file does not exist');
  end
  S = fread(fid, [1 inf], '*char');
  fclose(fid);

function C = strsplit(S, delim)
   C = regexp(S, delim, 'split');