Help!!! how to search for some xx xx xx xx(hex) in a dat file very fast!!!

Help!!! I have a dat file, about 40MB,

I want to search for xx xx xx xx (hex),

I can do it using for or while loop, but it's too slow because of 40 million Bytes !

how to speed up,thanks!

[dpb]

What format is the file?

[Eric Jiang]

binary

Answer 1

[Guillaume]

Unlike per isakson, I'm assuming that you're looking for a byte pattern (given in hexadecimal format) in a binary file. If you're looking for a pattern of hexadecimal characters in a text file see per's answer.

%input
hexpattern = ['41'; 'AB'; 'FF'; '7E'];  %you haven't specified how this is stored. Taking a guess
filetosearch = 'C:\somewhere\somefolder\somefile.dat';  %doesn't have to have .dat extension

%read file
fid = fopen(filetosearch, 'r');
assert(fid > 0, 'Failed to open file. Most likely the wrong path was specified');
filecontent = fread(fid, [1 Inf], '*uint8');  %read all bytes at once
fclose(fid);
%pattern search
patternvalues = hex2dec(hexpattern);
patternlocation = strfind(filecontent, patternvalues);  %despite its name strfind also works for numbers

sprintf('Hex pattern was found at byte(s) %s', strjoin(compose('%d', patternlocation), ', '));

edited as I got per isakson and dpb mixed up

[Eric Jiang]

thanks，you got what i meant。because the pc I use is a little bit old,when i try to read all bytes at once, it came out a out of memory error, how to solve this problem? i dont understand, the dat file is 40MB, howcome it would be out of memory?

[Guillaume]

Did you use the code in my answer? You shouldn't get an out of memory error with it for a 40 MB file.

What does

memory

say?

[Eric Jiang]

thank you!!!, you code is very usefull, but another problem came out, when i use strfind,the out of memory error happened, i guess the algrithm of strfind needs lots of memory to do the job, how to solve this problem?

Answer 2

[per isakson]

Your question is very vaque and leaves room for interpretation.

I assume that dat-file is an ordinary text file. I cannot guess in what form you want the hex-strings, which are found.

However, I made a little test

• created a 10MB text file, cssm.txt

• created a script, cssm.m

    %%
    tic
    txt = fileread( 'cssm.txt' );
    toc
    %%
    tic
    cac = regexp( txt, '([0-9A-F]{2} ){3}[0-9A-F]{2}', 'match' );
    toc

• ran cssm

    Elapsed time is 0.133106 seconds.
    Elapsed time is 0.357219 seconds.

• and peeked at the result

    >> cac{[1,2,3601]}
    ans =
        '01 23 45 67'
    ans =
        '89 AB CD EF'
    ans =
        '01 23 45 67'
    >> 

I doubt that you can do it significantly faster with plain Matlab on a standard desktop PC

Triggered by Guillaume's answer: To get the locations of the hex-strings replace

cac = regexp( txt, '([0-9A-F]{2} ){3}[0-9A-F]{2}', 'match' );

by

[cac,loc] = regexp( txt, '([0-9A-F]{2} ){3}[0-9A-F]{2}', 'match', 'start' );

and peek

>> loc([1,2,3601])
ans =
          33        2793     9936083

Answer 3

[dpb]

If it's performance you're looking for, pass the job off to a grep utility...there are any number of freeware versions available for Windows if not one already installed on your system...

ADDENDUM

Altho seem to now recall there may be a FEX submission in mex form...I didn't search to see if really is, but suggest probably worth doing so...