Getting MD5 hash of files with "bad" names (Windows, NTFS)

I tryed several variants of getting MD5 hash of files with "bad" names (e.g. 'C:\wrongname.' (with trailing dots); 'C:\wrongname ' (with trailing spaces); (path)names with accents like 'é', 'é' which occur in French, German, Hungarian; containing various forms of dashes (–) etc.).

None is working, except Var5, but is too slow (calculates a 10Mb file within 10 min).

Can you suggest any working variant, or how to speed up Var 5?

OS: Windows 10, File system: NTFS, Matlab 2015a

% Var 1 ---------

% dirinfo(i).name - is string containing full pathname (e.g. 'C:\myfolder\myfile.ext')

Opt.Format = 'HEX'; Opt.Method = 'MD5'; Opt.Input='file';
hash(i) = DataHash(['\\?\' dirinfo(i).name], Opt); % ERROR - not working with  "bad" names
% DataHash.m - https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash

%Var 2 ----------

hash(i)=mMD5(['\\?\' dirinfo(i).name]); % fast, works on 'C:\wrongname.' (with ending dots), 'C:\wrongname ' (with ending spaces), but do NOT works with file names (or pathes) with accents like 'é', é'
% (mMD5.c,  see https://www.mathworks.com/matlabcentral/fileexchange/7919-md5-in-matlab)

% Var 3 ---------

mddigest  = java.security.MessageDigest.getInstance('MD5');
bufsize = 8192;
[fid,errmsg] = fopen(['\\?\' dirinfo(i).name]); % ERROR here - matlab fopen don't understand "bad" names
if fid>=3 % if success
while ~feof(fid)
  [currData,len] = fread(fid, bufsize, '*uint8');     
  if ~isempty(currData)
      mddigest.update(currData, 0, len);
  end
end
fclose(fid);
hash(i) = reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);
else 
disp('can't open file');
end

% Var 4 ---------

file = java.io.File(['\\?\' dirinfo(i).name]);
digestream = java.security.DigestInputStream(file,mddigest);
file_bytes = typecast(org.apache.commons.io.FileUtils.readFileToByteArray(file),'uint8'); % ERROR: out of memory if BIG file
if ~isempty(file_bytes)
mddigest.update(file_bytes, 0, numel(file_bytes));
end
hash(i) = reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);

% Var 5 --------

    mddigest = java.security.MessageDigest.getInstance('MD5');
    filestream = java.io.FileInputStream(java.io.File(['\\?\' dirinfo(i).name]));
    digestream = java.security.DigestInputStream(filestream,mddigest);
    while(digestream.read() ~= -1), end % TOO LONG - never goes out this cycle
    hash(i)=reshape(dec2hex(typecast(mddigest.digest(),'uint8'))',1,[]);

Answer 1

[Guillaume]

As others have pointed and as Microsoft clearly says:

Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not.

As you've found out, matlab fopen does not support it. .Net (which you can directly from matlab) also does not. From your testing, it looks like Java does not handle it properly either.

The only way you can access such file is directly through the win32 api, e.g. with CreateFile. So if you really need to handle such paths you'll have to resort to mex.

But really, the best fix would be to fix the tool that creates these files in the first place so that it doesn't use bad filenames. It's not just matlab that can't handle them, it's also most backup tools, file transfer tools, etc.

As for path names with accents, there does not appear to be any problem there. Matlab (R2017a tested) handles them fine.

[bbb_bbb]

From your testing, it looks like Java does not handle it properly either.

Java works good. The code in example "Var 5" works with ANY file name, but is VERY slow. I asked if one can change this code for faster processing.

The loop

while(digestream.read() ~= -1), end

reads file byte by byte, I think therefore it is so slow.

But really, the best fix would be to fix the tool that creates these files

I agree, the best fix is to fix win32 api, e.g. CreateFile. But for some reason Microsoft don't do this... Now I just want to create tool for handling existing files.

[Guillaume]

No the tool to fix is the one that creates files ending with dots and spaces. Microsoft are not going to change the win32 api and you'll find plenty of other programs that cannot cope with these files.

Yes, you can work around the problem by using the '\\?\' syntax with some api (java it seems, and parts of win32), but you're still papering over the problem that these files shouldn't exist.

[Stephen23]

+1 Fixing the file names is the correct solution. Anything else will just be a total waste of time (as the duration of the current discussion already proves).

Sometimes creating hacks can be useful, when it creates some useful functionality that cannot be achieved efficiently using other methods. However in this case there is a much more efficient solution: naming the files by following Microsoft's advice. All of this time wasted on badly named files just takes you away from doing whatever data analysis that you are probably trying to do.

As Guillaume pointed out, using these badly named files will be unpredictable with any tool that you will try to use, so this is a pointless fight with no end.

[Jan]

+1 It is the best way to provide properly formatted input files. Insisting on using R2015a and Windows 10 and trailing dots and spaces and dates in the far future and unicode file names is simply messed up. There are too many obstacles before the actual problem can be solved.

Fixing the Windows API is not an option, because it is not broken. The behavior is well defined and documented. The method which creates your data set should be fixed to consider the well known limitations.

[bbb_bbb]

Good programs are reliable programs. Do not you want to create reliable programs?

I understood where the buggy programs come from! Their authors try to change user data for programs, and not vice versa. To make it look more convincing, they refer to supposedly authoritative sources (such as Microsoft), what users should do and what not. Sorry, but it's just ridiculous!

I would not want, for instance, to be a victim of a backup program that backups files through the times, because they have "strange" names (or dates).

[Guillaume]

Files ending with dots or spaces are in the realm of undefined behaviour. The underlying file system allows them and parts of the Win32 API (and I repeat: only parts of it) but a great number of other APIs don't. As I said, the whole .Net API (at one point supposedly the future of Windows) does not.

So I'm not sure what's reliable about a program that creates files that are guaranteed to cause problems for the majority of other programs.

In any case, this is not a matlab issue. This is a Microsoft issue. And I don't think microsoft documents which of their functions work or not with files ending with dots and spaces. So all that can be done is use the syntax \\?\ to bypass the checks in the current API and pass the path unprocessed to the underlying API and hope for the best. In my view, a good way to introduce less reliability.

By the way I'm not sure how you can say that Microsoft is a supposedly authoritative source. They are authoritative. The page I linked (link fixed, an extra : got in there) is not mere advice. It is the rules you have to follow for a program to work properly on Windows.

[bbb_bbb]

So I'm not sure what's reliable about a program that creates files that are guaranteed to cause problems for the majority of other programs.

This statement shows that you still did not understand what I want to do. The program code must calculate md5 sum of existing file sets, and must be smart enough to process all files, not only 99% of them. As well as a backup program don't creates files, it just makes the copies of existing file sets. A good backup program must do its job for 100%, not 99%.

What have Microsoft recommendations to do with it?

[Jan]

@bbb_bbb: Reliable programs work on well formatted input. There is no coding style which can change the old rule:

Garbage in, garbage out.

There are many limitation in file names, such that e.g. backup systems will fail: At first the old limitation to 260 characters for the path and for the file name also. Using \\?\ you can expand the limit to 32,767 characters - many, but limited. Then special characters > < ? | : " \ / * and forbidden file names: CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9. Such names can come e.g. from extracting folders, which have been created under linux, which has less limitations. Backup systems are influenced by the underlying file systems, and I have seen bugs concerning the limits of fat16, fat32, NTFS, AFS and BTRFS. Sometimes hard or soft links let the system crash, sometimes the Alternate Data Streams of NTFS. Some programs crashed during the change of the day-light-saving time, or during the extra second.

Any software has a limited reliability only, as any mechanical tool has a limited applicability also. "Reliable" does not mean that you can process everything you want and always get what you expect, but it means, that the software works correctly inside the specified limits and with inputs inside the specifications.

Therefore the Windows API does work reliably, because it is well documented, that trailing dots are not handled. Your input files violate the specifications. Then asking for modifying the Windows API and the tools I offer for free is the wrong approach. Even if you would get the MD5 hashes, you cannot copy, move or process these files with standard software, e.g. the Windows Explorer or any backup tool.

If you now insist on using such file names, you act like using a drilling machine to beat a nail into the wall.

We have suggested several work-arounds and explained clearly, that the reliable way to solve the problem is not to process weird inputs by even more weird code, but to remove the source of the problem by fixing the program, which creates the input data. Your ironic question "do not you want to create reliable programs?" is ignorant. I think it is you, who does "not understand the true deepness of the problem".

Answer 2

[Jan]

I think, that the problem occurs in

% dirinfo(i).name - is string containing full pathname (e.g. 'C:\myfile.ext')

and

['\\?\' dirinfo(i).name]

already. I see no reason to assume, that Matlab or any of the other functions "does not understand bad names". Please post the complete error messages instead of the rough description 'ERROR - not working with "bad" names', 'ERROR here - matlab fopen don't understand "bad" names'.

Do you obtain dirinfo by the dir command? Then try:

File = ['\\?\', fullfile(dirinfo(i).folder, dirinfo(i).name)];
exist(File, 'file')
disp(['\\?\' dirinfo(i).name])
exist(['\\?\' dirinfo(i).name], 'file')

What do you get as output?

[EDITED, Walter is right: The Windows Command shell suffers from trailing dots and spaces.]

[Walter Roberson]

"Names with trailing dots or spaces are not "bad" in any way at all."

Microsoft says not to use files whose name ends with dots, that individual file systems might support them but that the command shells do not.

[bbb_bbb]

I don't use Matlab dir command due "std::exception" error on files with future date (e.g. 27 July 2079). (see https://www.mathworks.com/matlabcentral/answers/165064-problem-with-dir-function-in-r2014a). Btw, there is no field folder in the output struct of this command. Let's suppose that dirinfo(i).name already contains a full path-name string (without prefix '\\?\'). So the code will be:

File = ['\\?\', dirinfo(i).name];
 exist(File, 'file')
 disp(['\\?\' dirinfo(i).name])
 exist(['\\?\' dirinfo(i).name], 'file')

On "bad" files:

ans =

       0

\\?\C:\mypath\wrongname.

ans =

       0

and on "normal" files:

ans =

       2

\\?\C:\mypath\normalname.ext

ans =

       2

[Jan]

Thanks Walter, you are right. The Windows Explorer cannot create such files or folder, but it works in a command shell:

mkdir \\?\C:\Users\Jan\asd.

A folder with the name 'asd.' is created, but you cannot create such a name in the Windows Explorer or enter this folder there.

@bbb_bbb: What about renaming the files?

[Walter Roberson]

Which MATLAB version are you using? The change to include the field named "folder" was about two releases ago.

[bbb_bbb]

What about renaming the files? This don't suit. The program must process 100000 and more files automatically, and among them not less than 100 files with "bad" names always occur. Matlab2015a

[Walter Roberson]

Is there are particular reason the use \\?\ for this?

[Walter Roberson]

However, you could copy them one by one to a different known-good file name, such as in tempdir(), and MD5 that name. Furthermore you appear to have a way to detect that the failure will occur, so you could handle those ones specially.

[Jan]

Sorry bbb_bbb, I'm not asking for fun, but I try to help you. Are you really sure, that the output is "\\?\C:\mypath\wrongname."? Then you are using strange folder and file names beside the trailing dot. I assume, the output is created manually and you use "wrongname." as a place holder. Please don't do this, because this can shadow the information, which actually contains the problem.

I asked for the error messages also. Please answer such questions to make it as easy as possible to help you.

I do not have problems under R2009a and Win7:

file = fullfile('\\?\', tempdir, 'asd.');
fid  = fopen(file, 'w');
if fid == -1, error('Cannot open file: %s', file); end
fwrite(fid, 'hello', 'char');
fclose(fid);

Opt.Input = 'file';
DataHash(file, Opt)

ans =
  5d41402abc4b2a76b9719d911017c592

Does this work with your R2015a/Win10 also?

I do not see the problem with renaming the 100 bad file names. This is done with a few lines of code and solved in a minute. movefile can rename such files:

movefile(file, [file, 'txt']);

You have files with a date of 27 July 2079? This is a really strange data set.

Perhaps the file names do not end with a dot in opposite to your assumptions. How did you get the list of file names? Could they contain non-printable unicode characters? The output is at least suspicious:

File = ['\\?\', dirinfo(i).name]
exist(File, 'file')

\\?\C:\mypath\wrongname.
ans = 0

This means that there is no such file. Check for unicode characters by converting the characters to double:

disp(dirinfo(i).name)
double(dirinfo(i).name)

Please with the real output, not some created pseudo-data.

[bbb_bbb]

Dear Jan, you apparently don't understand the true deepness of the problem!

This code

file = fullfile('\\?\', tempdir, 'asd.');
fid  = fopen(file, 'w');
if fid == -1, error('Cannot open file: %s', file); end
fwrite(fid, 'hello', 'char');
fclose(fid);

actually creates file named 'asd' , not 'asd.' One can see that in Windows Explorer. And this code:

Opt.Input = 'file';
DataHash(file, Opt)

again reads the file 'asd' - not the 'asd.' If I rename 'asd' to 'asd.' mandatory, the output will be:

Error in DataHash (line 226)
         Error_L('FileNotFound', 'File not found: %s.', Data);

Renaming hundreds of files (and also pathes!) automatically may be too complicated, and dangerous. And if you say it is so easy, why should be plain caculating md5 of these files so hard? And what to do with files which have similar names ('asd', 'asd.', 'asd ')? And names with diacritical signs which may come from other languages (e.g. 'Köszönöm')?

You have files with a date of 27 July 2079? This is a really strange data set.

I agree, such files occur not often. But If I have at least one such file, the dir function crashes and therefore will be useless. That bug is a serious problem, too. That's why I use a corresponding .NET function instead, although it is less comfortable.

And once again, your code (with a little modification, for simplicity):

badname='C:\myfolder\wrongname.';
File = ['\\?\', badname];
exist(File, 'file')
disp(['\\?\' badname])
exist(['\\?\' badname], 'file')

gives

ans =

       0

\\?\C:\myfolder\wrongname.

ans =

       0

I asked for the error messages also.

There is no error messages.

All what I did - I created the folder C:\myfolder' and the file 'wrongname.' in it. And run the code. Why do you assume, that the output is created manually?

movefile can rename such files

Unfortunately, movefile can't rename "bad" files:

movefile(badname, [badname, 'txt']);

Error using movefile
No matching files were found.

And even the prefix '\\?\' don't help.

Walter Roberson: However, you could copy them one by one to a different known-good file name

- A good idea, but which copy function will work correctly with these files? Besides that, if files are big, the copy operation may take extra time.

[Jan]

 This code [...] actually creates file named 'asd' , not 'asd.

No, not on my R2009b/Windows 7 computer: Here the file "asd." is created exactly as expected - with the dot. I will try it later on a R2016b system.

As far as I understand, on your R2015a/Win10 system a file called "asd" is created. Did you check this carefully? Do not trust the display of the Windows Explorer, which is obviously flawed in this point.

After the successful creation, fopen() can open the files also, as well as GetMD5.c and DataHash can process them directly.

E.g. L = dir(fullfile(tempdir, 'asd*')) finds the file, replies the correct name 'asd.', but the wrong size 0 Bytes and empty datenum. But

L = dir(fullfile('\\?\', tempdir, 'asd*'))

works correctly.

In the Windows Command Shell this works also:

mkdir \\?\C:\Temp\folder.

I do see the name in the Windows Explorer with a dot also and can access the folder from Matlab.

On my computer there is no "true deepness of the problem", but no problem at all. I still do not see any evidence, that dirinfo(i).name contains the correct names include the path and the \\?\ tag. You do not answer my corresponding question, what this replies:

disp(dirinfo(i).name)
double(dirinfo(i).name)

It would explain all your observations directly, if this is not the correct file name. When you do not post, how you have created it and its contents, the "assumption for simplicity", that might be the actual and only problem. Maybe there is another, but what's wrong with excluding this?! It would help to find out the different between out systems, which let the code fail on your computer, but not on mine.

Why do you assume, that dirinfo(i).name contains the correct file names?

You have 4 options:

1. Either your code contains another bug and you find and fix it
2. Or there is really a problem with processing the files and you can find and fix it
3. Or you rename the files and process them normally
4. or you give up.

Equivalently for the file dates: Either fix the source of the strange dates, or fix the dates by a small tool (FEX->Get/SetFileTime), or use strange .NET tools, which are currently suspected to provide wrong file names.

 And what to do with files which have similar names
 ('asd', 'asd.', 'asd ')?

You did not provide enough information to answer this. So it is your decision.

If you ignore my questions for clarifications, I cannot help you further with narrowing down the source of the problem.

Good luck.

[bbb_bbb]

I not ignore your questions. Please comment this. (The folder 'myfolder' was empty before running this code.)

file = '\\?\C:\myfolder\asd.';
fid  = fopen(file, 'w');
if fid == -1, error('Cannot open file: %s', file); end
fwrite(fid, 'hello', 'char');
fclose(fid);
dirinfo=dir('\\?\C:\myfolder\asd*')
double(dirinfo.name)

dirinfo = 

         name: 'asd'
         date: '17-окт-2017 01:28:11'
        bytes: 5
        isdir: 0
      datenum: 7.3699e+05

ans =

      97   115   100

As you see, file 'asd' is creating, not 'asd.'. That name (without dot) appears also in Windows Explorer.

[Jan]

Matlab 2009a/Win 7 creates "asd.", Matlab 2016b/Win7 creates "asd". So downgrading would be an option, but Win10 is supported since 2015a only, see https://www.mathworks.com/matlabcentral/answers/223444-is-matlab-compatible-with-windows-10.

movefile('asd.', 'asd') fails in R2016b:

 Error using movefile
 The filename, directory name, or volume label syntax is
 incorrect.

But the C-mex FEX: FileRename works.

My conclusion: Rename the files:

Folder   = '\\?\C:\myfolder\'
FileList = dir(fullfile(Folder, '*.*'));
FileName = {FileList.name};
NewName  = regexprep(FileName, '\.$', '_dot_');
NewName  = regexprep(NewName,  '\s$', '_space_');
toRename = find(~strcmp(FileName, NewName));
for k = toRename
  FileRename(fullfile(Folder, FileName{k}), ...
             fullfile(Folder, NewName{k}));
end

Now trailing dots are replaced by '_dot_', and trailing white space characters by '_space_'. Adjust this like you want.

Note: The WindowsAPI function GetFullPath converts "asd." to "asd" (see FEX: GetFullPath ).

[EDITED], GetMD5.c from the FileExchange is updated and you find a pre-compiled MEX file also now. It processes '\\?\C:\myfolder\asd.' directly.

[bbb_bbb]

So downgrading would be an option,

It's no option for me at all, because something else may go wrong...

Thank you for repairing GetMD5. It now processes files with trailing dots well. But some other files (with Unicode symbols in their names) fail:

file='\\?\C:\1\Chuck Berry – Fresh Berry - 1965\A.jpg'
double(file)
GetMD5(file, 'File')

file =

\\?\C:\1\Chuck Berry – Fresh Berry - 1965\A.jpg

ans =

    Columns 1 through 15

            92          92          63          92          67          58          92          49          92          67         104         117          99         107          32

    Columns 16 through 30

            66         101         114         114         121          32        8206        8211          32          70         114         101         115         104          32

    Columns 31 through 45

            66         101         114         114         121          32          45          32          49          57          54          53          92          65          46

    Columns 46 through 48

           106         112         103

Error using GetMD5
*** GetMD5[mex]: Cannot open file: [\\?\C:\1\Chuck Berry – Fresh Berry - 1965\A.jpg]

Can you repair this?

You see that renaming only files with dots and spaces isn't sufficient.

Make notice that two dashes which occur in the file name are not the same.

And when I remove Unicode symbols, it works correct:

file='\\?\C:\1\Chuck Berry - Fresh Berry - 1965\A.jpg'
 double(file)
 GetMD5(file, 'File')

file =

\\?\C:\1\Chuck Berry - Fresh Berry - 1965\A.jpg

ans =

    Columns 1 through 31

      92    92    63    92    67    58    92    49    92    67   104   117    99   107    32    66   101   114   114   121    32    45    32    70   114   101   115   104    32    66   101

    Columns 32 through 47

     114   114   121    32    45    32    49    57    54    53    92    65    46   106   112   103

ans =

dbb290b46c91c246223b6ea79a923fae

[Jan]

27 July 2079, trailing dots and spaces, Unicode characters - this seems to be a never ending story with a deeply disturbed data set. I will look, if I can find a reliable way to use unicode file names in GetMD5.c, but it looks more and more like fiddling. I recommend to follow Guillaume's advice.

 Thank you for repairing GetMD5. It now processes files with trailing
 dots well.

This function did not had any problems with the file names, but the compilation failed, because you used the weak LCC32 compiler.

file = '\\?\C:\1\Chuck Berry - Fresh Berry - 1965\A.jpg';
tmp  = fullfile(tempdir, 'tempfile');
FileRename(file, tmp);  % Handles Unicode and trailing dots
Hash = GetMD5(tmp, 'file');
FileRename(tmp, file);

But then you might come with the next restriction: Renaming of files work on the same disk only. And the disk or file cannot be write protected, such that it does not work on a DVD. Or if the file name has more than 32767 characters. Or contains a \0 character.

You cannot process garbage reliably.

[bbb_bbb]

Renaming is not quite good decision - not only because it does not work on a DVD. It violates the rule of good programming: don't change original data unless necessary. E.g. in case of system failure, the file that was renamed has to be recovered manually.

Or if the file name has more than 32767 characters. Or contains a \0 character.

That is your own sophistry. In my first post I clearly described what types of files should be processed.

[Walter Roberson]

It is not yet clear to me that copyfile() to tempname() is not an option for files detected to be a problem based upon exist() .

[bbb_bbb]

copyfile() to tempname()

is acceptable, but only as a last chance or work-around, because it is very time-consumpting, e.g. if there are many files in a "bad" folder or files themselves are big-sized.

[Stephen23]

'It violates the rule of good programming: "don't change original data unless necessary."'

No it doesn't, because the rule "fix bugs where they occur" exactly makes this change "necessary". Your filenames are outside of those specified to work correctly with Windows. Solution: change them so that they are suitable for Windows. Not only that, but filenames should not contain data at all, or at most only some very high-level meta-data, so changing the name should make no difference to your data.

"Renaming is not quite good decision"

Ensuring that the names are created correctly in the first place would be the best decision. Anything else will ultimately just waste more of your time. Like this discussion already proves.

"because it is very time-consumpting, e.g. if there are many files in a "bad" folder or files themselves are big-sized."

Nope, it would only take a few minutes with the right tool. I do this all the time with quite large files from our test department, to store the files systematically. It doesn't change the data at all.

It is not clear to me what the problem is. Why are you letting something as trivial as filenames get in the way of doing your work?