Your system is not going to know where your test data file lives, so you need to provide it more than just the name and extension to it. If it’s being created in the current MATLAB directory, then you can try something like this:
fTest = fullfile(pwd, ‘TEST.xyz’);
cmd = [VMD_path ‘ ‘ fTest];
Note, your VMD path is to the lnk file, which is an OS shortcut to the actual executable. You may need to use the actual location of the executable file rather than its shortcut.