Performance of system/dos function

[status, cmdout] = system(cmd)

has been reported to be slow for a while (see this and this), but there seems no satisfying solution yet.

Here is what I tested for Windows command reg.exe, trying to figure out a workaround.

 cmd = 'reg.exe query HKEY_LOCAL_MACHINE\HARDWARE /s'; % as an example

 tic; [status, cmdout] = system(cmd); toc
  Elapsed time is 3.517736 seconds.

 whos cmdout
  Name        Size            Bytes  Class    Attributes
  cmdout      1x95249        190498  char 

 tic; [status, cmdout] = dos(cmd); toc
  Elapsed time is 3.512847 seconds.

 tic; cmdout = evalc(['! ' cmd]); toc
  Elapsed time is 3.509961 seconds.

As we can see, system(), dos() and evalc('!') have similar performance. When I test the same cmd in other programs (Octave and python), it takes about 0.6 to 0.7 seconds. Running it at Windows command prompt takes less than a second based on my eye test. Then I tried to redirect output to a file:

 tic; [status, cmdout] = system([cmd ' >myFile']); toc
  Elapsed time is 0.886915 seconds.

This improves the performance significantly. The cmdout is empty now as expected.

Does this suggest there should be a way to improve system() for dealing with cmdout?

Is there a way to redirect cmdout to a Matlab string, so we can avoid to read and delete the temp file?

[Yair Altman]

Without directly answering your important note that redirecting the output seems to improve performance, note that Matlab's builtin winqueryreg() function can be used in this specific case instead of using the clunky system() call, probably improving performance.

[Xiangrui Li]

Yes, winqueryreg() is super fast, and should be used whenever possible. The example here just shows the performance issue. One of my real use of reg.exe is to find FTDI VCP:

[err, out] = dos('reg.exe query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\FTDIBUS /s');

winqueryreg likely won't help here, since the subkeys are unknown. On my computer, it takes 1.4 vs 0.13 seconds without vs with redirecting to a file!

Adding ' /v PortName' switch will give less output so speed up, but earlier reg.exe won't support both switches ' /s /v' together. I used HARDWARE example for the question since most people don't have FTDI VCP.

[Xiangrui Li]

I found that Windows's findstr helps at this case.

ftdiKey = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\FTDIBUS';
tic, [~, str] = dos(['reg.exe query ' ftdiKey ' /s']); toc
 Elapsed time is 1.442744 seconds.

ports = regexp(str, 'COM\d{1,3}(?=\s)', 'match'); % get all FDTI VCP

disp(numel(str))
  17740

Following is much fast, but does not work for early reg.exe:

tic, [~, str] = dos(['reg.exe query ' ftdiKey ' /s /f PortName']); toc
Elapsed time is 0.057759 seconds.

disp(numel(str))
  1631

findstr is similar to unix grep, although may not as powerful as grep:

tic, [~, str] = dos(['reg.exe query ' ftdiKey ' /s | findstr PortName']); toc
 Elapsed time is 0.141830 seconds. 
disp(numel(str))
   345

The file approach is similar to findstr:

tic, [~, str] = dos(['reg.exe query ' ftdiKey ' /s >myFile']);
fid = fopen('./myFile'); str = fread(fid, '*char')'; fclose(fid); toc
 Elapsed time is 0.130066 seconds.

Anyway, it seems the system/dos function has a huge room to improve. My wild guess is that the pre-allocation for the output may not be dealt with efficiently.

[Xiangrui Li]

Just found jsystem at Github based on java. It can be super fast. For example, for the FTDI example:

ftdi = 'HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\FTDIBUS';
cmd = ['reg.exe query ' ftdi ' /s'];

tic; [err, str] = jsystem(cmd, 'noshell'); toc
  Elapsed time is 0.045354 seconds.

With system():

tic; [err, str] = system(['reg.exe query ' ftdi ' /s']); toc
Elapsed time is 1.438463 seconds.

The time is 1.44 vs 0.045 seconds! However, with the HARDWARE example:

cmd = 'reg.exe query HKEY_LOCAL_MACHINE\HARDWARE /s';
tic; [err, str] = jsystem(cmd); toc % this won't work

This hangs Matlab with or without 'noshell' option. I have no idea why this happens.

[raym]

I encountered similar problems and finally got the reason, but not solution.

When matlab runs dos('command'), the thread was diverted to the dos command and wait it to finish. e.g., if ping www.baidu.com takes 10 s, the tic toc will take more than 10s. During this time, the command window is not responsive. One solution is add a "&" at the end. this will get thread back to command window immediately. A drawback is that it will failed to return the output of command, i.e, you lost monitoring of your command and do not know when it finishes.

[Xiangrui Li]

The jsystem at Github has fixed the hanging problem.

[Horshack Horshack]

I'm having the same issue. To demonstrate, I wrote a simple C program 'printstdout' that prints the number of characters it's been told to via a command line argument:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char* argv[]) {

    int     numCharsToPrint = argc >= 2 ? atoi(argv[1]) : 0;
    int     numCharsPrintedThisLine;

    numCharsPrintedThisLine = 0;
    while (numCharsToPrint-- > 0) {
        printf("*");
        if (++numCharsPrintedThisLine == 78) {
            numCharsPrintedThisLine = 0;
            if (numCharsToPrint-- > 0)
                printf("\n");
        }
    }
    return 0;
}

And tested it with the following script:

    numChars = uint32(1);
    while (numChars <= 2^20)
        fprintf("numChars = %7d: ", numChars);
        tic; [exitCode, output] = system(['printstdout ' num2str(numChars)]); toc;
        numChars = numChars*2;
    end

Here's the output for Matlab:

numChars =       1: Elapsed time is 0.035415 seconds.
numChars =       2: Elapsed time is 0.034140 seconds.
numChars =       4: Elapsed time is 0.033591 seconds.
numChars =       8: Elapsed time is 0.034396 seconds.
numChars =      16: Elapsed time is 0.033863 seconds.
numChars =      32: Elapsed time is 0.033824 seconds.
numChars =      64: Elapsed time is 0.033170 seconds.
numChars =     128: Elapsed time is 0.033617 seconds.
numChars =     256: Elapsed time is 0.033296 seconds.
numChars =     512: Elapsed time is 0.034161 seconds.
numChars =    1024: Elapsed time is 0.032989 seconds.
numChars =    2048: Elapsed time is 0.033904 seconds.
numChars =    4096: Elapsed time is 0.602469 seconds.
numChars =    8192: Elapsed time is 0.703521 seconds.
numChars =   16384: Elapsed time is 0.903753 seconds.
numChars =   32768: Elapsed time is 1.304120 seconds.
numChars =   65536: Elapsed time is 2.105577 seconds.
numChars =  131072: Elapsed time is 3.708676 seconds.
numChars =  262144: Elapsed time is 6.916353 seconds.
numChars =  524288: Elapsed time is 13.246939 seconds.
numChars = 1048576: Elapsed time is 26.452178 seconds.

And here is the output from Octave:

numChars =       1: Elapsed time is 0.0197449 seconds.
numChars =       2: Elapsed time is 0.0113029 seconds.
numChars =       4: Elapsed time is 0.0113032 seconds.
numChars =       8: Elapsed time is 0.0113418 seconds.
numChars =      16: Elapsed time is 0.011956 seconds.
numChars =      32: Elapsed time is 0.010741 seconds.
numChars =      64: Elapsed time is 0.011215 seconds.
numChars =     128: Elapsed time is 0.0108809 seconds.
numChars =     256: Elapsed time is 0.011487 seconds.
numChars =     512: Elapsed time is 0.010798 seconds.
numChars =    1024: Elapsed time is 0.012202 seconds.
numChars =    2048: Elapsed time is 0.0105009 seconds.
numChars =    4096: Elapsed time is 0.0121739 seconds.
numChars =    8192: Elapsed time is 0.01156 seconds.
numChars =   16384: Elapsed time is 0.0121598 seconds.
numChars =   32768: Elapsed time is 0.0127161 seconds.
numChars =   65536: Elapsed time is 0.014755 seconds.
numChars =  131072: Elapsed time is 0.0170429 seconds.
numChars =  262144: Elapsed time is 0.023046 seconds.
numChars =  524288: Elapsed time is 0.0346019 seconds.
numChars = 1048576: Elapsed time is 0.0581162 seconds.