Explore - Scientific Workflow Management Tool

Lightweight workflow management tool to automatically retrieve previous intermediate results and track provenance in scientific workflows.

Explore is a scientific workflow management tool. You can visualize the data provenance graph and when your code is changed and re-executed, only the necessary functions are re-executed to provide an updated version of your data. This can save a lot of time, especially when experiments are constantly refined. You can easily declare an experiment as a directed acyclic graph (DAG) where the nodes are functions and the edges represent variables that are produced and consumed by functions.

During the first execution of the graph, variables are persisted to the disk which implies a longer graph execution time (due to variable loading and saving duration). However, for future executions, if the node main function and all the sub-functions called during the node execution remain unchanged (a) and the input variables of the node also remain unchanged (b), then the results are retrieved from the disk and not re-computed.

For (a), the major time consumer is the code analysis to find out the sub-functions involved in the execution of a node. Therefore, this "node sub-function dependency" information is also persisted based on the content of the sub-function files. Then, if the sub-function files did not change, it is assumed the node's sub-function dependencies did not change.
For (b), either the variable content is hashed (i) or the variable mat-file date is taken (ii) as signature to check that the variables did not change. Option (i) is advantageous as re-computed nodes might still output the same values. In that case, unchanged child nodes will not be re-computed. However, option (ii) might be faster for bigger variables, where hashing would take too much time.

Thank you to:
- Jan (2019). DataHash (https://www.mathworks.com/matlabcentral/fileexchange/31272-datahash), MATLAB Central File Exchange. Retrieved April 15, 2019.
- Aslak Grinsted (2019). cachedcall (https://de.mathworks.com/matlabcentral/fileexchange/49949-cachedcall), MATLAB Central File Exchange. Retrieved April 15, 2019.
- Aslak Grinsted (2019). rgb.m (https://de.mathworks.com/matlabcentral/fileexchange/1805-rgb-m), MATLAB Central File Exchange. Retrieved April 15, 2019.
- All upstream dependencies

For tutorials, detailed information and bug reports, please check the Github (https://www.github.com/jahsue78/explore) or the corresponding conference paper [1].

[1] Ah Sue, Jonathan, et al. "Optimizing Exploratory Workflows for Embedded Platform Trace Analysis and Its Application to Mobile Devices." International Conference on Human-Computer Interaction. Springer, Cham, 2019.