script running for 13 hours. should I be worried?
1 visualización (últimos 30 días)
Mostrar comentarios más antiguos
Alex O'Neill
el 5 de Feb. de 2015
Respondida: Jan
el 5 de Feb. de 2015
I have a function going through 9 excel files with about 1.7mil cells. It imports them, puts everything into one column then goes through and removes all the 0's and saves the output of about 300k cells. Its been running for about 13 hours of so now and I'm concerned that I may have messed up the code so I was hoping someone would help me out a fresh set of eyes.
it does look like its working, maybe? the output files look with theyve been edited, but I can't really be sure. this is the code i'm using at the moment.
function [ m ] = combine( input_args )
source = 'C:\Users\aoneil\Desktop\ODEN\trimmed'; %input the directory to choose the source folder
dest_dir = 'C:\Users\aoneil\Desktop\ODEN\graphs'; %choose where to output the resulting file
files = dir(fullfile(source, '*.CSV'));
for j = 1:length(files) %loops through all the file in the directory folder
a = xlsread(fullfile(source, files(j).name));
for x = 1 : size(a,1) %steps through the matrix
d((1+((x-1)*size(a,2))):(size(a,2)*x), 1) = a(x,1:end); %saves all the values of original to matrix "d"
end
n=1;
for y=1:size(d,1)
if d(y,1) > 0
m(n,1) = d(y,1)
n=n+1;
end
end
csvwrite(fullfile(dest_dir, files(j).name), m); %outputs matrix "d" to the file name/directory chosen
end
end
0 comentarios
Respuesta aceptada
Jan
el 5 de Feb. de 2015
You urgently require a "pre-allocation". Letting arrays grow iteratively is extremely expensive. Example:
x = [];
for k = 1:1e5
x[k] = rand;
end
Now in each iteration a completely new vector x is created and the older values are copied. This means, that the OS has to reserve sum(1:1e5) * 8 bytes in total and copy almost the same amount of data in addition.
A vectorization solves the pre-allocation implicitly:
n=1;
for y=1:size(d,1)
if d(y,1) > 0
m(n,1) = d(y,1)
n=n+1;
end
end
Better without loop and IF:
m = d(d > 0);
0 comentarios
Más respuestas (1)
Alessandro Masullo
el 5 de Feb. de 2015
You can use a task manager to check if matlab is still using the CPU. If not, it probably got stuck.
0 comentarios
Ver también
Categorías
Más información sobre Data Import from MATLAB en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!