Big Data analysis using Matlab and Database connection, is it possible ?
2 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
Taufik Sutanto
el 13 de Oct. de 2013
Comentada: Taufik Sutanto
el 16 de Oct. de 2013
Hi,
I have a code that analyze large data in a database like follows:
for i=1:N %N is large
A=exec(con,query);*%retrieving ONLY one row of data from a database
A=fetch(A);
A=A.data;
... %other calculations
end
It always stall [GC/memory error] after few hours. I already tried both of these methods to get around it:
But still have no luck. I already increase my Java Heap memory to 8Gb and my machine memory is quite large 32Gb. Furthermore I only retrieve one row at a time in each iteration and use the same variable, therefore I don't think it's because of memory insufficiency.
Can anybody help me with this issue, any help will be greatly appreciated. Thanks.
0 comentarios
Respuesta aceptada
Yair Altman
el 15 de Oct. de 2013
Editada: Yair Altman
el 15 de Oct. de 2013
1. Try to disconnect from the DB every now and then, ensuring that all references are explicitly cleared so that there's no dangling references out there that cannot be GC'ed. After disconnecting and before you reconnect, perform a manual GC (JheapCl makes this easy, or run the simpler java.lang.System.gc).
2. Try to fetch data in bulks rather than in separate rows. This would improve performance as well as decrease resources.
3. Ensure that you're connecting to the DB directly via JDBC rather than an ODBC bridge
Yair Altman
1 comentario
Más respuestas (0)
Ver también
Categorías
Más información sobre Matrix Indexing en Help Center y File Exchange.
Productos
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!