Suggestions on how to detect anomalies in around 200k time series, maybe with deep learning, maybe in a fast way?

3 visualizaciones (últimos 30 días)
I have tried Time Series Anomaly Detection Using Deep Learning, with just a few thousands of time series, but it is very slow...
Any suggestion to get anomalies detected with 200k time series in a faster way?

Respuesta aceptada

Eamonn
Eamonn el 18 de Jun. de 2022
Hello
The DAMP algorithm (in Matlab) can do anomlay detecion at 100,000+ Hz
It is the only time series anomlay detecion algorthm to process a dataset with one trillion datapoints.
OH, and it beat deep learning on all the benchmarks.
It will be here https://www.cs.ucr.edu/~eamonn/MatrixProfile.html next week, but in the meantime, if you want the code/paper, I am happy to share it.
  6 comentarios
Sim
Sim el 21 de Jun. de 2022
Thanks @Eamonn, I think I will drop you an email later...
Meanwhile, just for a sake of completeness, I would like to inform you that I might have found "a way" to overtake the error coming out when using DAMP_topK. Indeed, I changed
CurrentIndex = 1000+1; % line 9 of DAMP_topK
into
CurrentIndex = 24;
getting this figure
However, if I change that parameter into
CurrentIndex = 50;
I get the following:
By changing CurrentIndex, the predicted scores/positions change a bit (i.e. the first two "peaks" in the previous case, i.e. with CurrentIndex = 24 disappear and, obviously, two "lower peaks" appear with CurrentIndex = 50, in order to always have K = 5)...
Therefore, what is the right value for the variable CurrentIndex ?
Sim
Sim el 21 de Jun. de 2022
Editada: Sim el 21 de Jun. de 2022
@Eamonn, email just sent to you :-)
P.S.: just in case I attach here again the ts.mat file (including both "w3" and "w4" time series)

Iniciar sesión para comentar.

Más respuestas (1)

Image Analyst
Image Analyst el 18 de Jun. de 2022
Are you talking about training or prediction?
For what it's worth, here is another anomaly detection demo from Mathworks.
Learn how to apply statistical and machine learning based anomaly detection techniques to industrial processes and machinery.
  3 comentarios
Image Analyst
Image Analyst el 19 de Jun. de 2022
Well of course you could always get a faster computer with more memory. Get the best video adapter you can afford with the most GPU memory you can afford. And get the Parallel Processing Toolbox, which will speed it up quite a bit (if you end up having to use the CPU because your GPU does not have enough memory and you get an "out of memory" error). The GPU does not know how to use disk space to allocate additional memory (everything happens on the GPU) but the CPU can get as much virtual memory from the disk drive as it needs (or that you allow).
Sim
Sim el 20 de Jun. de 2022
Thanks a lot @Image Analyst, in this moment I do not have access to faster machines, but I could try with the parallel processing toolbox. However, by considering the paper/s showed by @Eamonn, it looks like that (for time series) deep learning is outperformed by another algorithm based on the Matrix Profile :-)

Iniciar sesión para comentar.

Categorías

Más información sobre Reporting and Database Access en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by