Can I obtain the covariance matrix of a stochastic process with plenty of measurements?

Hello everyone.

I have implemented the Karhunen-Loève expansion as per this question.

I have tested it with samples of data obtaining possitive results.

However, now I am dealing with a matrix of dimensions 211302*50, meaning that I have roughly 200000 observations of 50 random variables.

If I try to calculate the covariance matrix of these using the observations as columns (as per the previus link), the program crashes and the error returned is:

Error using  * 
Requested 211302x211302 (332.7GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take
a long time and cause MATLAB to become unresponsive.

Error in cov (line 155)
c = (xc' * xc) ./ denom;

Related documentation

Is there a way to do what I want or is the matrix just too big?

Best regards.

Jaime.

[Sharmin Kibria]

As the error suggested, the covariance matrix you are trying to build is too big. Your solution needed to allocate memory that was larger than what is allowed for array storage. That is why MATLAB crashed.

Answer 1

[Bjorn Gustavsson]

To me it seems that you have misunderstood the dimensions. Your covariance-matrix should, to my understanding be 50x50 when you have 50 random variables observed 200000 times. With a call like this:

tic,C = cov(randn(211302,50));toc

I get a 50-by-50 covariance-matrix C in ~0.22 s.

HTH