Estimate the memory for SVD

Hello All,

I have a matrix F with size 35*66192. I calculated the co-variance matrix by using R = F'*F, and the size of matrix R is 66192*66192. It meas the memory used by the matrix is: 66192*66192*4 = 16.33G.

Then I calculate SVD by using [U,S,V] = svd®, where the sizes of matrix U,S,V are all 66192*66192, and thus the memory needed to store these three matrix U,S,V are 16.33*3=48.99G.

So, then the total memory is 65.33G. I am using a 64-bit server computer, and the operating system is Windows Server 2008 Enterprise. The physical memory is 64G, and the virtual memory is set to 10G. I typed 'memory' command in Matlab, and I got the following information:

Maximum possible array: 71580 MB (7.506e+10 bytes) * Memory available for all arrays: 71580 MB (7.506e+10 bytes) *

That means the total memory is enough for computing the matrix and its SVD. But when I run the following code, I still got the error: "Error using svd. Out of memory."

Codes:

F=detrend(F,'constant');

R = F'*F;

[U,S,V] = svd®; %error happened at this line

Any of your reply and suggestion is highly appreciated and anticipated!

Answer 1

[Matt J]

Seems much more efficient to do

[Uf,Sf]=svd(F.',0);

and recognize that the SVD of F.'*F can be represented in economical form as

 F.'*F = Uf*(Sf.^2)*Uf.';

In this decomposition, Uf will be the same size as F and Sf is merely 35x35

[Xintao]

Thank you! I want to explain my question as follows. Could you please have a look?

    N=3500; % number of variables

    M=35; % number of samples

    r = rand(M, N);

    f = detrend(r);

    [P,D,V] = svd(X/sqrt(M-1),0);     P(:,M)=[];

    D=diag(D);    D(M)=[];

What I got from the above codes will be the same as below? That is, P=P1, D=D1 and V=V1?

C = (X*X')/(M-1)

[P1,D1,V1] = svd(C, 0);

[Matt J]

You haven't shown the dimensions of X, or its relationship to r and f.

In any case, C is symmetric and therefore you would have P1=V1, whereas unless X is symmetric, you may not have P=V. However, P=P1 for columns corresponding to non-zero singular values. Also, D=sqrt(D1) for the non-zero singular values. You can test all this yourself, using example X with small dimensions.

Answer 2

[Jan]

When did you test the available memory? The two lines before the SVD call can reserve memory already, such that there are still enough free bytes, but perhaps not in contiguous blocks. Storing 4 arrays with 16GB in 71 GB is a very hard job and I'm not surprised, that it fails.

In addition SVD needs a not negligible chunk of memory as workspace for the storing the intermediate values. Therefore I'd suggest to increase the RAM by a factor of two.

[Xintao]

Thanks Jan!

1. I tested the memory before I call the three lines.

2. Yes, you are right. The two lines before the SVD call can reserve memory already. Then I added memory command just after the first two lines and before the third line as follow:

F=detrend(F,'constant');

R = F'*F;

memory;

[U,S,V] = svd( R); % out of memory still occurred here.

And I got the available memory as: "Maximum possible array: 54829 MB (5.749e+10 bytes) * Memory available for all arrays: 54829 MB (5.749e+10 bytes) *". And the out of memory error remained the same.