out of memory

I am trying to find covariance matrix of a matrix of size 3 by 65536, and getting "out of memory" error. I want to avoid this error please suggest some way to get rid of this error.

Thanks

Answer 1

[Anton Semechko]

Here is an example:

N=6E4; % number of variables
M=10; % number of samples

% Simulate M samples and store them into one matrix
X=zeros(N,M);
for i=1:M
    X(:,i)=[4*randn(3E4,1);randn(3E4,1)];
end

% Compute the mean and center
Xave=mean(X,2);
X=bsxfun(@minus,X,Xave);

Now your covariance matrix would be defined as C=(X*X')/(M-1). Because N>>>M, this covariance matrix will only have (M-1) non-zero eigenvalues. Since we want to avoid computing C explicitly, we can get the required eigen-decomposition with the use of SVD:

[P,D,~] = svd(X/sqrt(M-1),0);
P(:,M)=[];
D=diag(D);
D(M)=[];

where P is N-by-(M-1) matrix containing the eigenvectors of C and D is the list of corresponding eigenvalues.

[Sean de Wolski]

Very Interesting +1

[Anton Semechko]

sorry, to get the eigenvalues you will have to take a square root of D

[Muhammad]

Thank alot Anton Semechko

[Muhammad]

Thanks alot Anton Semechko

[Muhammad]

Thanks alot Anton Semechko

[Xintao]

If I do it in the following way, then the results should be the same? That is: P=P1 and D=D1?

C=(X*X')/(M-1);

[P1,D1,~] = svd(C);

Answer 2

[Anton Semechko]

I am assuming what you really meant to say is that you have 65536 data points in R3 and you are trying to compute the covariance matrix of size 65536. Unless your machine has at the minimum (65536^2)*8*3/2^20 =98304 GB of memory you will not be able to do that.

There are ways around it, however. It all depends on why you need the covariance matrix to begin with. If for instance you are going to perform orthogonal decomposition (i.e. find the eigenvalues and eigenvectors), then this operation can be done with the help of SVD which does not require explicit calculation of the covariance matrix.

[Walter Roberson]

Gigabyte is 2^30 not 2^20, so it would be 96 Gigabytes rather than 98304 Gigabytes.

[Muhammad]

Thanks Anton Semechko for answering

Basically I am trying to find PCA, for that I first have to find covariance matrix. I want to know what should I do to avoid "out of memory" error, by increasing the size of virtual memory or RAM or something else, please guide me in this regard. Thanks

[Walter Roberson]

There are routines in the stats toolbox, but I do not know if they have ways of reducing memory; see http://www.mathworks.com/help/toolbox/stats/brkgqnt.html#f75476

But yes, make sure you use a 64 bit version of MATLAB, push up your virtual memory to at least 100 gigabyte (256 Gb would be safer), and let it go. It probably won't take more than 4 years.