Taking the 3D FFT of a matrix with significant memory, ~GB

I am trying to take the 3D FFT of a large 3D array, but as soon as the memory of the array reaches 5 GB (i.e. a 8192x256x256 matrix) or so, the run times seem to increase exponentially, even though I'm on a 16GB system (and the same happens on another 16GB system). In the following code, I try several ways of taking the 3D FFT, and in each case I run into this problem.

aa=zeros(6,7);
for na=6:6

aa(na,1)=256*2^na;

tic;
VV=rand(aa(na,1),(256),(256));VV1=VV;
aa(na,2)=toc

tic
VV1=fft(VV,[],2);
aa(na,3)=toc
%
% tic
% VV=fft(VV1,[],3);
% aa(na,4)=toc

% tic
% VV1=fft(VV,[],1);
% aa(na,5)=toc

tic
for it=1:256
    for iq=1:256
        VV1(:,it,iq) = fft(VV(:,it,iq));
    end
end
aa(na,6)=toc

tic
VV=rand(aa(na,1),(256),(256));
VV1=fftn(VV);
aa(na,7)=toc

end

aa(:,1)=10^-9*256*256*aa(:,1);
figure()
hold on
plot(aa(:,1),aa(:,2))
plot(aa(:,1),aa(:,3))
plot(aa(:,1),aa(:,4))
plot(aa(:,1),aa(:,5))
plot(aa(:,1),aa(:,6))
plot(aa(:,1),aa(:,7))
hold off
ylim([0,max(aa(:,3)+aa(:,2))]);
xlim([0,max(aa(:,1))]);
xlabel('GB')
ylabel('time')

The output of the code is given below. The results are similar for Matlab version 2016a and 2018b, and if I restructure the matrix while maintaining its same memory. The first column is proportional to the memory (the last row corresponds to a ~ 5 GB memory matrix), and the other columns correspond to the run times associated with different ways of taking the 3D FFT. It can be seen that around 5 GB the computation time seems to begin increasing exponentially. Over 10 GB or so, it is not clear that the fftn can be completed at all. The output is given below:

10^4 *

Is there any way to prevent this tremendous decrease in speed when taking the 3D FFT of a large matrix? In particular, I'd like to be quickly taking the FFT of a 5 GB matrix or 10 GB matrix in a simulation on my 16 GB computer.

Answer 1

[Vishal Bhutani]

Refer to the link attached below, it might be helpful to resolve the issue:

https://www.mathworks.com/matlabcentral/answers/435424-i-need-to-perform-a-3d-fourier-transform-on-a-large-3d-array-10-9-data-points-is-there-anyway-i-c?s_tid=answers_rc1-1_p1_MLT

As the data size is increasing it might be taking time to compute FFT. Try clearing the variables with each iteration which are not required.