have a 60027x3397 matrix. how to multiply each column with every other colum of same matrix (using mapreduce) as MATLAB runs out of memory

Hello, I have a matrix A=60027x3397. I need to multiply each of the 3397 column with each other into a new matrix B. The combinations nC2 for 3397 turns out to be 5678106 and hence the MATLAB runs out of memory. I want to use MapReduce for this and was hoping to get an initial push for the same. I already have another matrix "k" 5678106x2 with the column indexes for the A for multiplication.

[Walter Roberson]

What are you going to do with the values afterwards? It might perhaps not be necessary to store the results.

[Anurag Passi]

I have to then take the column sum of the new matrix which i believe would be 1X5678106 vector. Convert this vector to an adjacency matrix with the values from the sum.

[CS Researcher]

Can you not sum that on the fly? Like

total = 0;
for i = 1:3397
     for j = 1:3397
         total = total + sum(A(:,i).*A(:,j));
     end
end

This way you will not have to save every value.

[Anurag Passi]

I tried this but MATLAB runs out of memory as the combinations are to many.Also, with this the A(i,i) will also be included. I do not want diagonal elements. So, i=j is NOT suitable.

[CS Researcher]

You can skip that with an if condition

if i ~=j
   total = total+ ...
end

Also, what if you break it in parts

for i = 1:500
    for j = 1:500
        if i ~= j
           total = ...
        end
    end
end

 for i = 501:1000
      for j = 501:1000
          if i ~= j
             total = ...
          end
      end
 end

and so-on...

You can find better ways to write it.

[Anurag Passi]

nope....not working. I think MapReduce is the only option. Any help is appreciated

[Walter Roberson]

How large are you expecting the adjacency matrix to be? And can it possibly be sparse? Will most values be non-zero or zero?

If you are taking the column sum, then use dot()

C = size(A,2);
numout = C * (C-1) / 2;
adj = zeros(numout, 1);
idx = 0;
for C1 = 1 : C - 1
  AC1 = A(:,C1);
  for C2 = C1+1 : C
    idx = idx + 1
    adj(idx) = dot(AC1, A(:,C2));
  end
end

This if you are using the dot product directly as the weight. If you are filtering based upon value and that can be done by looking at individual results, then it would be better to write into a sparse matrix, in a different order than what I use here.