preallocating sparse 4d matrix

Hi

I've got some code that preallocates some matrix L = zeros(50,50,50,50); and then iterates along the first two dimensions (for i = 1:50, for j = 1:50...) to populate L. At the end of my code I reshape L into a 2500x2500 matrix, and this is it in usable form.

I want to increase the dimensions from 50 to 100, but it's hitting a memory bottleneck when preallocating, and I can't use spalloc to generate a 4d matrix. Rather than re-writing all my loops, can anyone think of another way to get around this? The end result (a sparse 10000x10000 matrix) should be easy to create, but I can't make it sparse until it's 2d, which is after my calculations.

Thanks for any help Mike

[Walter Roberson]

Could you use 'float' or even one of the integer data types instead of double for L ?

[Michael]

I'm not sure what float does, but L is only ever 0, 1, or some simple fraction -1/2, -1/3, -1/4. I suppose I can use uint8 and map -1/2 to -2, -1/3 to -3, etc., then after I've reshaped it and made it sparse, I can remap -2 to -1/2?

still, I'm having a problem writing L = uint8(zeros(50,50,50,50)), is there another way I can write this which doesn't automatically generate the double array?

[Walter Roberson]

L = zeros(50,50,50,50, 'uint8');

Though if you want negatives, you will want 'int8' instead of 'uint8' :)

[Michael]

this method is working, although it doesn't seem I can generate a sparse matrix with int8 inputs, and this is now creating memory problems further down the line where I need to calculate L'*L.

[Walter Roberson]

Ah yes, sparse is restricted to double or uint8 (or logical, I think).

You could encode the negatives into uint8, e.g., 254 for -2 (which would be the two's complement to make it easier to track.) Doing the translation later might prove to be time consuming. It might be faster to record indices into a lookup table, and do something like

usedidx = find(L);
[idx1, idx2] = idx2subs([N*N, N*N], usedidx);
sparse(idx1, idx2, LookupTable(L(usedidx)), ...)  %recheck arg order!

Notice I did an implicit reshape there by way of the first argument to idx2subs()

Answer 1

[James Tursa]

For multi-dimensional sparse arrays, see this FEX submission by Matt J:

http://www.mathworks.com/matlabcentral/fileexchange/29832-n-dimensional-sparse-arrays

MATLAB only supports double and logical sparse arrays. For int8 sparse arrays, you will have to wait until I release my int8sp class in a month or two (basic code is mostly complete, but not fully tested yet).

Answer 2

[Teja Muppirala]

You can't make a sparse array with more than 2 dimensions, but you could make a 100x100 cell array that is filled with 100x100 sparse matrices. And then you can work on each cell separately, and then go back and shape it into a 10000x10000 afterwards.

L = repmat({sparse(100,100)} ,100,100);

for i = 1:100
  for j = 1:100
     L{i}{j} = ...
  end
end