store a huge number of rows in a in a matrix (out of memory error)

Hello, I really do appericate any kinds of help, after 6 months that I've worked on this code now I found i could not store results because of out of the data memory.!?!

I have a code for light tracing, for creating a holographic image, which for a single point source calculates the desire location with parallel computing. in this way for each point source as an input, I have around approximately thousands or millions of vector stores in a matrix. now I wanted to use of a point cloud, instead of on a single point, and after calculation, the result for each point in the point cloud, store them in a new matrix. but by runnig, i see the out of memory error because my numbers of rows exceed more than 500 milion rows. I searched through the internet and become familiar with tall matrix but as far as I understand that matrix is just read-only and could not write my analyses in each for-loop.

one other option that I found is save my data after a specific number of rows in a data store and clear my memory. but according to this code, I should know the total numbers of my data. but my code can find the numbers of the row for each point within the code.

this is some part of my code:

%% in here i loade my point cloud

obj=load('source/Flower.xyz'); % object point data file load

hologram_plane_file='holo.jpg'; % hologram plane save file name

hologram_simul_file='recon.jpg'; % hologram simulation save file name

x0=obj(:,1); % object plane

y0=obj(:,2);

z0=obj(:,3);

%%after applying the rotation on scaling on my point cloud i use of the for loop to have analys for each point:

for v = 1:length(x0)

posSource(1)=x0(v);

posSource(2)=y0(v);

posSource(3)=z_pointcloud(v);

%%%%%

afer numbers of code line in here i have the desire location of insident light in here:

ux=uxBottom(mod(nr_low,2)==1 & mod(nr_up,2)==1);

%% because i use of the paralle computing by useng the matrix i use of the find function to acess the col and row of desire file( inhere i will find how many rows i have for a single point and it is different for each point)

[ row, column]=find(uxBottom(mod(nr_low,2)==1 & mod(nr_up,2)==1));

x_h=x_h(:);

y_h=y_h(:);

z_h=z_h(:);

uxBottom_xyz=uxBottom_xyz(:);

uyBottom_xyz=uyBottom_xyz(:);

uzBottom_xyz=uzBottom_xyz(:);

%% now i can find the locations which i needed and i should save this X_f, Y_f, Z_f with its posSource in a new matrix:

x_f=x_h(row ,1);

y_f= y_h(row ,1);

z_f= z_h(row ,1);

unitx=uxBottom_xyz(row ,1);%%unit vector for the specific location

unity=uyBottom_xyz(row ,1);

unitz=uzBottom_xyz(row ,1);

sz=size(x_f);

posSourcex=posSource(1).*ones(sz);

posSourcey=posSource(2).*ones(sz);

posSourcez=posSource(3).*ones(sz);

bigdata=[bigdata;posSourcex,posSourcey,posSourcez,x_f,y_f];%% here is where i see the error after have calculation of some part of my point cloud.

Answer 1

[Stephen23]

"I searched through the internet and become familiar with tall matrix but as far as I understand that matrix is just read-only..."

Tall tables are writable, the documentation shows how to assign to tall arrays using indexing:

https://www.mathworks.com/help/matlab/import_export/index-and-view-tall-array-elements.html#bvexfnf

Note that the RHS data must be a scalar or a tall table, so if you want to assign multiple data values at once you will need to convert it to a tall table.

[sahar kheibarihafshejani]

I think my results are not encluded any data type which is allowed to assign to a tall array. in each for loop, I have a matrix which its rows are varied for each input.