Incorrect table size, unsure why.

start_ROI = 1

stop_ROI = 9

bin = 180*60

start_time = 172827;

final_time = start_time + 259200;

for j = (start_ROI:stop_ROI)

%Table to hold frames moved data

Frames_movedabove = [];

Frames_movedbelow = [];

for i = (start_time:bin:final_time)

%Filter out respective thresholds and calculate total # frames moved

abovetest = asabove(i:(i+bin),j);

belowtest = sobelow(i:(i+bin),j);

%Create tables with total # frames moved above threshold

Frames_movedabove(length(Frames_movedabove)+1) = sum(abovetest);

Frames_movedbelow(length(Frames_movedbelow)+1) = sum(belowtest);

end

end

geno_1 = 1:9

for i = (start_time:bin:final_time)

Dummyabove = [];

Dummybelow = [];

%Filter out

Abovetable = asabove(i:(i+bin),geno_1);

Belowtable = sobelow(i:(i+bin),geno_1);

Dummyabove(length(Dummyabove)+1,:) = sum(Abovetable);

Dummybelow(length(Dummybelow)+1,:) = sum(Belowtable);

end

Hello,

I have some code I wrote for analyses that I have been tasked to do. My individual analysis code, above, works fine. I am able to go throughout all my RegionsOfInterest, per specified timeslot, and then graph it all.

When trying to recreate it for a population analysis, is when I run into issues. My idea was to go into my "overall data table", pull out only a genotype's worth of data (rather than simply 1 ROI-- in this case 1 genotype is ROIs 1:9), average out the values for each timeslot, and then graph the averages for a 25hr "every ROI in this genotype" graph. My main hiccup is thus: when I run the "individual" code, my Frames_movedabove/below tables are 1x25, which is what I want, as it's 1 data point per hour. However, my equivalent table for the population analysis, rather than being 9x25, is 1x9.

I attached a photo of what my final graph for individual analysis looks like. End goal is to output one like that, but for an entire population rather than simply 1 ROI. I am new to matlab so sorry if this is a trivial question and thanks in advance for the help.

[Matt J]

However, my equivalent table for the population analysis

Which variable in your code represents this table? If it's Dummyabove and Dummybelow, you are discarding all of the results they contain every time you do this at the top of the loop,

    Dummyabove = [];
    Dummybelow = [];

In any case, this looks like the wrong approach if all you are trying to do is partition matrices into bins and average them, see my answer below.

Answer 1

[Matt J]

It's not clear what the sizes of all the given variables in your code are. However, to divide a matrix into BINx1 strips and average them individually, you don't need to use such a complicated pattern of loops. Just use appropriate reshaping, e.g.,

bin=180*60;

asabove=rand(bin*9,25);

asaboveBinned=reshape( mean( reshape(asabove,bin,[]) )  ,[9,25]);

whos asabove*

  Name                   Size               Bytes  Class     Attributes

  asabove            97200x25            19440000  double              
  asaboveBinned          9x25                1800  double              

[Matt J]

Or, even more simply, download sepblockfun,

https://www.mathworks.com/matlabcentral/fileexchange/48089-separable-block-wise-operations

bin=180*60;
asabove=rand(bin*9,25);

asaboveBinned=sepblockfun(asabove,bin,'mean');

whos asabove*

  Name                   Size               Bytes  Class     Attributes

  asabove            97200x25            19440000  double              
  asaboveBinned          9x25                1800  double              

[Miguel]

I've been using the loops because the size that I need is unfortunately not static. For context: I use a platform called MARGO to track fruitflies. I then wrote the first script to determine whether they were above or below a certain threshold, adding it into my dummy tables (for above or below) and then graphing. This then gets repeated for however many regions of interest I have (which can be as little as 2 or 50+).

My "main table" that I get all my data from also varies in size. For this example, they're both 649762x54. From here, I scan for specific frames in a time and then do my tabulating. Because the size of every table is not static, and I have to scan the "main tables" for specific times/data, which changes per run, I'm unsure if I'd be able to do the reshaping. In this case, my "working averaging table" would be 9x25, but could change to be 12x25, 2x25, etc. It depends on the populations of my flies.

When I try using a similar method as my working code, with

        Dummyabove(length(Dummyabove)+1) = sum(Abovetable);

I get the following error: "Unable to perform assignment because the indices on the left side are not compatible with the size of the right side." which makes sense since I'm trying to put 1x25x9 (if that is correct?) data into a 1x25. Would using reshape fix this?

[Matt J]

In this case, my "working averaging table" would be 9x25, but could change to be 12x25, 2x25, etc.

Not sure why that would be a problem. All that seems to say is that you will be re-applying sepblockfun to different data sets, but with different 'bin'.

To be clearer, you should provide a data set for which this,

asaboveBinned=sepblockfun(asabove,bin,'mean');

doesn't work and we can take it from there.

[Miguel]

Seems sepblockfun isn't cooperating with my data set. When attempting to use it, I get the following error "Error using reshape. Size arguments must be real integers. +Error in sepblockfun (line 110)

X=reshape(X,args{:}); Error in PopulationLightDarkPOC (line 4). testing = sepblockfun(asabove,bin,'mean');"

I tried installing the companion, N-dimensional sparse arrays since it mentioned working on marices, but it did not work either.

I attempted these three:

testing = sepblockfun(asabove,bin,'mean')

testing = sepblockfun(asabove,[9,25],'mean')

for i = start_time:bin:final_time
testing = sepblockfun(asabove,[9,25],'mean')
end

I have attached my asabove dataset to help out with understanding why. I had to compress it, as it is quite large.

[Matt J]

Please attach asabove in a .mat file, not as .csv.

But regardless, the error message is saying that the number of rows in your matrix is not an even multiple of 'bin'. Your original code should issue the same complaint.

[Miguel]

Fixed attachment.

In original code, it works just fine. I'm able to, for every specified ROI range, go into the data tables, pick out my frames of importance from my start time to my end time for every bin (which is equal to frames in an hour), save it to a dummytable for frames above/below certain threshold per bin(hour), and then graph it all like the image shows.

However, when attempting to do these same analyses for an entire collection of ROIs at once (genotype 1 equaling ROI 1-9 for example) to display a population's trends for crossing the threshold rather than an individual's, is when I run into some problems. Thank you for taking the time to answer and try to help. I really appreciate it.

[Matt J]

What are start_time, bin, and final_time for your attached example?

[Miguel]

start_time is 172827; bin is 180*60, final_time is 432027

[Matt J]

Like I said, you have to extract the rows of asabove that you are trying to bin. And the number of rows in this section has to be an even multiple of 'bin':

load asabove

bin = 180*60;
start_time = 172827;
final_time = 432027;

asaboveBinned = sepblockfun(asabove(start_time:final_time+bin-1,:),bin,'mean');

whos asaboveBinned

  Name                Size            Bytes  Class     Attributes

  asaboveBinned      25x54            10800  double              

Note that your original code appears to have a mistake if 'bin' is supposed to be the number of elements that you are binning. In particular, this line,

abovetest = asabove(i:(i+bin),j);

will extract a sections of length bin+1, rather than length bin. And, the sections will overlap with each other, because in your loop, i is jumping in increments of bin, rather than in increments of bin+1.

[Miguel]

Thank you so much! I am new to matlab and this week has been hard... so brain power has been running at ~5% or so and quite foggy.

I'm having trouble interpreting what the resulting table's data means, however. Would these then represent the means of the values at each hour? And if so, would I then average together the hour/population that I am interested in from asaboveBinned, in order to then get the values I am looking for?

How would you recommend I change that error? My end goal was to scan asabove, from my start time in bin increments (which is equal to amount of frames in 1 hour) to my final time. When I make it (i:bin,j) I instead get no data.

[Matt J]

I'm having trouble interpreting what the resulting table's data means, however

It is dividing the original data into bin x 1 tiles and finding the average within each tile. Below is a simplified illustration with bin=5.

>> sepblockfun(asabove, 5,'mean')

ans =

3.4000 3.2000 4.6000 4.8000 4.2000

4.6000 5.8000 3.6000 3.4000 6.4000

2.4000 3.2000 2.8000 5.4000 6.0000

How would you recommend I change that error?

Well, obviously I would recommend you abandon the loop altogether in favor of sepblockfun. However, I believe what you really wanted in the original loop was,

abovetest = asabove(i:(i+bin-1),j);