General question, should I use indexing

I have a lot of field data and will be getting more in the future. I'm not great at matlab but I'm trying to get better and improve my code past the terrible (call in each file I want for 50 or more days of data) code I currently have. My question (in example form) is this: I have over 40 days of radiometer data, now I need to read in each day file and then manipulate the data, for example there are 4 columns with "flags" I need to write some code saying that when certain flags=1 then only grab (1:2:end,:) data. Would it be better to collate all of the days of data into one large matrix and try indexing by the date and messing with the data that way, or would it be better to write a for loop which reads in each day and manipulates the data in each day sequentially? Does it matter? I don't really know how to do either of these methods and I'm just trying to decide which method to spend my time on. So any advice would be appreceated.

[Stephen23]

Generally it is easier to process data when it is stored in fewer arrays. More arrays just means more loops.

Perhaps a table might be suitable storage, then you can use inbuilt functions to process the groups of data:

https://www.mathworks.com/help/matlab/matlab_prog/perform-calculations-by-group-in-table.html

[Steven Lord]

Because of this sentence fragment I'd suggest a timetable instead of a table.

"try indexing by the date"

They're very similar, and you can do most if not all of what you can do to a table with a timetable, but timetable also lets you do some time-based processing (like retime and synchronize) that would be trickier with a table.

Answer 1

[Neev]

Hey Brandy

So. whenever dealing with a large amount of field data, you will always have two options:-

1. Collate the data into one large matrix:

- Read each day's data and concatenate them vertically into a single matrix.

- Manipulate the data using indexing and logical operations to filter based on the flags.

- This approach allows for efficient indexing and manipulation of the entire dataset, but it can be memory-intensive.

2. Use a for loop to process each day's data sequentially:

- Read each day's data and process it within a loop.

- Manipulate the data within each iteration, filtering based on the flags.

- This approach is memory-efficient but may be slower compared to the first approach.

So, I don't want to focus your mind on any one out of the two, but you can go through how optimised your work might become if you implement either of the approach from above. Now, it is for you to choose the approach based on factors such as memory constraints, computational efficiency, and code organization. Optimize your code by vectorizing operations and using MATLAB's built-in functions effectively. Start with a smaller subset of data to test and optimize your code before applying it to the entire dataset.

[Brandy]

Hey Neev,

Thanks for the answer and advice. I'm glad I was at least right that these are the two main ways to approach it and I'll have to just see which one works better for my setup. Also, I didn't know about vectorising so thanks and I'll look into that too.

Cheers,

Brandy