How do I read selective data from a fixed file

I have a gigantic file (aprox 27 gig) of weather data. The file has a few lines in

10i8 format ( 10 integers per line with each integer using 8 characters) and the rest of the lines are of 10f8.2 format (10 floating points per line with 8 characters before and 2 after the decimal points).

I know the precise composition of the file and am aware of the position of the data that I need to use. How do I do this?

The data consists of aproximately 40 million lines. Each line doesn't have more than 10 values per line (as mentioned above).

1 st line: 100 100 30 500 500

31k lines of data

Date hour

254 k lines of data

date next-hour

and the format repeats for the date-hour to next date-hour part. I need to pull out data lines from 104 k till 134k out of the entire 254k lines set for each date-hour.

This data is alligned in the format of a matrix of 100(i) x 100(j) for a loop:

for {j=1,<=100,++1

{i=1,<=100,++1}

}

Here is a preview of the data I have:

100 100 30 500 500

2532.53 2554.49 2576.44 2575.69 2574.94 2574.18 2560.82 2547.46 2534.09 2513.32

2492.54 2471.76 2471.50 2471.23 2470.97 2483.21 2495.45 2507.69 2503.25 2498.81

2494.37 2498.43 2502.48 2506.53 2510.69 2514.84 2519.00 2529.82 2540.63 2551.45

20031020 1

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

255.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00

255.00 255.00 255.00 254.00 254.00 254.00 254.00 254.00 254.00 254.00

254.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00 255.00

-0.03 -0.05 -0.13 -0.13 -0.06 0.01 0.07 0.11 0.11 0.08

-0.04 -0.18 -0.15 -0.03 0.02 0.04 0.09 0.14 0.09 -0.07

-0.14 -0.03 0.16 0.31 0.31 -0.01 -0.56 -0.94 -0.98 -0.78

20031020 2

231.39 238.82 260.27 257.82 251.88 245.91 237.37 225.47 226.96 217.60

198.81 188.80 187.51 188.71 189.24 195.29 200.77 195.77 180.46 170.13

171.69 170.24 170.01 170.10 169.09 141.01 147.26 149.64 149.26 169.63

Sorry for the long question. But I have going at it for a while now and cant really get it to work properly. The file opens with fopen as its an ASCII file.

Thank heaps in advance !

Answer 1

[David Sanchez]

fid = fopen('your_file.txt');
A = fscanf(fid, '%g %g %g %g %g %g %g %g %g %g', [10 inf]);
fclose(fid);

% Transpose so that A matches
% the orientation of the file
A = A';

[Monish]

Thanks David,

But wouldn;t it just turn rows into columns. It still wont enable me to pull out the data from the file !

[David Sanchez]

With the given example, you will end up with a matrix A containing your data in the same form as in the text file. What do you want to do with the data?

[Monish]

I need to pull out data from lines 104003 to 105003 out of the entire 254k lines set for each date-hour. and lay it over a 100 x 100 grid for seperate hours.

[David Sanchez]

% to extract the desired lines:

A_new = A(104003:105003,:);