How to get daily temperatures of past 30 years for a given location from NOAA online database

Hi all and Merry Christmas.

I want to download average daily temperatures of the past 30 years for a given location from NOAA database. I am given a base URL as below:

https://www.ncei.noaa.gov/cdo-web/api/v2/{endpoint}

and also I received a token from NOAA website that I need to use that to access the data. According to the NOAA instructions, assigned token is required to use these queries and must be in the header. Can anyone tell me how to set the token in the header? This is the first time I want to do that.(please visit the below link for further infirmation)

https://www.ncdc.noaa.gov/cdo-web/webservices/v2

[Image Analyst]

FYI: Your first URL brings up an error screen.

Good question though. I think a lot of people would want to know this to do some fun queries on their local weather. I might tackle it later if I get time.

[Behrooz Daneshian]

Thank you in advance for your time.

[the cyclist]

This link is a working starting point for using NOAA's API.

[Behrooz Daneshian]

Exactly, and the endpoints are to fetch specific data we want.

[the cyclist]

This problem was already solved, but I wanted to add some info I stumbled across later. I learned about a site/utility that will convert curl commands to MATLAB (and many other languages).

For clarity, I am talking about cURL, the library and unix command line tool for transferring data over a network -- not the mathematical curl operator.

Answer 1

[the cyclist]

Here is how you can do this using webread:

myToken = 'PUT_YOUR_TOKEN_BETWEEN_THESE_QUOTES';
myURL = 'https://www.ncei.noaa.gov/cdo-web/api/v2/locations/FIPS:37';

opt = weboptions('KeyName','token','KeyValue',myToken);
data = webread(myURL,opt);

[Behrooz Daneshian]

Thank you for your time and I really appreciate it. It seems that your code is working. However, my problem is not solved yet. I get csv file of daily tempereture for specific station( stationid=GHCND:USW00013872) manually by submitting my resuest via NOAA website. However, when I want to do it via Matlab, I would face an error again. The below code is completely consistent with my submited request.(please see the attached jpg file)

myToken = 'Mytoken';
myURL = 'https://www.ncei.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&stationid=GHCND:USW00013872&datatypeid=TMIN,TMAX&startdate=1991-01-01&enddate=2022-10-01&format=csv';
opt = weboptions('KeyName','token','KeyValue',myToken);
data = webread(myURL,opt); 

[the cyclist]

When I try to use this URL outside of MATLAB, I get a developer message from NOAA: "The date range must be less than 1 year".

When I change your URL to span less than a year it works -- both outside MATLAB and from within MATLAB. For example

myToken = 'Mytoken';
myURL = 'https://www.ncei.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&stationid=GHCND:USW00013872&datatypeid=TMIN,TMAX&startdate=2022-01-01&enddate=2022-10-01&format=csv';
opt = weboptions('KeyName','token','KeyValue',myToken);
data = webread(myURL,opt);

works for me. (I changed the start of the date range to 2022.)

[Behrooz Daneshian]

Sorry, I got confused .what do you mean outside the Matlab? Do you mean manually from the website? The attahced is a csv file I got for 30 years of tempreture data, something that I am looking for in MATLAB.(Please see the attached file).

Indeed, my goal is that, by just having the station id, I can access 30 years temperature data and I do not want to download the data for each station manually, which is obviously time-consuming.

[the cyclist]

By "outside MATLAB", I meant by using the command

curl -H 'token:MyToken' 'https://www.ncei.noaa.gov/cdo-web/api/v2/data?datasetid=GHCND&stationid=GHCND:USW00013872&datatypeid=TMIN,TMAX&startdate=2022-01-01&enddate=2022-10-01&format=csv'

from the command line.

I haven't tried to use the web site directly.

[Behrooz Daneshian]

Thank you, but even for one year span, it does not give a correct value. It should present tempretures for 365 days of the year.However, if you look at the created data, it has very few elements. Something is wrong here.

[the cyclist]

How did you generate the CSV file you posted here? Did you use the API, but not via MATLAB? If you can be as specific as possible on the exact steps you used, it might be easier to figure out how to replicate those steps via MATLAB.

I have not really looked at the output of any of the commands that I put through MATLAB. I just assumed it gave sensible output.

[Behrooz Daneshian]

I did not use API to get CSV file. I used the NOAA website's searching tool(https://www.ncdc.noaa.gov/cdo-web/search). I filled out the inqury (please see the atached jpg file) and finally the CSV file containg data for the selected range(lets say 1991 to 2020) was emailed to me. This is not a reseanobale approach for me, since I must extract data of as many stations as possible and keep doing this is really time consuming. Hence, I think if I would able to code in a way that I can use API, I do not need to download CSV file every time for each weather station. I mean just by entering a station id as an input, accessing data would be piece of cake. Please let me know if you need more explanation on this.

[the cyclist]

OK. This isn't really a MATLAB problem at this point. We've solved the MATLAB part, which is how to get MATLAB to pull data from the NOAA API.

I think you need to now figure out the correct URL to pull the data you want, using the API rules. It is possible that the API is not configured to do exactly what you want. You might, for instance, need to use a for loop to issue multiple API requests, to get the full dataset you want.

[Behrooz Daneshian]

Thank you so much. I found a toolbox to get data from NOAA database.However, I still have a problem. can you run the the last code presented in the link below ?

https://www.mathworks.com/matlabcentral/fileexchange/119213-matlab-interface-for-noaa-data

[the cyclist]

That toolbox is definitely helpful!

I was able to run all the examples from that page, including the last one.

[Behrooz Daneshian]

Could you please guide me to run this code? I do not know why I face error. The error is:

No public property 'Token' for class 'noaa'.

Error in datasets (line 38)

HttpHeader = matlab.net.http.HeaderField("token",c.Token,"Content-Type","application/json; charset=UTF-8");

Error in Example (line 2)

d = datasets(n);

n = noaa("NpMoDLCgEVBMZTdPFQHQUOwllVQNHYjz");
d = datasets(n);

[the cyclist]

I was able to run your code without error, and retrieve the data.

Did you download the code from github, and put it somewhere in your path? What do you get if you enter

which noaa

in the command line?

[Behrooz Daneshian]

I got this from the command line:

C:\Users\behroozdaneshian\Downloads\NOAA\noaa.m % noaa constructor

Yes I download the code from github and my code(the code that I sent here) was within the script in that folder.

[Behrooz Daneshian]

I can run it .Thank you very much for taking time on my questions. I learned a lot from you.

[the cyclist]

I'm glad it worked out for you

Answer 2

[the cyclist]

Here is an example that worked for me (using my own token, of course).

It uses MATLAB's system function to call the system's curl function.

myToken = 'PUT_YOUR_TOKEN_BETWEEN_THESE_QUOTES';
myURL = 'https://www.ncei.noaa.gov/cdo-web/api/v2/locations/FIPS:37';
cmdString = ['curl -H ''token:',myToken,''' ''',myURL,''''];
[status, curlOutput] = system(cmdString)

The trickiest part is actually constructing the string to send out to the system, because the string itself has quotes in it. There is probably a more elegant way, or even a MATLAB function that will do this more directly, but I know this solution works.

[Behrooz Daneshian]

I did this but I did not get any data defined in the URL.

[the cyclist]

There are many small things that could go wrong here, and it might not be easy to debug it. But we can try.

I am on a Mac. If you are on a PC, you might not natively have the curl command.

Can you run the line

curl -H 'token:PUT_YOUR_TOKEN_HERE' 'https://www.ncei.noaa.gov/cdo-web/api/v2/locations/FIPS:37'

in a command window? What happens?

[Behrooz Daneshian]

I got this error on my PC:

''Error using curl

Volume data arguments must all be the same size.''

[Image Analyst]

Maybe try

commandString = sprintf("curl -H 'token:PUT_YOUR_TOKEN_HERE' 'https://www.ncei.noaa.gov/cdo-web/api/v2/locations/FIPS:37'")
status = system(commandString)

Though I've never heard of curl. Maybe you can try webread or websave instead. Or try FTP.

[Behrooz Daneshian]

I tried that and again I did not get any data.

[the cyclist]

@Image Analyst, "curl" is unix utility for pulling URL info. (Some web searching suggests that it is, in fact, shipped with Windows. But, I can't really help there.) Given the error that @Behrooz Daneshian reported directly from the command line, I'm guessing we're not going to get that to work via MATLAB.

I agree that webread or websave is promising. Probably one just needs to figure out how to send the token along with the URL. I tried a couple simple Name-Value input combinations, but did not get it to work (yet).

[the cyclist]

Just after I posted the above comment, I figured out (well, found out) how to do this. See my other answer.