Problems of loading http urls in matlab.

I can access websites using the matlab web browser.

I can NOT download a url link using 'urlread'.

Is there any other matlab function to download the text of a url link?

Another alternative way is to use wget in matlab. I know how to save the url to a local text file and then read it into matlab, but it's inefficient since I have plenty of urls to read. Is there any way I can assign the wget-downloaded text to a variable in matlab straightforwardly?

Answer 1

[Geoff Hayes]

Is an error being thrown when you used urlread, or is it just that the output is an empty string? What is the line of code that you are using to access the web content?

I don't have wget but (on my Mac) I use curl and can save the output from the call to a variable as per the system examples

 % build the command string
 cmd = ['curl ' 'http://www.mathworks.com/matlabcentral/answers'];

 % execute the command
 [status,cmdout] = system(cmd);

 % what is the output?
 cmdout

Try the above and see what happens!

[Ray Lee]

For example, when I run

s=urlread('http://www.google.com','timeout',5)

I got an error of

Error using urlreadwrite (line 85)
Connection timed out.
Error in urlread (line 36)
[s,status] = urlreadwrite(mfilename,catchErrors,url,varargin{:});

If there is no 'timeout' option, it has no response until very long time.

However, the solution of curl you suggested is excellent. The variable 'cmdout' is actually the html text of the url.

[Ray Lee]

but for every page, there is a line of

curl: /usr/local/MATLAB/R2013a/bin/glnxa64/libcurl.so.4: no version information available (required by curl)

at the beginning.

[Geoff Hayes]

I don't see that line when I run the equivalent. (I'm using MATLAB 2014a.)

Interesting that urlread('http://www.google.com','timeout',5) times out, but web('http://www.google.com') works fine (that is what you meant by MATLAB web browser?). Do you need to specify proxy server settings to connect to the internet (due to firewall perhaps)?

[Ray Lee]

Yes, you are all correct.

Now all problems are fixed. Thx a lot.