Webread Query Not Working

I am looking to query a forum style page (this forum, actually), but cannot seem to the the queries working as desired.

My goal is to essentially get the most viewed questions about Simulink (as woud be done via this page: https://www.mathworks.com/matlabcentral/answers/?product_base_code%5B%5D=Simulink&sort=views+desc)

However, when I try this via webread, the page I get is just the base Answers page, and it isn't giving me Simulink, nor the sorting.

I am using the following (where indexed will be looped for multiple pages):

url = 'https://www.mathworks.com/matlabcentral/answers/';
data = webread(url,'page', index,'product_base_code%5B%5D','Simulink','sort','views+desc');

Any advice/help would be greatly appreciated.

Thanks!

Answer 1

[Kanishk]

Hi Eric,

I understand you're attempting to access the MATLAB forums page using the ‘webread’ function with specific query parameters. However, directly utilizing the URL for the desired page with ‘webread’ results in the following error.

'''

The connection to URL 'https://www.mathworks.com/matlabcentral/answers/?product_base_code%5B%5D=Simulink&sort=views+desc' timed out after 5.000 seconds. The reason is "Waiting for response header". Perhaps the server is not responding or weboptions.Timeout needs to be set to a higher value.

'''

Increasing the ‘Timeout’ value using ‘weboptions’ as suggested in the error works and fetches the correct page. Here is the code for same.

webopts = weboptions('Timeout', 10); 
data = webread('https://in.mathworks.com/matlabcentral/answers/?page=1&sort=views+desc&term=product_base_code%3ASL', webopts); 

To incorporate a page index into the URL within a for loop, you can utilize the ‘strcat’ function.

for index = 1:5 
    webopts = weboptions('Timeout', 10); 
    url = strcat('https://in.mathworks.com/matlabcentral/answers/?page=',num2str(index),'&sort=views+desc&term=product_base_code%3ASL'); 
    data = webread(url, webopts); 
end 

The above code will fetch 5 pages from MATLAB central. You can use ‘websave’ if you wish to save the fetched pages in HTML files.

Please go through following MATLAB documentation to learn more about ‘weboptions’, ‘strcat’ and ‘websave’:

• https://www.mathworks.com/help/matlab/ref/weboptions.html
• https://www.mathworks.com/help/matlab/ref/strcat.html
• https://www.mathworks.com/help/matlab/ref/websave.html

Hope This helps!

Thank You

Kanishk Singhal