getting full HTTP response to HTTP request in urlread

Currently urlread uses the command: urlConnection.getInputStream

and then proceeds to do something fairly confusing to me with copying streams via some unsupported stream copier to produce the final output

Unfortunately when there is an error in my request I get nebulous errors instead of the actual error response. Even when I get a successful request, I am still unable to see the headers which can contain important information for what I am doing.

Example error: java.io.IOException: Server returned HTTP response code: 400 for URL:

By using Matlab's proxy settings I can observe the details of the requests through "Fiddler", which has been most helpful, but it would be nice to be able to observe things more concretely in Matlab.

Answer 1

[Jim Hokanson]

Here's my answer to this problem, although it might have some bugs ...

getHeaderField was needed, note getHeaderField(0) is the response like 400 or 401

In addition, I added getErrorStream, which the server I am working with sends when there is an error

I still don't like the stream copier ...

allHeaders = struct('Response',char(urlConnection.getHeaderField(0)));
done = false;
headerIndex = 0;
while ~done
    headerIndex = headerIndex + 1;
    headerValue = char(urlConnection.getHeaderField(headerIndex));
    if ~isempty(headerValue)
        headerName = char(urlConnection.getHeaderFieldKey(headerIndex));
        headerName(headerName == '-') = '_';
        allHeaders(1).(headerName) = headerValue;
    else
        done = true;
    end
end
status = struct('value',urlConnection.getResponseCode(),...
    'msg',char(urlConnection.getResponseMessage));

isGood = true;
try
    inputStream = urlConnection.getInputStream;
catch ME
    try
        inputStream = urlConnection.getErrorStream;
    catch ME %#ok<*NASGU>
        isGood = false;
    end
end

if isGood
    byteArrayOutputStream = java.io.ByteArrayOutputStream;
    % This StreamCopier is unsupported and may change at any time. OH GREAT :/
    isc = InterruptibleStreamCopier.getInterruptibleStreamCopier;
    isc.copyStream(inputStream,byteArrayOutputStream);
    inputStream.close;
    byteArrayOutputStream.close;
    output = native2unicode(typecast(byteArrayOutputStream.toByteArray','uint8'),'UTF-8');
else
    output = '';
end

[Jim Hokanson]

This code appears to work fine with the exception of cookies since the Set-Cookie header can be repeated (maybe others?). Maybe I'll eventually get around to releasing a urlread v2.0, with much more support for both request and response headers ...

[Ray Lee]

so, have u released your urlread v2?

[Jim Hokanson]

Yes. It is available on the FEX exchange. http://www.mathworks.com/matlabcentral/fileexchange/35693-urlread2

For an outline of the code see: http://undocumentedmatlab.com/blog/expanding-urlreads-capabilities/

I'm also currently working on a v3 (not sure of the name yet). It will be class based and provide a much more intuitive interface for constructing input parameters and parsing output parameters.

In particular the outline will be: 1) Create request object 2) Modify request properties via methods 3) Make request, get response object