Converting downloaded binary data to decimal

Hey folks,

I'm using a web service that is responding to queries with binary data and I'm really struggling to convert this to a more usable format. Fortunately, the service does provide an example data case and they explain the formats and definitions. The problem is that I'm not familiar with converting data types like this and haven't found the right recipe.

I'm attaching a .mat file with three examples in it. Each example has what the web service query returned (raw), what the web service documentation says should be returned (not sure what this format is called, but I called it hex... maybe it's unicode???), and what the web service documentation says the data represents (decimal values).

Example 1: The Matlab web query returns "A33" (without quotations). The documentation says this should show up as "47 97 33 33". Is this unicode, hex, other??? I call it "hex" in my file format. According to documentation, this represents a float number of length (8-bit bytes) of 4. In this case, the represented number is "18.90". Furthermore, they define "Float" as a single-precision floating point 32-bit IEEE 754 size/format.

I'm struggling to get the raw "A33" to look like the documentation, but I can get close if I use:

dec2hex(unicode2native('A33','US-ASCII')) % returns "41 1A 33 33" (1A element SHOULD be 97)

Example 2: raw returned data is "G" (without quotes). Documentation says this should be "47 0F C6 14" that represents 3,680,608. Here's what I get:

dec2hex(unicode2native('G','US-ASCII')) % returns "47 0F 1A 14" (1A element SHOULD be C6)

Example 3: raw returned data is " jh" (without quotes). Documentation says this should be "00 00 01 16 6A E0 68 80" that represents a timestamp of November 23, 2007 (I assume it's a number of days measured from Jan 1, 1970, or perhaps from year 0. What I get:

dec2hex(unicode2native('  jh','US-ASCII')) % returns "20 20 01 16 6A 1A 68 1A" (four of these elements are wrong)

So here are my questions on the above three examples:

1) Do you have input on how I'm trying to convert the raw data to usable numbers? I'm totally unfamiliar with this area of computing.

2) Once I do get the raw data to look like what the documentation specifies (e.g., "47 97 33 33"), do you know how I convert it to decimal (e.g., 18.90)?

3) Is it possible that the web request itself is not correct and the data I'm getting back does not match what is expected? In this case, could it be something where I need to change the web options in the web request?

Any input on this matter is much appreciated!!!

Thanks,

Paul

Answer 1

[Walter Roberson]

>> examples(2).dec
ans =
     3680608

>> bytes = uint8(cellfun(@hex2dec, examples(2).hex)); typecast(fliplr(bytes),'single')
ans =
  single
        36806.08

This is exactly 1/100 of value listed as expected.

>> typecast(fliplr(uint8(examples(2).raw+0)),'single')
ans =
  single
        36863.08
>> sprintf('%02x', uint8(examples(2).raw+0))
ans =
    '470fff14'
>> examples(2).hex
ans =
  1×4 cell array
    '47'    '0F'    'C6'    '14'

The .hex that was given does not agree with the bytes stored in the .raw .

>> examples(1).dec
ans =
                      18.9
>> typecast(fliplr(uint8(cellfun(@hex2dec, examples(1).hex))),'single')
ans =
  single
            18.9

>> sprintf('%02x', uint8(examples(1).raw+0))
ans =
    '41ff3333'

The hex that was given does not agree with the bytes stored in the .raw

>> examples(3).hex
ans =
  1×8 cell array
    '00'    '00'    '01'    '16'    '6A'    'E0'    '68'    '80'
>> sprintf('%02x', uint8(examples(3).raw+0))
ans =
    '000001166aff68ff'

The hex that was given does not agree with the bytes stored in the .raw .

In each of the three cases of disagreement, the raw has an FF instead of the expected byte.

I have not been able to figure out the structure of the third item.

[Walter Roberson]

Got it:

bytes = uint8(cellfun(@hex2dec, examples(3).hex));
>> datetime(double(typecast(fliplr(bytes),'uint64'))/1000, 'convertfrom','posixtime')
ans = 
  datetime
   23-Nov-2007 05:00:00

[Paul Shoemaker]

Walter,

This is HUGELY helpful! I spent so many hours trying to crack this and you've already made far more progress much more quickly.

OK, so here's some additional info...

It's odd that the values in the hex fields don't agree with the raw data. The "raw" data is exactly what I downloaded from the web service using the parameters they provided (I used urlread/webread). The "hex" values are what they provided as documentation on what the raw data should represent. Perhaps there is a problem in the documentation (note that in example 3, the date is Nov 2007, implying the example may be based on 10yr old data, so maybe the downloaded data has since been changed and no longer matches the documentation). With the big help you provided, I will be investigating this by looking at other data I can test easily enough.

As for the third item, I found additional info in the documentation. It should be "Long" format with 8 bytes (8-bit each). Documentation isn't entirely clear, but it could be 64-bit two's complement integer, whatever that means.

Either way, I'm pretty certain the value that example 3 should return in decimal is "13840", which represents the number of days between Jan 1, 1970 (reference date) and November 23, 2007.

I'll crack away at this more and follow up once I get it nailed down. If you have additional ideas I'd really appreciate hearing back.

Thanks again for the huge leap forward in figuring this out!

[Walter Roberson]

As I show in my comment, the timestamp is posix time format, times 1000 -- so it is the number of miliseconds since the base date. Easy enough to convert if you have a version of MATLAB recent enough to have the posixtime conversion.

[Paul Shoemaker]

Walter,

Quick update: I found some fine print that says that example 2 is reported as 1/100th of actual value, so you did it right! The big questions remaining then are why raw and hex don't match and how to handle example 3.

[Walter Roberson]

My top comment shows how to handle example 3.

[Paul Shoemaker]

Apologies for the rapid messages, but you gave me a lot to go on... I think we're close!

I've been using urlread to read this data, which gives the unfriendly "raw" data you saw above. However, I just noticed that if I use webread I get the data back in a more usable format:

Example1: using webread for same query returns raw of...

raw = 
1×4 uint8 row vector
  65   151    51    51

typecast(fliplr(raw),'single') % Equals 18.9 (correct!)

Example 2: using webread for same query returns raw of...

raw =
  1×4 uint8 row vector
    71    15   198    20

100*typecast(fliplr(raw),'single') % Equals 3680608 (correct!)

Example 3: using webread for same query returns raw of...

raw =
  1×8 uint8 row vector
     0     0     1    22   106   224   104   128

However, using the same technique does not yield the expected result of 13840.

Now the only remaining mystery is how to solve the third example.

[Walter Roberson]

>> raw = uint8([0     0     1    22   106   224   104   128])
raw =
  1×8 uint8 row vector
     0     0     1    22   106   224   104   128
>> datetime(typecast(fliplr(raw),'uint64')/1000,'convertfrom','posixtime')
ans = 
  datetime
   23-Nov-2007 05:00:00

If you are getting 22-Nov-2007 or 24-Nov-2007 then you might be in a different time zone. 23-Nov-2007 05:00:00 UTC corresponds to 23-Nov-2007 00:00 EST:

    T = datetime(typecast(fliplr(raw),'uint64')/1000,'convertfrom','posixtime','TimeZone', 'UTC');
   T.TimeZone = 'America/Toronto'

T = 
  datetime
   23-Nov-2007 00:00:00

[Paul Shoemaker]

Minor update on example 3: it is the number of milliseconds measured from 00:00:00 UTC Jan 1, 1970. In this case, that's 13840*24*60*60*1000 = 1,195,776,000,000

[Walter Roberson]

Yes, milliseconds since that time is 1000 times Posix Time (which is seconds since then.) datetime() with 'posixtime' can convert that.

[Paul Shoemaker]

Walter, I didn't see all of your responses as I typed my messages above (wasn't doing page refreshes), but thanks to your huge help it's solved.

Thank you so much again!

Answer 2

[Rohit chand]

Your data logger shows the decimal numbers 47 and 21875.

[Walter Roberson]

Could you explain how you derive that information?