MATLAB Answers

0

Script to access Amazon S3 files

Asked by André Bibaud on 25 Sep 2017
Latest activity Commented on by Jhon Wine on 17 Nov 2018
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks

  0 Comments

Sign in to comment.

2 Answers

Answer by Jhon Wine on 24 Jan 2018

I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks

  0 Comments

Sign in to comment.


Answer by David Fink on 28 Sep 2017

Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.

  1 Comment

Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Sign in to comment.