MATLAB Answers

Script to access Amazon S3 files

89 views (last 30 days)
André Bibaud
André Bibaud on 25 Sep 2017
Commented: Jhon Wine on 17 Nov 2018
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks

  0 Comments

Sign in to comment.

Answers (2)

Jhon Wine
Jhon Wine on 24 Jan 2018
I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks

  0 Comments

Sign in to comment.


David Fink
David Fink on 28 Sep 2017
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.

  1 Comment

Jhon Wine
Jhon Wine on 17 Nov 2018
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Sign in to comment.

Sign in to answer this question.


Translated by