Execute AWS CLI commands within MATLAB using the 'system' command

Hi there.

I am using Matlab R2018b on Windows 7. My AWS CLI has 1.16.129

My goal is to load and save specific files from my local mashine to an S3-Bucket on my AWS account. I already implemented a solution suggested in the documentation https://de.mathworks.com/help/matlab/import_export/work-with-remote-data.html

However, this solutions are about big-data. For me, it is important that the file(s) i load from and save to my S3-Bucket do have a particular location and the specified name. Those aspects can not really be considered when using a datastore to "stream" data to and from my bucket. It works fine for big-data use cases, however it does not for mine. When using tall arrays to write data, the destination folder has to be empty or non-existend. So, back to my problem.

I want to save .mat files with specified names in a particular location on my bucket. The easiest way to do that, is by using the AWS CLI, which provides convenient methods to do so. In my case 'get-object' and 'put-object' commands can be used. My strings look like this:

userxxxxx>aws s3api get-object --bucket myBucket --key myRemoteLocation/myFile.mat myLocalLocation\myFile.ma
userxxxxx>aws s3api put-object --bucket myBucket --key myRemoteLocation/ --body myLocalLocation\myFile.mat

When i execute them within my windows shell, everything is working just fine. I just have to define my https_proxy in advance.

When i try to execute that code within matlab, i get the status 255 with the cmdout: Could not connect to the endpoint URL: "https://myBucket.s3.my_aws_region.amazonaws.com/myRemoteLocation/myFile.mat"

I tried to set the proxy first:

        com.mathworks.mlwidgets.html.HTMLPrefs.setUseProxy(true)
        com.mathworks.mlwidgets.html.HTMLPrefs.setProxyHost('ip')
        com.mathworks.mlwidgets.html.HTMLPrefs.setProxyPort('host')

However, i still get the same status and command out message.

Am i missing anything?

I also tried to set the AWS environment variables, which should not be necessary because the CLI uses externaly configured credentials. It did not help either.

Regards,

David

Answer 1

[David Reiter]

My solution is to use a shell script which is executed with the MATLAB command 'system()'. In that case, the destination bucket, key, body and so on have to be handed as input argument for that .sh file. Within the executing shell (in my case git bash), http and/or https proxies can be defined if necessary. The aws cli has to be installed on your local mashine and you need the aws credentials configured within the .aws file in your user directory. (C:\Users\<username>\.aws)

To change the credentials using the bash, the command 'aws configure' will do the job. I used the exact same commands within my .sh file to successfully load and save specific files to and from S3

userxxxxx>aws s3api get-object --bucket myBucket --key myRemoteLocation/myFile.mat myLocalLocation\myFile.ma
userxxxxx>aws s3api put-object --bucket myBucket --key myRemoteLocation/ --body myLocalLocation\myFile.mat