Data

Set up S3 for Grey Matter Data

This guide goes over the steps to set up an S3 bucket to use to persist Grey Matter Data. These steps assume the AWS CLI is configured for your desired profile and region.

1. Create a new S3 bucket

Choose the name of your desired S3 bucket, and run the following to create it:

export DATA_S3_BUCKET=<data-bucket-name>
aws s3api create-bucket --bucket $DATA_S3_BUCKET
aws s3api put-public-access-block --bucket $DATA_S3_BUCKET --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

You may get a warning that the bucket name is not available, if so choose a new name and rerun.

2. Create an IAM policy

In your browser, open the AWS IAM console and click on Policies under Access Management in the left pane. Click Create Policy.

In the JSON tab, copy the following and paste it in. Replace <data-bucket-name> with the name of the bucket you created in step 1:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DataReadWrite",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:GetBucketAcl",
                "s3:GetBucketPolicy",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<data-bucket-name>/*",
                "arn:aws:s3:::<data-bucket-name>"
            ]
        }
    ]
}

Click Review Policy and name the policy gmdata-s3. Create policy.

This policy will give a user for Grey Matter Data access to the bucket.

3. Create an IAM user for the policy

Now create an IAM user named gm-data:

aws iam create-user --user-name gm-data

To attach the policy created in step 2, run:

aws iam list-policies | grep -A 8 gmdata-s3

The output will look like this:

    "PolicyName": "gmdata-s3",
    "PolicyId": "<some-policy-id>",
    "Arn": "arn:aws:iam::<user-id>:policy/gmdata-s3",
    "Path": "/",
    "DefaultVersionId": "v1",
    "AttachmentCount": 0,
    "PermissionsBoundaryUsageCount": 0,
    "IsAttachable": true,
    "CreateDate": "2020-11-25T14:48:36+00:00",
    "UpdateDate": "2020-11-25T14:48:36+00:00"
},

Copy the value in the Arn field from the output, fill it into <policy-arn>, and run the following:

aws iam attach-user-policy --user-name gm-data --policy-arn <policy-arn>

4. Get user credentials

To get programmatic access credentials for the user created in step 3, run:

aws iam create-access-key --user-name gm-data

Immediately save the credentials for AccessKeyId and SecretAccessKey somewhere secure and accessible during the Grey Matter installation.

You are now ready to install Grey Matter Data to persist to S3!