Golang / Go Crash Course 09 | Connecting our REST API with Amazon (AWS) DynamoDB

In this article we are going to create a new data repository in our Golang application in order to use Amazon DynamoDB as the database.

What is Amazon DynamoDB?

• DynamoDB is a key-value and document NoSQL database.
• DynamoDB is fully managed, multiregion, multimaster, durable database with built-in security, backup and restore, and in-memory caching.
• Lyft, Airbnb, Samsung, Toyota, are currently using DynamoDB to support their mission-critical workloads.
• DynamoDB organises data as tables and each table contains serveral items (or rows) and each item has Keys and Attributes (or columns)

DynamoDB and AWS Permissions Setup

In order to be able to use the DynamoDB from our Golang application, we need to setup the AWS SDK for Golang by following some steps bellow:

• Create a new user in your AWS account from IAM. IAM is where you setup all the permissions include users, groups and policies.
• Generate the Access Keys from IAM for that user
• The AWS SDK for Golang will load those credentials from the shared credentials file ~/.aws/credentials.
• Sample credentials file:

[default]
aws_access_key_id=YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key=YOUR_AWS_SECRET_ACCESS_KEY

• The AWS SDK for Golang will load the region from the shared configuration file ~/.aws/config.
• Sample config file:

[default]
region=us-east-1

• Create a “posts” table in DynamoDB

ARN: arn:aws:dynamodb:us-east-1:[ACCOUNT_NUMBER]:table/posts

• Create a group “rest-api-developers” from IAM
• Attach the “posts-table-policy” to that group:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PostTablePolicy0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem",
                "dynamodb:Scan"
            ],
            "Resource": "arn:aws:dynamodb:us-east-1:[ACCOUNT_NUMBER]:table/posts"
        }
    ]
}

• Assign the group “rest-api-developers” to the new user
• Get the AWS SDK for Golang:

$ go get -u github.com/aws/aws-sdk-go

Create IAM user group

We are going to create “rest-api-developers” IAM user group

Create IAM Policy

We are going to create IAM policy to allow access DynamoDB “posts” table

Attach IAM Policy to IAM User group

Create IAM User

Add IAM User to IAM Group

IAM User is created successfully

Create Access Key for IAM User

Access key is created, then you can download it as CSV file.

Create “posts” DynamoDB table

With default table settings

AWS CLI with Profile

After we have the Access Keys for IAM User, we can config profile for multiple AWS accounts.

• Set access – file and content: ~/.aws/credentials

[default]
aws_access_key_id={{aws_access_key_id}}
aws_secret_access_key={{aws_secret_access_key}}

[{{profile_name}}]
aws_access_key_id={{aws_access_key_id}}
aws_secret_access_key={{aws_secret_access_key}}

• Set profile – file and content: ~/.aws/config

[default]
region={{region}}
output={{output:"json||text"}}

[profile {{profile_name}}]
region={{region}}
output={{output:"json||text"}}

• Run with params –profile

$ aws ec2 describe-instances -- default
$ aws ec2 describe-instances --profile {{profile_name}}

In our case, we are going to use profile name is “tvt_favtuts“, let’s try to use AWS CLI to test profile in some cases. Example IAM user favtuts does not have permissions to list all DynamoDB tables:

$ aws dynamodb list-tables --profile=tvt_favtuts
An error occurred (AccessDeniedException) when calling the ListTables operation: User: arn:aws:iam::475797023758:user/favtuts is not authorized to perform: dynamodb:ListTables on resource: arn:aws:dynamodb:us-east-1:475797023758:table/* because no identity-based policy allows the dynamodb:ListTables action

But IAM user favtuts can query all items of “posts” DynamoDB table (expected policy):

$ aws dynamodb scan --table-name posts --profile=tvt_favtuts
{
    "Items": [],
    "Count": 0,
    "ScannedCount": 0,
    "ConsumedCapacity": null
}

Install AWS SDK for Golang library

$ go get -u github.com/aws/aws-sdk-go

go: added github.com/aws/aws-sdk-go v1.44.235
go: added github.com/jmespath/go-jmespath v0.4.0

Dynamodb Data Repository implementation

We are going to start working on DynamoDB implementation of Post Repository interface.

package repository

import (
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/dynamodb"
	"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
	"github.com/favtuts/golang-mux-api/entity"
)

type dynamoDBRepo struct {
	tableName string
}

func NewDynamoDBRepository() PostRepository {
	return &dynamoDBRepo{
		tableName: "posts",
	}
}

func createDynamoDBClient() *dynamodb.DynamoDB {
	region := "us-east-1"
	profile := "tvt_favtuts"
	// Create AWS Session
	sess := session.Must(session.NewSessionWithOptions(session.Options{
		SharedConfigState: session.SharedConfigEnable,
		Config: aws.Config{Region: aws.String(region),
			CredentialsChainVerboseErrors: aws.Bool(true)},
		Profile: profile,
	}))

	// Return DynamoDB client
	return dynamodb.New(sess)
}

func (repo *dynamoDBRepo) Save(post *entity.Post) (*entity.Post, error) {
	// Get a new DynamoDB client
	dynamoDBClient := createDynamoDBClient()

	// Transforms the post to map[string]*dynamodb.AttributeValue
	attributeValue, err := dynamodbattribute.MarshalMap(post)
	if err != nil {
		return nil, err
	}

	// Create the Item Input
	item := &dynamodb.PutItemInput{
		Item:      attributeValue,
		TableName: aws.String(repo.tableName),
	}

	// Save the Item into DynamoDB
	_, err = dynamoDBClient.PutItem(item)
	if err != nil {
		return nil, err
	}

	return post, err
}

func (repo *dynamoDBRepo) FindAll() ([]entity.Post, error) {
	// Get a new DynamoDB client
	dynamoDBClient := createDynamoDBClient()

	// Build the query input parameters
	params := &dynamodb.ScanInput{
		TableName: aws.String(repo.tableName),
	}

	// Make the DynamoDB Query API call
	result, err := dynamoDBClient.Scan(params)
	if err != nil {
		return nil, err
	}
	var posts []entity.Post = []entity.Post{}
	for _, i := range result.Items {
		post := entity.Post{}

		err = dynamodbattribute.UnmarshalMap(i, &post)

		if err != nil {
			panic(err)
		}
		posts = append(posts, post)
	}
	return posts, nil
}

func (repo *dynamoDBRepo) FindByID(id string) (*entity.Post, error) {
	// Get a new DynamoDB client
	dynamoDBClient := createDynamoDBClient()

	result, err := dynamoDBClient.GetItem(&dynamodb.GetItemInput{
		TableName: aws.String(repo.tableName),
		Key: map[string]*dynamodb.AttributeValue{
			"id": {
				N: aws.String(id),
			},
		},
	})
	if err != nil {
		return nil, err
	}
	post := entity.Post{}
	err = dynamodbattribute.UnmarshalMap(result.Item, &post)
	if err != nil {
		panic(err)
	}
	return &post, nil
}

// Delete: TODO
func (repo *dynamoDBRepo) Delete(post *entity.Post) error {
	return nil
}

Run REST API server and testing

$ export PORT=8000
$ go run *.go
Mux HTTP server running on port :8000

Create a new Post

$ curl -v --location 'http://localhost:8000/posts' \
--header 'Content-Type: application/json' \
--data '{
    "title": "Title 1",
    "text": "Text 1"
}'
{"id":7479757279343595441,"title":"Title 1","text":"Text 1"}

Then we verify the row is existed on DynamoDB table

$ aws dynamodb scan --table-name posts --profile=tvt_favtuts
{
    "Items": [
        {
            "text": {
                "S": "Text 1"
            },
            "id": {
                "N": "7479757279343595441"
            },
            "title": {
                "S": "Title 1"
            }
        }
    ],
    "Count": 1,
    "ScannedCount": 1,
    "ConsumedCapacity": null
}

Download Source Code

$ git clone https://github.com/favtuts/golang-mux-api.git
$ cd golang-mux-api

$ git checkout golang-aws-dynamodb
$ go build