In our previous article, we dug deep into how AWS CloudFormation works and provided an analysis of a VPC template we created. Our next template example is that of SFTP Gateway, a product that makes it easy to transfer files via SFTP to Amazon S3. We’ll incorporate S3, EC2, IAM, Security Groups, and more to facilitate this file transfer.
Here’s the template in its entirety:
AWSTemplateFormatVersion: 2010-09-09
Mappings:
RegionMap:
ap-northeast-1:
AMI: ami-0d1a9e6b
ap-northeast-2:
AMI: ami-0ca50362
ap-south-1:
AMI: ami-989fd6f7
ap-southeast-1:
AMI: ami-db7a25b8
ap-southeast-2:
AMI: ami-8c14e0ee
ca-central-1:
AMI: ami-3d13a859
eu-central-1:
AMI: ami-fd6fe192
eu-west-1:
AMI: ami-e8922c91
eu-west-2:
AMI: ami-09223c6d
eu-west-3:
AMI: ami-9563d4e8
sa-east-1:
AMI: ami-eb4f0887
us-east-1:
AMI: ami-c599febf
us-east-2:
AMI: ami-6a1c350f
us-west-1:
AMI: ami-99b983f9
us-west-2:
AMI: ami-a7ca10df
Resources:
SFTPGatewayInstance:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
config:
commands:
setup:
command: /usr/local/bin/sftpgatewaysetup
Properties:
IamInstanceProfile: !Ref RootInstanceProfile
ImageId: !FindInMap
- RegionMap
- !Ref AWS::Region
- AMI
InstanceType: !Ref EC2Type
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize: !Ref DiskVolumeSize
VolumeType: gp2
KeyName: !Ref KeyPair
SecurityGroupIds:
- !Ref SFTPGatewaySG
SubnetId: !Ref SubnetID
Tags:
- Key: Name
Value: SFTPGateway Instance
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance \n --region ${AWS::Region}
SFTPGatewayBucket:
DeletionPolicy: Retain
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub sftpgateway-${SFTPGatewayInstance}
DependsOn:
- SFTPGatewayInstance
RootInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref S3WritableRole
S3WritableRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
Path: /
RolePolicies:
Type: AWS::IAM::Policy
DependsOn:
- SFTPGatewayInstance
Properties:
PolicyName: SFTPGatewayInstancePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 's3:*'
Resource: '*'
Roles:
- !Ref S3WritableRole
SFTPGatewaySG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SFTPGateway Security Group
VpcId: !Ref VPCIdName
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
IPAddress:
Properties:
Domain: vpc
InstanceId: !Ref SFTPGatewayInstance
Type: AWS::EC2::EIP
Parameters:
EC2Type:
Description: SFTPGateway Instance Type
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
- t2.small
- t2.medium
- t2.large
- m3.medium
- m3.large
- m3.xlarge
- m4.large
- m4.xlarge
- c3.large
- c3.xlarge
- c4.large
- c4.xlarge
- r3.large
- r3.xlarge
KeyPair:
Description: EC2 KeyPair
Type: AWS::EC2::KeyPair::KeyName
ConstraintDescription: Existing EC2 KeyPair.
DiskVolumeSize:
Default: 32
Description: Disk volume size in GB. Must be at least 32.
ConstraintDescription: Must be a number greater or equal to 32
MinValue: 32
Type: Number
VPCIdName:
Description: Select the VPC to launch the SFTPGateway into
Type: AWS::EC2::VPC::Id
SubnetID:
Description: Subnet ID
Type: AWS::EC2::Subnet::Id
Outputs:
ElasticIP:
Value: !Ref IPAddress
Description: Elastic IP addressYou can download the SFTP Gateway template here.
There’s a lot going on in the template, so we’ll just give a “brief” overview of what’s happening and point out some interesting syntax that you might use for your own projects.
Create an EC2 instance and S3 bucket
SFTP Gateway uses an EC2 server to upload files to S3. So we start off with an EC2 instance and S3 bucket:
Resources:
SFTPGatewayInstance:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
config:
commands:
setup:
command: /usr/local/bin/sftpgatewaysetup
Properties:
IamInstanceProfile: !Ref RootInstanceProfile
ImageId: !FindInMap
- RegionMap
- !Ref AWS::Region
- AMI
InstanceType: !Ref EC2Type
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize: !Ref DiskVolumeSize
VolumeType: gp2
KeyName: !Ref KeyPair
SecurityGroupIds:
- !Ref SFTPGatewaySG
SubnetId: !Ref SubnetID
Tags:
- Key: Name
Value: SFTPGateway Instance
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region}
SFTPGatewayBucket:
DeletionPolicy: Retain
Type: AWS::S3::Bucket
Properties:
BucketName: !Sub sftpgateway-${SFTPGatewayInstance}
DependsOn:
- SFTPGatewayInstanceHere are a few things worth noting:
- The EC2 instance has a Metadata section in addition to its properties. We’ll cover these in more detail below.
- The S3 bucket has a Deletion Policy of “Retain”. This means you keep the S3 bucket if you delete the CloudFormation stack.
- The S3 BucketName uses an intrinsic function called “!Sub”, which lets you do string interpolation. The syntax “${SFTPGatewayInstance}” gives you the EC2 instance ID, just like the “!Ref” function.
This is what we have so far:
Let’s take a closer look at the EC2 instance metadata and properties:
SFTPGatewayInstance:
Type: AWS::EC2::Instance
Metadata:
AWS::CloudFormation::Init:
config:
commands:
setup:
command: /usr/local/bin/sftpgatewaysetup
Properties:
IamInstanceProfile: !Ref RootInstanceProfile
ImageId: !FindInMap
- RegionMap
- !Ref AWS::Region
- AMI
InstanceType: !Ref EC2Type
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
VolumeSize: !Ref DiskVolumeSize
VolumeType: gp2
KeyName: !Ref KeyPair
SecurityGroupIds:
- !Ref SFTPGatewaySG
SubnetId: !Ref SubnetID
Tags:
- Key: Name
Value: SFTPGateway Instance
UserData:
Fn::Base64: !Sub |
#!/bin/bash
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init --stack ${AWS::StackName} --resource SFTPGatewayInstance --region ${AWS::Region}There’s a lot going on here:
“CloudFormation::Init” – This is a powerful tool that lets you define config files and commands. In this case, we run a command called “sftpgatewaysetup” to initialize the software.
“ImageId” – This uses the “!FindInMap” intrinsic function that looks up an AMI ID based on the current region. The AMI mappings are located in the Mappings section of the CloudFormation template.
“InstanceType” – This refers to a parameter that we named “EC2Type” which gives you a drop-down list of common EC2 instance types.
“BlockDeviceMappings” – This sets the disk drive type to solid state (gp2). It also points to a parameter named “DiskVolumeSize” which allows the user to define disk size at stack creation.
“KeyName” – This refers to an SSH key that you use to log into the server.
“UserData” – This lets you run bash commands on server launch. In this case, we use “cfn-init” to read the “CloudFormation::Init” metadata we defined earlier.
A lot of the properties above reference parameters. Below is a snippet of the Parameters section of the template, which includes the “EC2Type”, “DiskVolumeSize”, and “KeyPair” parameters mentioned earlier:
.
.
.
Parameters:
EC2Type:
Description: SFTPGateway Instance Type
Type: String
Default: t2.micro
AllowedValues:
- t2.micro
- t2.small
- t2.medium
- t2.large
- m3.medium
- m3.large
- m3.xlarge
- m4.large
- m4.xlarge
- c3.large
- c3.xlarge
- c4.large
- c4.xlarge
- r3.large
- r3.xlarge
KeyPair:
Description: EC2 KeyPair
Type: AWS::EC2::KeyPair::KeyName
ConstraintDescription: Existing EC2 KeyPair.
DiskVolumeSize:
Default: 32
Description: Disk volume size in GB. Must be at least 32.
ConstraintDescription: Must be a number greater or equal to 32
MinValue: 32
Type: Number
Parameters let you pass dynamic values to make your template more flexible. Here are a few things to note:
- “AllowedValues” – This presents the user with a drop-down, so you don’t have to worry about form validation.
- “AWS::EC2::KeyPair::KeyName” – This is a special type that automatically presents the user with a list of key pairs in their AWS account.
- “MinValue” – This provides form validation that gives an error if the user puts in a value that is too small.
Create an IAM role
In order for our EC2 instance to access S3, we need to grant it permissions using an IAM role.
The architecture looks like this:
The diagram shown above simplifies what’s actually happening. If you look at the CloudFormation template, you’ll see that there’s more to it:
Resources:
.
.
.
RootInstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: /
Roles:
- !Ref S3WritableRole
S3WritableRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: ec2.amazonaws.com
Action: sts:AssumeRole
Path: /
RolePolicies:
Type: AWS::IAM::Policy
DependsOn:
- SFTPGatewayInstance
Properties:
PolicyName: SFTPGatewayInstancePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 's3:*'
Resource: '*'
Roles:
- !Ref S3WritableRoleThere are three resources involved when assigning permissions to an EC2 instance:
- “InstanceProfile” – You can’t assign an IAM role directly to the EC2 instance, but you can assign an instance profile, which passes role information to the EC2 instance.
- “IAM::Role” – The EC2 instance can assume a role and inherit any permissions from the role, via the instance profile.
- “IAM::Policy” – This contains the actual permissions. The policy is associated with the role.
Using an existing public subnet
The EC2 instance needs to be in a public subnet so that end users can access it via SFTP. This CloudFormation template doesn’t create this public subnet. Rather, you select an existing subnet and pass it as a parameter to the template.
This happens here:.
.
.
.
Parameters:
.
.
.
VPCIdName:
Description: Select the VPC to launch the SFTPGateway into
Type: AWS::EC2::VPC::Id
SubnetID:
Description: Subnet ID
Type: AWS::EC2::Subnet::IdHere’s what’s going on:
- “AWS::EC2::Subnet::Id” – This is a special parameter type that lists existing subnets in your AWS account. The EC2 instance is provisioned in this subnet.
- “AWS::EC2::VPC::Id” – This is another special parameter type and it lists existing VPCs. In the next section, we will define a security group that gets provisioned in this VPC.
The architecture looks like this:
Incorporate Security Group settings
In order for SFTP users to access the server, we use a Security Group to expose port 22 for specific IP addresses.
Here’s the relevant code:
Resources:
.
.
.
SFTPGatewaySG:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: SFTPGateway Security Group
VpcId: !Ref VPCIdName
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0The security group has a property called “SecurityGroupIngress”, which accepts an array of rules. Here we have a single rule that allows all traffic (0.0.0.0/0) on TCP port 22.
Adding an elastic IP
Finally, we need to create a static IP so that our public IP address doesn’t change each time the server shuts down.
Resources:
.
.
.
IPAddress:
Properties:
Domain: vpc
InstanceId: !Ref SFTPGatewayInstance
Type: AWS::EC2::EIPHere’s a brief explanation:
- “Domain” – This is set to “vpc”, since we’re using VPC instead of classic networking.
- “InstanceId” – This is the instance ID of the EC2 server that receives the IP address
We wind up with this final wonderful architecture:
Outputs section
The Outputs section lets you display concise information for easy access. Here, we show the public IP address to make it easier to connect for the first time.
Outputs:
ElasticIP:
Value: !Ref IPAddress
Description: Elastic IP addressWell-Architected
The product’s CloudFormation template enables high availability (HA) by automatically deploying the necessary resources in one cohesive stack, making it easy to see all the resources involved and remove these from your account if necessary.
The core HA resources include:
- EC2 instances that are provisioned in an Autoscaling Group that spans two Availability Zones.
- A Network Load Balancer that provides a single endpoint and routes SFTP traffic to all of the EC2 instances.
- A common Elastic File System that is configured and mounted to the servers to ensure data is not lost if a server fails.
The HA CloudFormation template is freely available and can be customized for your needs.
Conclusion
Now you can easily and securely upload your files to Amazon S3 via SFTP!
You’ve also learned how to incorporate EC2, IAM, S3, Security Groups, and more to facilitate this file transfer.
We hope this has been helpful!