Looking to import manually created (existing) AWS resources into your SAM or CDK project (with CloudFormation) ? This is the video for you.
Often many developers start learning AWS by manually creating resources in the console. One thing leads to another and we end up launching full blown services by manually creating resources.
Fast forward 6 months and we learn about CloudFormation, a powerful Infrastructure as code toolkit that allows you to define your AWS using template files (or code if using CDK).
It turns out, using tools like SAM or CDK are the preferred ways of defining resources in AWS for any serious developer. It isn’t long when we try to switch over to using the ‘right’ approach.
Now we run into a problem – we now need to integrate those existing AWS resources into our project so we can start managing them using code. It turns out there’s no ‘suggested’ way to do this via AWS documentation, but there are some tricks you can follow to integrate these resources into your CloudFormation stack, and subsequently your SAM or CDK project.
In this article, I’ll show you an example of how to migrate a DynamoDB table into a SAM based project.
Now lets figure out how its done.
Our starting state is a CloudFormation stack called
cf-import-demo. Currently, the only AWS resource that it manages is an SNS topic.
This stack was setup using the SAM toolkit and running the
sam init command and selecting the Hello World option. All boilerplate has been stripped out leaving us with a bare bones
AWSTemplateFormatVersion: '2010-09-09' Description: > cf-import-demo Sample SAM Template for cf-import-demo Resources: DemoSNSTopic: Type: AWS::SNS::Topic Properties: TopicName: DemoSNSTopic
Now here’s our DynamoDB table called
TableToImport that was created manually in the console. This is our target for importing into CloudFormation.
Note that I’ve made almost zero modifications to this table – the only configuration to note is the ProvisionedReadCapacity and ProvisionedWriteCapacity which are currently set to 1. Remember this for later on.
Our end state will be our CF stack having control over our DynamoDB Table.
So lets get into the how.
Step 1 – Modify Our
Our first step is to make some modifications to our
template.yaml file in our SAM project file.
What we need to do is effectively create a CloudFormation resource that mimics the configuration we have specific on our manually created DynamoDB table.
We need to define an exact match (or as close as possible match) to this entry. This means manually looking through the configuration on your table, and looking up CloudFormation documentation to add the corresponding settings.
For example, in my manually created
TableToImport table, the options that I’ve configured are a) Partition Key on the table, and b) read and write capacity. This means in my corresponding entry into my
template.yaml file, I’ll have to specify my Partition key and capacity values.
Here’s what the corresponding DynamoDB resource looks like:
TableToImport: Type: AWS::DynamoDB::Table DeletionPolicy: Retain Properties: TableName: TableToImport AttributeDefinitions: - AttributeName: "demo" AttributeType: "S" KeySchema: - AttributeName: "demo" KeyType: "HASH" ProvisionedThroughput: ReadCapacityUnits: "1" WriteCapacityUnits: "1"
We’ll append the resource to our
template.yaml file alongside our SNS topic.
Now we have a modified CloudFormation file that contains our soon to be imported resource. We now need to link the resource with our CloudFormaiton stack. To do this, we go to AWS console.
template.yaml file looks like this:
AWSTemplateFormatVersion: '2010-09-09' Description: > cf-import-demo Sample SAM Template for cf-import-demo Resources: DemoSNSTopic: Type: AWS::SNS::Topic Properties: TopicName: DemoSNSTopic TableToImport: Type: AWS::DynamoDB::Table DeletionPolicy: Retain Properties: TableName: TableToImport AttributeDefinitions: - AttributeName: "demo" AttributeType: "S" KeySchema: - AttributeName: "demo" KeyType: "HASH" ProvisionedThroughput: ReadCapacityUnits: "1" WriteCapacityUnits: "1"
Note that if using SAM + Serverless, you’ll need to remove the
Serverless::Transform resource from the header of your generated
template.yaml file. Not doing so will cause the next step to fail at the upload step.
Another important note is the DeletionPolicy: Retain component of our DynamoDB Table resource. THIS IS A REQUIRED ATTRIBUTE THAT YOU MUST SET. Again, failure to set this option on all resources that you are trying to port over will result in an error at the next step.
Step 2 – Link Our Resource to CloudFormation
We need to head over into the CloudFormation section of the AWS console and click on our Stack. From here, we select the Stack Actions menu and click Import resources into stack.
Clicking it will launch a Wizard that guides you through the import process. What we need to do in this step is upload our newly created
template.yaml file into the wizard. This will give CloudFormation the ability perform a diff between what it thinks your stack is composed of, and what resources you’ve attempted to add.
It will automatically recognize the diff and import the existing resource into your current stack.
To keep moving forward, go the second step of the wizard and upload the updated file and click next.
Next, give your imported resource a logical ID to refer to late. I just name this the same name as my resource, in this case TableToImport.
You can effectively skip to the overview part of the wizard (unless you want to specify any additional parameters). CloudFormation will perform a diff of your template file and what it has on record and let you preview the resources it is about to import.
Before you proceed with the next step, PLEASE make sure to test out this process on your beta/test/development environment. Although this process is straightforward you want to be sure it is working correctly in safe environment before performing these steps in production.
If you’re satisfied, you can go ahead and click on Import Resources to perform the link.
If all went well, you should be forwarded to the Events section of the stack, and should see an update stream and hopefully IMPORT COMPLETE as the final update.
We can go to the Resources tab and should see our newly linked DynamoDB table!
Before we claim success, lets go back to our editor and make some updates to our imported resource. If all worked correctly, we should see corresponding updates to our cloudformation stack and DynamoDB table.
Step 3 – Test It Out
To see if the our setup is working, I’m going to modify a trivial value as part of the ProvisionedThroughput option. Originally, we set this value to 1. I’m going to change it to 2 and see if the update occurs.
ProvisionedThroughput: ReadCapacityUnits: "2" WriteCapacityUnits: "2"
After making the change and saving our work, we run
sam build and
sam deploy to deploy our changes. We’re greeted with a confirmation step asking us to make sure we want to deploy the change.
Notice that the prompt is showing that we are modifying our DynamoDB table.