What are cloudformations custom resources and how to use them
What is AWS Cloudformation ?
Cloudformation is a very important tool in the life of people working on AWS cloud platform. It helps you to quickly implement any AWS resources, in a fast and reliable way. It is the best example of what we call IaC :Infrastructure as Code.
Cloudformation makes deployment easier, maintenance and updates for entire environments, but also templatises your infrastrucuture, making it easy to reuse it.
AWS services are continuously updated with a lot of nice new features, and each service has its own lifecycle. We can compare AWS cloud platform to a huge microservices’ infrastructure.
Nonetheless, we might wonder if cloudformation is always up-to-date with all the other services.
This is not the case. Sometimes, a feature or a specific configuration could be missing in the cloudformation code, and you won’t be able to continue this way.
Let’s see how to use cloudformation and create every AWS ressources with any configuration we want !
Custom resources to the rescue
Luckily, AWS cloudformation custom resources are here to help.
Custom resources allow you to programmatically provision AWS resources anytime a cloudformation stack has been created, updated or deleted.
So, how does it work?
In a very simple way, on a change, cloudformation will call a lambda function with a specific event for trigger, and will wait a callback from this function, to define if the resource has been successfully modified or not.
Here we can notice that we can use our favorite SDK to provision AWS resources thanks to the use of lambdas (Python, Java, Node.js, …)
As explained above, the cloudformation custom resource will call a lambda, and will use the event input topass essential information and parameters to your function. You can find below a sample of what the input will look like:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
{ "RequestType" : "Create", "ResponseURL" : "http://pre-signed-S3-url-for-response", "StackId" : "arn:aws:cloudformation:us-west-2:123456789012:stack/stack-name/guid", "RequestId" : "unique id for this create request", "ResourceType" : "Custom::TestResource", "LogicalResourceId" : "MyTestResource", "ResourceProperties" : { "Name" : "Value", "List" : [ "1", "2", "3" ] } } |
NB : The ResourceProperties input is where you can customize the parameters of your function
You can notice that a response url is also given by cloudformation. This URL corresponds to the endpoint to call back when your function is done, and it is waiting to know the status of your actions. Please find below a sample of what the response url is waiting for:
|
1 2 3 4 5 6 7 8 9 10 11 |
{ "Status" : "SUCCESS", "PhysicalResourceId" : "TestResource1", "StackId" : "arn:aws:cloudformation:us-west-2:123456789012:stack/stack-name/guid", "RequestId" : "unique id for this create request", "LogicalResourceId" : "MyTestResource", "Data" : { "OutputName1" : "Value1", "OutputName2" : "Value2", } } |
Custom resources: use case & implementation
As I often do when I work for clients, I first build all the solutions “by hand”, using the console, allowing me to be fast for the validation of the Proof Of Concept with them. After this first step, we can start to build the IaC using cloudformation, allowing us to have a template that will be used for all the working environments, up to the production.
By working on an AWS AppStream project, I have been surprised to see that it was impossible to join IAM roles neither to the Image Builder nor to the Fleet using cloudformation native resources.
Checking the documentation, I saw quickly that it was possible to do it using the AWS SDK, so I chose to use Cloudformation Custom resources to build both the Image Builder and the Fleet.
Building the custom resource lambda function
First, I wrote the lambda code that will create and delete the needed resources.
We retrieve the parameters from the event:
|
1 2 |
appstream_image_builder = event['ResourceProperties']['AppstreamImageBuilder'] appstream_fleet = event['ResourceProperties']['AppstreamFleet'] |
Then, we check in the event what is the request type, and we do actions accordingly:
- Create :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
if event['RequestType'] == "Create": LOGGER.info('appstream_image_builder: \n %s', appstream_image_builder) LOGGER.info('Creating ilmage builder') response = client.create_image_builder( Name=appstream_image_builder['Name'], ImageName=appstream_image_builder['ImageName'], InstanceType=appstream_image_builder['InstanceType'], Description=appstream_image_builder['Description'], DisplayName=appstream_image_builder['DisplayName'], VpcConfig={ 'SubnetIds': appstream_image_builder['SubnetIds'], 'SecurityGroupIds': appstream_image_builder['SecurityGroupIds'] }, IamRoleArn=appstream_image_builder['IamRoleArn'], DomainJoinInfo={ 'DirectoryName': appstream_image_builder['DirectoryName'], 'OrganizationalUnitDistinguishedName': appstream_image_builder['OrganizationalUnitDistinguishedName'] }, Tags=appstream_image_builder['Tags'] ) LOGGER.info("Image bulder created") LOGGER.info("Creating fleet") LOGGER.info('appstream_fleet: \n %s', appstream_fleet) LOGGER.info('Creatin fleet') response = client.create_fleet( Name=appstream_fleet['Name'], ImageName=appstream_fleet['ImageName'], InstanceType=appstream_fleet['InstanceType'], FleetType=appstream_fleet['FleetType'], ComputeCapacity={ 'DesiredInstances': int(appstream_fleet['DesiredInstances']) }, VpcConfig={ 'SubnetIds': appstream_fleet['SubnetIds'], 'SecurityGroupIds': appstream_fleet['SecurityGroupIds'] }, Description=appstream_fleet['Description'], DisplayName=appstream_fleet['DisplayName'], DomainJoinInfo={ 'DirectoryName': appstream_fleet['DirectoryName'], 'OrganizationalUnitDistinguishedName': appstream_fleet['OrganizationalUnitDistinguishedName'] }, Tags=appstream_fleet['Tags'], IamRoleArn=appstream_fleet['IamRoleArn'], StreamView=appstream_fleet['StreamView'] ) LOGGER.info("Fleet created") send_response(event, context, "SUCCESS", {"Message": "Resource creation successful!"}) |
- Delete : In this example, the fleet has to be stopped before deleting
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
elif event['RequestType'] == "Delete": LOGGER.info('Deleting Image builder') response = client.delete_image_builder( Name=appstream_image_builder['Name'] ) LOGGER.info('Stoping fleet') response = client.stop_fleet( Name=appstream_fleet['Name'] ) LOGGER.info('Waiting for fleet to be stopped ...') response = client.describe_fleets( Names=[ appstream_fleet['Name'], ] ) state=response['Fleets'][0]['State'] while state != 'STOPPED': time.sleep(5) response = client.describe_fleets( Names=[ "adobeAndNotepadd-Fleet", ] ) state = response['Fleets'][0]['State'] LOGGER.info('Waiting for fleet to be stopped ...') LOGGER.info('Waiting for fleet to be stopped ...') response = client.delete_fleet( Name=appstream_fleet['Name'] ) send_response(event, context, "SUCCESS", {"Message": "Resource deleted successful!"}) |
As explained above in this article, the cloudformation custom resource is waiting for a status from your lambda, which will be sent by an http call on a custom url. There is a code sample that allows you to do so, knowing that the custom URL is given in the lambda event (you can see the calls to this functions in previous examples) :
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
def send_response(event, context, response_status, response_data): '''Send a resource manipulation status response to CloudFormation''' response_body = json.dumps({ "Status": response_status, "Reason": "See the details in CloudWatch Log Stream: " + context.log_stream_name, "PhysicalResourceId": context.log_stream_name, "StackId": event['StackId'], "RequestId": event['RequestId'], "LogicalResourceId": event['LogicalResourceId'], "Data": response_data }) LOGGER.info('ResponseURL: %s', event['ResponseURL']) LOGGER.info('ResponseBody: %s', response_body) opener = build_opener(HTTPHandler) request = Request(event['ResponseURL'], data=response_body) request.add_header('Content-Type', '') request.add_header('Content-Length', len(response_body)) request.get_method = lambda: 'PUT' response = opener.open(request) LOGGER.info("Status code: %s", response.getcode()) LOGGER.info("Status message: %s", response.msg) |
Building the cloudformation template
Then I created the lambda and the custom resource in cloudformation.
You will see that the Image Builder and the Fleet resource is a Custom::IBAndFleetBuilder type, and the ServiceToken field corresponds to the lambda ARN.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
ImageBuilderAndFleetCreationFunction: Type: AWS::Lambda::Function Properties: Code: S3Bucket: "appstream-image-sources" S3Key: "ImageBuilderAndFleetCreationFunction/handler.zip" Handler: "handler.handler" Timeout: 300 Runtime: python2.7 Role: Fn::ImportValue: !Sub "${Env}-LambdaAppstreamCreationRoleArn" ImageBuilderAndFleetCreationCustom: Type: Custom::IBAndFleetBuilder Properties: ServiceToken: !GetAtt ImageBuilderAndFleetCreationFunction.Arn AppstreamImageBuilder: Name: !Sub "${Project}-Image-Builder" ImageName: !Ref ImageBuilderBaseImageName InstanceType: !Ref ImageBuilderInstanceType Description: !Sub "Appstream Image builder for projet ${Project}" DisplayName: !Sub "${Project}-Image-Builder" SubnetIds: !Ref ImageBuilderSubnetlist SecurityGroupIds: !Ref ImageBuilderSecuritygroupslist IamRoleArn: Fn::ImportValue: !Sub "${Env}-AppstreamImageBuilderRoleArn" DirectoryName: !Ref ImageBuilderDirectoryName OrganizationalUnitDistinguishedName: !Ref ImageBuilderOrganizationalUnitDistinguishedName Tags: Env: !Ref Env Project: !Ref Project AppstreamFleet: Name: !Sub "${Project}-Fleet" ImageName: !Ref FleetDefaultImageName InstanceType: !Ref FleetInstanceType FleetType: !Ref FleetType DesiredInstances: !Ref FleetDesiredInstances SubnetIds: !Ref FleetSubnetlist SecurityGroupIds: !Ref FleetSecuritygroupslist Description: !Sub "Appstream Fleet for projet ${Project}" DisplayName: !Sub "${Project}-Fleet" DirectoryName: !Ref FleetDirectoryName OrganizationalUnitDistinguishedName: !Ref FleetOrganizationalUnitDistinguishedName Tags: Env: !Ref Env Project: !Ref Project IamRoleArn: Fn::ImportValue: !Sub "${Env}-AppstreamFleetRoleArn" StreamView: !Ref FleetStreamView |
Conclusion
Thanks to the custom resource, I managed to deliver to the customer a fully efficient cloudformation template, that he can use to automatically create all of his AppStream resources in one click, but also delete when they don’t need it anymore.
This article provides you a small overview of what can be done with the cloudformation resources.
Furthermore, it is important to note that the cloudformation stack also calls the custom resource when there is an update, and that we could implement an update function that will update the AWS resources.