Leveraging existing event sources (S3 and CognitoUserPools)

Cloud computing introduced a drastic shift when it was first introduced in the early 2000s. Providers such as Amazon Web Services (AWS) finally made it possible to rent compute, storage and other web related services on a monthly or yearly subscription basis. Getting projects into production was a matter of a credit card swipe and a deployment. Contrasting this with the usual, months long capacity planning and procurement processes shows why cloud computing was such a game changer in the tech industry.

As a result more and more developers and companies adopted cloud computing as their go-to solution to host their applications in an easy, future proof and cost effective way.

Given that AWS started in the early 2000s it’s easy to imagine how applications grew within the AWS ecosystem and how much data such applications generated, processed and accumulated. The big data trend turned the attention to the value in the data. One of AWS reactions to this trend was the introduction of the Lambda compute service back in 2014.

The very first event sources AWS introduced for Lambda was support for events generated by S3 buckets. To this day the infamous „Image resizer“ demo where an image is uploaded to an S3 bucket and automatically resized by a Lambda function is still the most prominent „Hello World“ application to showcase the power of serverless architectures.

Existing event sources

Our dive into the AWS history and Serverless computing above showed us that AWS Lambda was initially built around the idea of working with data from existing infrastructure components such as storage buckets.

Once API Gateway support as a Lambda event source was announced it was clear that the serverless paradigm will be huge. The JAWS CLI tool (The Serverless Framework precursor) was born in 2015 out of the necessity to provide an easy way to leverage the power serverless unleashes by making it as simple as possible to author serverless applications.

In 2016, the Serverless team sat down and worked together with the community to define the integral parts for our upcoming Serverless Framework v1 release. One main takeaway during this exercise was that we should support CloudFormation which is the de-facto standard to define and deploy infrastructure on AWS in a declarative way.

While CloudFormation gives us a lot of upsides, mainly in the form of a battle-tested and well-known platform to perform resilient infrastructure changes, it not short of downsides as well. One such downside is the focus on infrastructure creation and teardown which means that CloudFormation was never designed to deal with resources which were not part of the initial creation phase.

It’s clear that this restriction prevents CloudFormation users from introducing existing infrastructure components such as S3 buckets or Cognito User Pools into their current setup. This problem trickles down to the Serverless Framework which utilizes CloudFormation under the hood.

Introducing event support for existing S3 buckets or Cognito User Pools turned into a problem only Serverless Plugins were able to solve. Such plugins relied on raw AWS SDK calls, and their usage introduced a second method to allow the management of application infrastructure.

Within recent versions of the Serverless Framework, we finally found a way to support existing resources via CloudFormation which means that we still get all the benefits CloudFormation has to offer while making it possible to introduce existing, potentially legacy infrastructure components, into the application stack.

S3

We already read above that S3 is one of the first and most widely used AWS services. Given that S3 has been around for such a long time, it’s easy to imagine that a lot of data was accumulated in S3 buckets throughout the years. S3 buckets are often used as the central object store to save and retrieve files such as images, personal documents or log files.

S3 buckets can emit a variety of different event notifications such as s3:ObjectCreated:* which can be used to get notifications when a new object is uploaded to the bucket or s3:ObjectRemoved:* which will send a notification when an object was deleted from the bucket. Different prefix and suffix configurations make it possible to further filter down when such events should be emitted.

The Serverless Framework comes with native support for the s3 event source. Let’s take a look at an example where we want our notify function to be called whenever a new .pdf file is created in the documents directory of our acme-cloud-storage bucket:

service: acme-cloud

provider:
  name: aws
  runtime: nodejs10.x

functions:
  notify:
    handler: handler.hello
    events:
      - s3:
          bucket: acme-cloud-storage
          events: s3:ObjectCreated:*
          rules:
            - suffix: .pdf
            - prefix: documents

Deploying this service with the Serverless Framework will create the acme-cloud-storage bucket and sets up the bucket configuration to call the notify function accordingly.

What if our acme-cloud-storage bucket is already in production and we want to setup our notify Lambda function after the fact?

Thanks to our recent changes this can be achieved via the existing: true flag:

service: acme-cloud

provider:
  name: aws
  runtime: nodejs10.x

functions:
  notify:
    handler: handler.hello
    events:
      - s3:
          bucket: acme-cloud-storage
          events: s3:ObjectCreated:*
          rules:
            - suffix: .pdf
            - prefix: documents
          existing: true

The Serverless Framework will automatically detect that the acme-cloud-storage bucket was already created previously and will just setup the configuration on the bucket. Once deployed our notify function will be called whenever files that match our rules are uploaded to the existing bucket.

Cognito User Pools

Sometimes it’s useful to react to specific user behavior within your application. A typical user-related use case is the sending of “Welcome” or “Getting started” emails whenever a user decides to sign up for your service.

Cognito and especially Cognito User Pools are a widely used service to safely integrate and manage user accounts within web-facing applications.

Cognito User Pools offer an easy way to create customized workflows via Triggers which can be setup on User Pools. Triggers include PreSignUp, PostAuthentication, CustomMessage and more.

The Serverless Framework supports Cognito User Pools natively. Let’s create a service in which a greeter Lambda function is invoked whenever a new user signup is registered in our AcmeUsers User Pool:

service: acme-user-mgmt

provider:
  name: aws
  runtime: nodejs10.x

functions:
  greeter:
    handler: users.handler
    events:
      - cognitoUserPool:
          pool: AcmeUsers
          trigger: PreSignUp

Deploying this service will create the AcmeUsers Cognito User Pool and sets up the configuration so that the greeter function is called whenever a new user signs up for our application.

Given that the service definition above will always attempt to create a new Cognito User Pool it seems to be hard to just setup the trigger configuration without interfering the production system.

Thanks to the recent changes we made it’s dead simple to setup trigger configurations on existing Cognito User Pools. The only configuration which needs to be added is the existing: true flag. This flag tells the Serverless Framework to skip the Cognito User Pool creation phase and instead configure the triggers on the existing one:

service: acme-user-mgmt

provider:
  name: aws
  runtime: nodejs10.x

functions:
  greeter:
    handler: users.handler
    events:
      - cognitoUserPool:
          pool: AcmeUsers
          trigger: PreSignUp
          existing: true

Using this service setup makes it possible to introduce a previously created Cognito User Pool (possibly running in production for a while) into our current serverless setup.

Conclusion

Using event-driven application patterns with existing infrastructure components such as S3 buckets or Cognito Users Pools is not a rare circumstance.

Given that the Serverless Framework is built on top of CloudFormation to provide production-grade reliability and resilience it was only possible to introduce external event sources via Serverless Plugins. We thought hard about potential solutions to mitigate this shortcoming. While working on the solution one of our main goals was to not give up the benefits CloudFormation provides us.

Recently we finally found a way to integrate support for existing infrastructure components into core. The first event sources for which we added this support include S3 buckets and Cognito User Pools. Setting them up requires just one line of code:

existing: true

You can read more about the existing: true flag in the S3 and Cognito User Pools docs.

How are you using this new feature with the Serverless Framework? Share your thoughts and ideas in the comments below or tweet us @goserverless.