AWS | Text Analysis Via SNS Post Processing

This example demonstrates how to setup a simple data processing pipeline.

user  

Adam Bergman

Data processing

This example demonstrates how to setup a simple data processing pipeline. The service exposes one HTTP endpoint that allows you to add a text note. This HTTP endpoint returns instantly to provide a good user experience while the actual analysis is deferred. Only messages above a certain sentiment level are actually saved.

Instead of invoking another Lambda function directly it's considered best practice to store the note as a message in a SNS queue. The queue has certain benefits compared to invoking the analyzeNote function directly. The queue supports retries in case the analyzeNote function fails as well as back-off to avoid too many concurrent invocations.

Setup

npm install

In order to use SNS you need to add your AWS account ID to config.js. There is already a placeholder: XXXXXXXXXXXX.

You can retrieve the your account ID by running this command (you need the AWS SDK installed)

aws sts get-caller-identity --output text --query Account

Explanation

  • sns topic will be added by default

Deploy

In order to deploy the you endpoint simply run

serverless deploy

The expected result should be similar to:

Serverless: Packaging service…
Serverless: Uploading CloudFormation file to S3…
Serverless: Uploading service .zip file to S3…
Serverless: Updating Stack…
Serverless: Checking Stack update progress…
............
Serverless: Stack update finished…
Serverless: Removing old service versions…

Service Information
service: text-analysis-via-post-processing
stage: dev
region: us-east-1
api keys:
  None
endpoints:
  POST - https://5cvfn0wwv7.execute-api.us-east-1.amazonaws.com/dev/notes
functions:
  text-analysis-via-post-processing-dev-analyzeNote: arn:aws:lambda:us-east-1:377024778620:function:text-analysis-via-post-processing-dev-analyzeNote
  text-analysis-via-post-processing-dev-addNote: arn:aws:lambda:us-east-1:377024778620:function:text-analysis-via-post-processing-dev-addNote

Usage

In order to add a note run

curl -X POST https://XXXXXXXXX.execute-api.us-east-1.amazonaws.com/dev/notes --data '{ "note": "This is such a great Day" }'

You should see the following output

{"message":"Successfully added the note."}%

To verify that the note has been processed run

serverless logs --function analyzeNote

This command will show you the logged output and looks liked this

START RequestId: 75a970ba-ab11e6-809d-435833490828 Version: $LATEST
2015 17:56:32.497 (+01:00)	75a970ba-ab11e6-809d-435833490828	Positive note - will be published: This is such a great Day
END RequestId: 75a970ba-ab11e6-809d-435833490828
REPORT RequestId: 75a970ba-ab11e6-809d-435833490828	Duration: 3.45 ms	Billed Duration: 100 ms 	Memory Size: 1024 MB	Max Memory Used: 15 MB

You can play with the system and see which notes will be published and which won't.

Scaling

TODO

Latest commit b2f54ec on Sep 24, 2017

New to serverless?

To get started, pop open your terminal & run:

npm install serverless -g

Made with love in San Francisco + Atlanta, Austria, Germany, Pakistan, Poland, Nebraska & Thailand

Serverless, Inc. © 2018

Join our monthly newsletter to get the latest Serverless news, updates and happenings.