When we decided to launch and bootstrap CloudForecast in 2018, we faced questions that every startup face including "What’s the right way to build this product with our resources without creating technical debts for the future?"
Around the time we started CloudForecast, I was reading "$100 Startup" by Chris Guillebeau. In his book, Chris advises his readers to get their product out fast while being cost-effective to limit the damage if your idea fails (definitely possible, unfortunately!). We try to apply this advice on every side of our business including our technical decisions.
With the "$100 Startup" concept in mind, we started by listing the requirements that were important to us while building CloudForecast:
AWS Lambda + Serverless = Easy + Focus + Cost-Effective
We brainstormed on how we could achieve our goals and requirements. We discussed managing our own instances, using containers and half a dozen other ideas but we wanted something simple so we decided to use a Serverless solution (or FaaS). While there are a few downsides (e.g cold starts, …), we believed going serverless better suited our use case (nearly zero administration, pay-per-execution with no idle cost and auto-scaling). Cold starts is a known downside for Lambda but as we are mostly transforming and loading data from S3 in an offline fashion we decided that cold starts weren't a major concern for us.
For the dev setup and deploy process, we decided to use Serverless Framework with AWS Lambdas for the following reasons:
Our original requirements were fairly simple: We needed 4 AWS lambda functions each on their own cron job, and each lambda would need to talk to various AWS products (RDS, DynamoDb, SQS et al.). All that with an easy way to manage multiple environments (dev vs prod) and an easy/effective way to manage resource permissions.
Here is how we do it:
With these two code snippets, we were able to set up most of our architecture. 4 functions that will interact with each other through SNS. Two functions will run check on a cron schedule (via the ‘schedule.rate’ params) to check a file need to be reprocessed and trigger another function via an SNS. This configuration will be able to scale effortlessly while keeping our costs under control. We are able to fully silo our environments using iamRoleStatements to configure the permissions.
We originally used a simple YAML file to control our environment variables but we quickly switched to a
DotEnv file using the DotEnv plugin.
We considered running a couple of small instances to do the job which would cost us at least $1k for the year. However, the AWS Lambda cost is effectively $0 since we are only running a couple of functions a day which could easily be covered by the AWS free tiers.
Like every new startup, we made (and are probably still making) some mistakes along the way but picking AWS Lambda and the serverless framework wasn’t one of them. Here are a few reasons why it was the right choice for us:
Over time, our product evolved and so did our functions but Serverless and Lambda were always able to deliver.
If you have any questions related to this post or what we do at CloudForecast.io, feel free to reach out to me at firstname.lastname@example.org. We would love to hear from you!