AWS: How to deploy ScyllaDB cluster
Amazon Web Services has long been the go-to destination for startups and enterprises alike looking to launch new projects swiftly and efficiently. The allure lies in its extensive array of managed services, designed to streamline the development process and allow businesses to focus on their core competencies. However, as time passes and your project matures, the financial realities of operating within the AWS ecosystem may start to emerge. The convenience of AWS-managed services comes at a cost, and what begins as an economical and expedient choice can gradually evolve into a budgetary challenge. Scaling resources vertically and horizontally, while maintaining the performance and reliability expected by your users, can become a substantial financial burden.
For some, DynamoDB — the NoSQL database service provided by AWS — has been the backbone of their data storage needs. Coupled with DynamoDB Accelerator (DAX), it seemed like the perfect solution for fast, scalable, and fully managed databases. However, the reality of operating DynamoDB and DAX at scale has shown that they may not always live up to their promises of reliability and cost-effectiveness.
In this article, we embark on a journey to explore an alternative: ScyllaDB, open source edition. From cost considerations to reliability concerns, we’ll uncover the compelling arguments that drive this shift in database strategy, and how it might benefit your long-term operations. Apart from this, I’ll also show you how to deploy your very own ScyllaDB cluster in AWS.
Part 1. Why switching?
In the realm of software engineering, the saying “if it works, don’t touch it” is often mentioned. It’s a principle that makes perfect sense — why tamper with a well-functioning system? However, in the dynamic landscape of modern technology, staying agile and cost-effective is essential. This often necessitates considering alternatives, even when you have a functional solution in place.
For those of us who’ve embarked on this journey of optimization, AWS offers a valuable tool: the AWS Pricing Calculator. It serves as our compass when navigating the complex terrain of cloud service costs. Let’s illustrate this with a quick cost calculation example.
In our DynamoDB scenario, we’re dealing with a workload that ranges from 1000 to 5000 write/read operations per second, supported by three DAX nodes of r5.large size.
Now, let’s compare this with a self-hosted solution where you’d manage three EC2 i3.large instances, each equipped with ScyllaDB.
The difference is nothing short of staggering.
Of course, the above is a simplified example, reflecting minimal instance types and counts. With ScyllaDB, the cost curve tends to be linear; scaling often means a straightforward doubling of expenses. DynamoDB and DAX, however, present a more intricate cost structure — try experimenting with it in the calculator.
It’s worth noting that while ScyllaDB does require investments in setup and potentially ongoing maintenance, the same holds true for DynamoDB and DAX however. In my personal experience, spanning the past two years, there have been seven (!) production incidents of different scale involving the AWS-managed solution. While individual experiences may vary, based on my own journey, I’ve come to strongly recommend exploring alternatives to DynamoDB and DAX.
Now that we’ve addressed the “why,” let’s delve into the technical aspects of the deployment process.
Part 2. Building AMI
Before we can launch our cluster, the first step will be the creation of an Amazon Machine Image (aka AMI) to run these EC2 instances. ScyllaDB provides prebuilt AMIs, ready to launch directly from the AWS console. However, you might find it necessary to install additional packages on these AMIs, such as Datadog or SSM agents. To accomplish this, we’ll begin by creating an Ansible playbook that installs ScyllaDB and sets up the required configuration files.
Notably, ScyllaDB is versatile when it comes to supported operating systems, with the recommended one being Ubuntu 22.04 at the time of writing. But bear in mind that the landscape can change, so always refer to the latest ScyllaDB documentation for the most up-to-date information.
Here’s the Ansible playbook, main.yaml, which accomplishes this task:
The playbook itself is rather straightforward. Let’s focus on the essential configuration file, scylla.yaml, and the pivotal scripts needed to bootstrap the database.
The scylla.yaml file serves as the central hub for tailoring ScyllaDB’s behavior. Below is a portion of the configuration file with key settings:
Depending on your use case you might need to set up more (or less) configurations. Make sure to configure Ec2Snitch since we are running in the AWS environment!
We will have a couple of bash scripts in order to facilitate the bootstrap process. These scripts ensure that ScyllaDB is properly configured and initialized.
A cluster requires at least one seed node. Keep in mind that the scylla_setup script configuration is something you might need to adjust as per your specific use case.
Every other node will be bootstrapped with a reference to the seed node. Make sure that scylla_setup configurations are identical!
The database schema should be set up on the seed node only. All other nodes will “receive” it while joining the cluster.
Finally setup.cql is the CQL script to set up the initial state of DB. It would usually contain the creation of a keyspace and tables in it. Sample file is below:
As you can see we replace <DATACENTER> here with the region (but without a “-1” part). You can read more about this here.
Even though Bash scripts might seem old-school in today’s tech landscape, we use them here for a straightforward reason: they make things simple and reliable. These scripts automate the essential configurations, ensuring a reliable ScyllaDB deployment. However, if the prospect of dealing with Bash scripts feels antiquated or you prefer a more hands-off approach, there’s an alternative: ScyllaDB offers a managed service that handles these tasks for you, albeit for a fee.
Now that we have our AMI ready, it’s time to proceed with deploying it in the AWS environment, which we’ll explore in the upcoming section.
Part 3. Deploying cluster
Without further ado, let’s jump straight into the CloudFormation template.
Let’s get an overview of the key sections in the CloudFormation template:
ScyllaInterNodeSecurityGroup: This is a security group for enabling communication between ScyllaDB nodes. It is crucial to have the required ports opened in the security group, otherwise the nodes won’t be able to communicate.ScyllaDBSeedInstance: An EC2 instance serving as the seed node for the ScyllaDB cluster. It runs thesetup-seed-node.shscript during initialization.ScyllaDBAutoscalingGroup: An Auto Scaling group that manages the ScyllaDB not seed nodes. It runs thesetup-not-seed-node.shscript during initialization.
Make sure to have the DependsOn attribute correctly configured — AutoScalingGroup relies on the seed node to be up and running with ScyllaDB bootstrapped.
The CloudFormation template allows customization by providing parameters like VPC ID, subnet IDs, AMI ID, and instance type. You can configure as many regions as you need like this:
As you might have noticed the given template will only launch two instances — one seed and one not seed. Bootstrapping a cluster might be a very time consuming activity, but it can be mitigated with automation — a Jenkins job to deploy the cluster and also upscale it to 3 instances in total.
Due to the way ScyllaDB replicates data the cluster can only be increased one node at a time. By automating this process, you can ensure that nodes are started sequentially, maintaining data consistency and cluster health. You can extend the Jenkins job to add more nodes, but in this case don’t forget to add the checks to guarantee that all ScyllaDB nodes are up and running.
Afterwards
In the ever-evolving world of cloud computing, the allure of AWS managed services can be irresistible. They offer convenience, scalability, and reliability for your projects. However, as we’ve explored in this article, it’s essential to keep a watchful eye on costs.
While AWS services can kickstart your project with ease, their pricing models may not be as friendly to your budget in the long run. In the pursuit of efficiency and cost-effectiveness, it’s worth considering open-source alternatives.
Embracing open-source solutions like ScyllaDB may require a bit more setup and ongoing maintenance, but the benefits can be substantial. You gain more control over your infrastructure, avoid unforeseen cost spikes, and often achieve better performance.
So, the next time you embark on a cloud journey, remember that striking the right balance between managed services and open source can be the key to success. It might be “harder” in some aspects, but in the end, it’s a strategic move that can yield superior results for your projects and your budget alike.
Support
If you like the content you read and want to support the author — thank you very much!
Here is my Ethereum wallet for tips:
0xB34C2BcE674104a7ca1ECEbF76d21fE1099132F0
