Where Elasticity Meets Open Source — and How to Adapt

Operating more efficiently with reduced infrastructure costs, maintenance and downtime risks has never been more important to businesses, even when customers demand best-in-class experiences.

The current volatile macro environment – ​​driven by the fallout from the pandemic, an impending economic recession and political turmoil – is causing organizations to think about cost-cutting strategies and reduce the total cost of ownership (TCO) for IT infrastructure quickly, but intelligently.

But even as companies seek to reduce IT costs, they prioritize strategies that drive innovation and agility and help better serve customers. As a result, we’re seeing more and more organizations focus on modernizing their data infrastructure to gain insights and extract value from all the data they collect – and drive these initiatives forward. And as they move and build to the cloud, organizations expect their data systems to be seamlessly and transparently elastic.

Elasticity, one of the cornerstones of cloud native computing, allows disparate infrastructure resources to be allocated to meet an organization’s needs. Businesses can operate more efficiently with lower infrastructure costs by only using and paying for what they need, when they need it.

Simply put, you want to scale your cloud workloads in a way that doesn’t increase your costs exponentially. And elasticity helps adjust workloads to optimize your cost curve.

More importantly, elasticity isn’t just about scaling faster, it’s also about the ability to shrink infrastructure resources as needed to ensure you’re not overpaying for infrastructure—a key requirement in these uncertain times.

And as cutting costs, modernizing data infrastructure, meeting or exceeding service level objectives (SLOs), and accelerating time to market become key priorities—while adapting quickly to dynamic market demands—more organizations are embracing open source software and technologies.

The rise of open source projects in the enterprise – and their hidden costs

Open source software and technologies are a critical component of modern IT success and the backbone for driving digital innovation. A new report from Red Hat found that 95% of IT leaders see open source as important to their organization’s overall infrastructure.

Take Apache Kafka, an open source distributed data streaming platform. As organizations recognize the value of prioritizing data as the foundation for digital transformation and differentiation, Apache Kafka has quickly become the cornerstone of modern data infrastructure.

Today, more than 70% of Fortune 500 companies use Kafka as their real-time data streaming platform of choice – helping to drive business growth and unparalleled customer experiences.

Despite the many benefits of open source software, the costs associated with deployment and maintenance can manifest themselves in a number of ways.

For example, while Kafka is free to download, modify, use, and redistribute, self-managing a complex distributed system at scale still entails significant operational and development costs, the risk of costly downtime and security breaches, and increases business expenses. Despite the many advantages of open source software, the costs and lack of elasticity associated with administration can manifest themselves in a number of ways.

These costs can be divided into several categories:

• infrastructure costs: Organizations must pay for the underlying infrastructure and its management before Kafka software development and operational costs. Over-provisioning is required to support fluctuating demand and there is an inability to scale storage without increasing compute resources.
• Scalability costs: Another challenge is scalability at a reasonable cost. As usage increases, companies face exorbitant scaling costs due to manual processes.
• FTE and operating costs: Includes allocation of engineering resources to administer Kafka, in addition to requiring engineers to create components and tools for a more complete data streaming architecture rather than building data streaming applications or other high-value projects for the enterprise. In addition, it is becoming increasingly difficult to hire and retain Kafka talent.
• Downtime costs: Includes costs related to unexpected cluster failures and maintenance as Kafka covers more use cases, data systems, teams, and environments. Valuable resources are then redirected to address unplanned downtime and violations. While these costs are difficult to quantify, their importance is immediately apparent when an incident occurs.

When it comes to adding elasticity to the streaming infrastructure, scaling out individual Kafka clusters has its limitations. As the number of brokers increases, the number of connections per broker required to manage replication becomes difficult and impacts performance. And it increases operating costs. Vertical scaling also brings challenges. Scaling up storage increases the time to recover from a failure.

How to solve the Kafka elasticity and cost problem

The latest development in open source offerings is in the area of ​​managed services. They give companies even more ways to benefit from the open source community by making configuration, monitoring, and management of open source software easier and more reliable, while saving costs and resources.

With a fully managed service for a popular open source project like Apache Kafka, your organization can shift critical resources to higher-value work. They guarantee zero operational overhead by ensuring clusters are deployed instantly and maintenance is seamlessly managed. Plus, you can skip the part where you spend six to nine months hiring and training employees and instead focus immediately on product development.

Additionally, it does not involve capacity planning, data balancing, or other typical operational loads associated with scaling data infrastructure, ensuring your business runs faster and more efficiently, with reduced infrastructure costs, maintenance, and downtime risks.

In fact, offloading Kafka infrastructure and operations to a fully managed cloud-native data streaming service can minimize technology burden and risk, focus your best people on their critical projects, and improve total cost of ownership in a number of ways, including:

• Avoid oversupply: Apache Kafka best practices traditionally recommend deploying your cluster for peak usage. This is often months or even quarters ahead of the traffic. Using a fully managed service helps reduce this to just a few hours, allowing you to scale right before traffic kicks in, rather than wasting weeks of idle capacity.
• Increase resilience and availability: Building a production-level Service Level Agreement (SLA) requires a lot of effort and resources, including the engineer’s time. And the slightest difference in SLA means exponentially more downtime for your customers. When broker failures occur, fully managed services ensure you’re able to automatically and quickly heal or add new brokers, which is critical to ensure a cluster meets its SLAs and provides the user with the appropriate amount of capacity provides.
• One-click deployment: For fully managed services, scaling can be automated or done at the push of a button. That means you don’t have to spend time on capacity planning, network setup, load balancing, or anything like that. Instead, your highly paid full-time equivalents (FTEs) could focus on innovation and developing mission-critical applications.

Building scalability into infrastructure has never been more important, and leveraging cloud elasticity has become a powerful way for organizations to ensure they’re ready for anything.

A key differentiator of Confluent Cloud, a fully managed cloud-native service for Apache Kafka, is elastic scaling, which applies to both growing and shrinking. You can shrink clusters just as quickly as you can expand them. The result? You avoid overpaying for excess capacity when traffic slows.

Want to learn more about how Confluent took Apache Kafka’s horizontal scalability to the next level and made Confluent Cloud 10x more elastic? Watch our video overview and customer stories now.