Architectural Considerations

While a discussion on different types of architectures deserves much more than this section provides, this page discusses three topics:

  1. Serverless architectures
  2. Stateless microservices
  3. Cloud-native applications

Serverless architectures

Serverless architectures rely extensively on either ephemeral functions reacting to events (FaaS or Lambda) or third party services that are exposed only through API calls. Applications require serverless architectures to be elastic, with usage-based pricing and zero operational burden. As such, Confluent Cloud Basic and Standard clusters are elastic, automatically scaling up when there is higher demand, i.e., more events and more API calls, and automatically scaling down when demand is lower. They have usage-based pricing that is based on per-event or per-API call. Other than calling an API or configuring the function, there is no user involvement in scaling up or down, failure recovery, or upgrades; Confluent is responsible for the availability, reliability, and uptime of your Kafka clusters. Confluent Cloud’s serverless offering includes not only core Kafka broker services but also event stream processing with ksqlDB and data migration into and out of end systems with fully managed connectors. At a high level, this achieves an ETL pipeline: move data into Confluent Cloud (extract), create long-running, auto-scaling streams transformations by publishing SQL to a REST API (transform), and persist this data (load).

Stateless microservices

You also may build your application to speak to Confluent Cloud with stateless microservices. Microservices architectures build applications as a collection of distributed, loosely coupled services, which works well in a cloud environment where the cloud providers themselves give you access to distributed services. Data storage in its many forms is typically handled by external services, whether it’s fully managed Kafka with Confluent Cloud or any of the cloud provider services. This means that the microservices that make up your cloud-native application can be stateless and rely on other cloud services to handle their state. Being stateless also allows you to build more resilient applications, since loss of a service instance doesn’t result in a loss of data because processing can instantly move to another instance. Additionally, it is far easier to have components and usage that scale automatically, such that you can deploy another microservice comonent as elastically as you are able to grow your Confluent Cloud usage elastically.

Cloud-native applications

When using Confluent Cloud, your Kafka client applications may be cloud-native such that they are also running in the cloud. While new applications can be developed on Confluent Cloud from inception, it may be the case that some of your legacy applications migrate to the cloud over time. The path to cloud may take different forms:

  1. Application is cloud native, and is also running in the cloud
  2. Application runs in an on-premise Kafka cluster, and then you have the bridge-to-cloud pattern in which Confluent Replicator streams data between your Kafka cluster and Confluent Cloud
  3. Application runs on prem to Confluent Cloud, and then you migrate the application to cloud over time

Developers who run applications in the cloud for the first time are often surprised by the volatility of the cloud environment. IP addresses can change, certificates can expire, servers are restarted, entire instances are sometimes decommissioned, and network packets going across the WAN are lost more frequently than in most data centers. While it is always a good idea to plan for change, when you are running applications in the cloud, it is mandatory to plan for it. Since cloud environments are built for frequent change, a cloud-native design lets you use the volatile environment to your advantage. The key to successful cloud deployments is to build applications that handle the volatility of cloud environments gracefully, which results in more resilient applications.