Architectural considerations for Confluent Cloud clients

While a discussion on different architectures deserves much more than this section provides, this page discusses three topics:

  • Serverless architectures
  • Stateless microservices
  • Cloud-native applications

This topic summarizes the Confluent Cloud client architectural considerations for serverless architectures, stateless microservices, and cloud-native application architectures.

Serverless architectures

Serverless architectures rely extensively on either ephemeral functions reacting to events (FaaS or Lambda) or third party services that are exposed only through API calls. Applications require serverless architectures to be elastic, with usage-based pricing and zero operational burden. As such, Confluent Cloud Basic, Standard, Enterprise, and Freight clusters are elastic. They have usage-based pricing that is based on throughput and storage.

Other than calling an API or configuring the function, there is no user involvement in failure recovery or upgrades; Confluent is responsible for the availability, reliability, and uptime of your Kafka clusters.

Serverless offerings in Confluent Cloud include core Kafka broker services and event stream processing with ksqlDB and data migration into and out of end systems with fully managed connectors. At a high level, this achieves an ETL pipeline: move data into Confluent Cloud (extract), create long-running, auto-scaling streams transformations by publishing SQL to a REST API (transform), and persist this data (load).

Stateless microservices

You also may build your application to speak to Confluent Cloud with stateless microservices. Microservices architectures build applications as a collection of distributed, loosely coupled services. loosely coupled services work well in a cloud environment where the cloud providers themselves give you access to distributed services.

Data storage is typically handled by external services, whether it’s fully managed Kafka with Confluent Cloud or any of the cloud provider services. This means that the microservices that make up your cloud-native application can be stateless and rely on other cloud services to handle their state. Being stateless also allows you to build resilient applications as loos of a service instance doesn’t result in a loss of data. Data processing can move to another instance.

Cloud-native applications

When using Confluent Cloud, your Kafka client applications may be cloud-native such that they are also running in the cloud. While new applications can be developed on Confluent Cloud from inception, it may be the case that some of your legacy applications migrate to the cloud over time. The path to cloud may take different forms:

  • Application is cloud native, and is also running in the cloud
  • Application runs in an on-premise Kafka cluster, and then you have the
    bridge-to-cloud pattern in which Confluent Replicator streams data between your Kafka cluster and Confluent Cloud
  • Application runs on prem to Confluent Cloud, and then you migrate the application to
    cloud over time

Developers running cloud applications for the first time often find the cloud environment surprisingly volatile:

  • IP addresses can change,
  • certificates can expire,
  • servers can restart,
  • entire instances can be decommissioned, and
  • network packets crossing the WAN can be lost more frequently than in most data centers.

You should plan for change when running cloud applications. Applications designed for cloud-native environments handle the volatility of the cloud gracefully, resulting in more resilient applications.