# Architectural considerations for streaming applications on Confluent Cloud Stream Catalog provides data discovery and lineage tracking to understand downstream impacts of schema changes. Schemas can define quality expectations and tag sensitive data (PII, financial). Schema Registry enforces schema compatibility and serves as the central source of truth for data structures. BYOK encryption lets you manage your own encryption keys for compliance requirements. Key architectural topics: - **Stream Catalog and impact analysis** - Data lineage, schema change impact, compliance reporting, and data discovery - **Schemas and PII tagging** - Quality rules, sensitive data labeling, metadata management - **Schema Registry and schema compatibility** - Centralized schema management, evolutionary compatibility, schema enforcement - **Encryption architecture** - Self-managed keys (BYOK), client-side field-level encryption (CSFLE) - **Platform architecture diagrams** - Core services, cluster types, networking patterns - **Cluster configuration planning** - Critical decisions that impact security and compliance - **Stream processing integration** - Real-time data processing with Apache Flink® - **Serverless and cloud-native patterns** - Event-driven, elastic streaming architectures - **Network and security architecture** - Networking patterns and access control strategies - **Observability and monitoring** - Operational monitoring and compliance strategies ## Cluster configuration planning When planning your streaming application architecture, consider cluster-level configuration decisions that must be made during cluster creation and cannot be changed later. These decisions can impact your application’s security, compliance, and data handling requirements. ### Self-managed encryption keys (BYOK) If your application handles sensitive data or operates in a regulated industry (finance, healthcare, government), you may need to use self-managed encryption keys to encrypt data at rest. This option provides greater control over your encryption keys and may be required for compliance purposes. #### IMPORTANT Self-managed encryption keys must be configured when creating Enterprise or Dedicated clusters. You cannot switch between automatic (default) and self-managed encryption modes after cluster provisioning. For applications requiring self-managed encryption keys: - Plan your key management strategy before creating your cluster. - Consider your cloud provider’s key management service (AWS KMS, Azure Key Vault, or Google Cloud KMS). - Review compliance and regulatory requirements that may mandate this option. For detailed information about configuring and managing self-managed encryption keys, see [Protect Data at Rest Using Self-Managed Encryption Keys on Confluent Cloud](../security/encrypt/byok/overview.md#byok-encrypted-clusters). ## Data schema architecture and governance Modern streaming applications require robust data schema management and governance to ensure data quality, compatibility, and organizational alignment. Confluent Cloud’s [Schema Registry](../sr/schemas-manage.md#sr-prv) and [Stream Governance](../stream-governance/packages.md#stream-gov-packages) provide comprehensive tools for managing schemas, enforcing schema rules, and maintaining data quality across your streaming architecture. ### Schema Registry architectural value Schema Registry serves as the central nervous system for data compatibility in streaming architectures, providing several critical architectural benefits: #### Schema enforcement - **API-like contracts**: Schemas define explicit data contracts between producers and consumers, similar to API contracts in REST services. - **Evolutionary compatibility**: Enable safe schema evolution while maintaining backward and forward compatibility guarantees. - **Organizational alignment**: Provide a single source of truth for data structures across teams and services. #### Operational resilience - **Prevent data corruption**: Block incompatible schema changes that could break downstream consumers. - **Reduce deployment risks**: Validate schema compatibility before deploying new application versions. - **Centralized versioning**: Track schema evolution and enable rollback capabilities. #### Development efficiency - **Code generation**: Auto-generate strongly-typed classes from schemas for Java, C#, Python, and other languages. - **IDE integration**: Enable autocomplete, type checking, and refactoring support in development environments. - **Documentation**: Schemas serve as living documentation of data structures. ### Schema design patterns #### Subject name strategies Choose subject name strategies that align with your architectural patterns: - **TopicNameStrategy**: Default pattern using `-key` and `-value` subjects. Suitable for single-team ownership of topics. - **RecordNameStrategy**: Uses fully qualified record names as subjects. Enables schema reuse across multiple topics. - **TopicRecordNameStrategy**: Combines topic and record names. Provides namespace isolation while enabling schema reuse. For detailed implementation guidance, see [broker-side schema validation](../sr/broker-side-schema-validation.md#cloud-schema-validation) which covers subject name strategies for Dedicated clusters. Consider your organizational structure when selecting strategies: ```none Organizational Pattern → Recommended Strategy │ ├── Single Team per Topic → TopicNameStrategy ├── Shared Schemas → RecordNameStrategy └── Multi-tenant → TopicRecordNameStrategy ``` #### Schema evolution strategies Plan schema evolution based on your compatibility requirements: **Backward compatibility** (Default) : New schemas can read data written with previous schemas. Suitable for consumer-driven evolution where new consumers must handle old data. **Forward compatibility** : Previous schemas can read data written with new schemas. Suitable for producer-driven evolution where old consumers must handle new data. **Full compatibility** : Combines backward and forward compatibility. Provides maximum flexibility but requires careful schema design. **No compatibility** : No compatibility checks. Use only when you control both producers and consumers and can coordinate schema changes. For comprehensive schema evolution guidance, see the [Schema Registry tutorial](../sr/schema_registry_ccloud_tutorial.md#schema-registry-ccloud-tutorial) and [schema compatibility management](../sr/schemas-manage.md#cloud-schema-compat-mode). Compatibility selection framework: ```none Deployment Pattern → Compatibility Mode │ ├── Consumer-first (new consumers, old data) → BACKWARD ├── Producer-first (old consumers, new data) → FORWARD ├── Independent deployment → FULL └── Coordinated deployment → NONE (with caution) ``` ### Data governance patterns Stream Governance provides enterprise-grade data governance capabilities for streaming data, enabling organizations to maintain data quality and compliance at scale. #### Schema rules [Schema rules](../sr/fundamentals/data-contracts.md#sr-data-contracts) define quality expectations and business rules for your streaming data: - **Field-level validation**: Enforce data type, format, and range constraints on individual fields. - **Business rule validation**: Implement custom validation logic for complex business requirements. - **Quality monitoring**: Track data quality metrics and alert on violations. Example schema rule patterns: ```json { "schemaType": "AVRO", "schema": "...", "qualityRules": [ { "name": "email_format", "expression": "email REGEXP '^[^@]+@[^@]+\\.[^@]+$'", "onFailure": "ERROR" }, { "name": "amount_range", "expression": "amount > 0 AND amount < 1000000", "onFailure": "WARN" } ] } ``` #### Data lineage and catalog [Stream Catalog](../stream-governance/stream-catalog.md#cloud-stream-catalog) provides comprehensive data discovery and lineage tracking: - **Automatic discovery**: Discover and catalog all data streams, schemas, and their relationships. - **Impact analysis**: Understand downstream impacts of schema changes before implementation. - **Compliance reporting**: Generate data lineage reports for regulatory compliance (GDPR, CCPA, SOX). #### Multi-environment schema management Design schema promotion strategies for different environments: - **Environment isolation**: Use separate Schema Registry instances or contexts for development, staging, and production. See [schema contexts](../sr/schemas-manage.md#work-with-schema-contexts) for advanced namespace management. - **Schema promotion**: Implement CI/CD pipelines that promote schemas through environments with compatibility validation. Use the [Schema Registry Maven plugin](../sr/develop/maven-plugin.md#sr-maven-plugin) for automated compatibility testing. - **Version alignment**: Ensure schema versions are synchronized across environments during deployments. ### Schema Registry integration patterns #### Stream processing integration Schema Registry integrates seamlessly with stream processing platforms: - **Flink integration**: [Apache Flink](../flink/overview.md#ccloud-flink) automatically handles schema evolution and serialization when reading from and writing to Kafka topics with registered schemas. - **Automatic deserialization**: Stream processors automatically deserialize data using the appropriate schema version. - **Schema evolution handling**: Processing applications can handle multiple schema versions within the same stream. #### Client application patterns Design client applications for optimal Schema Registry integration: - **Schema caching**: Configure appropriate cache settings to balance performance and memory usage: ```properties # Java client example schema.registry.url=https://psrc-xxx.us-east-1.aws.confluent.cloud basic.auth.credentials.source=USER_INFO schema.registry.basic.auth.user.info=key:secret # Cache up to 1000 schemas max.schemas.per.subject=1000 ``` - **Auto-registration control**: Disable auto-registration in production environments to enforce schema governance: ```properties # Disable auto-registration in production auto.register.schemas=false ``` - **Specific Avro reader**: Use specific readers for better performance and type safety when schema evolution is expected. For detailed client configuration examples, see [Schema Registry client configuration](../sr/schemas-manage.md#sr-prv) and the hands-on [Schema Registry tutorial](../sr/schema_registry_ccloud_tutorial.md#schema-registry-ccloud-tutorial). #### Security and access patterns Implement appropriate security patterns for Schema Registry access: - **API key management**: Use dedicated [service accounts](../security/authenticate/workload-identities/service-accounts/overview.md#service-accounts) for Schema Registry access with minimal required permissions. - **RBAC integration**: Leverage [Schema Registry RBAC](../sr/schemas-manage.md#cloud-sr-rbac) to control schema management permissions at the subject level. - **Environment separation**: Use separate API keys for different environments to prevent accidental cross-environment schema changes. ### Performance and operational considerations #### Schema Registry performance patterns Optimize Schema Registry usage for high-throughput applications: - **Connection pooling**: Reuse HTTP connections to Schema Registry to reduce connection overhead. - **Batch operations**: Use batch APIs when registering multiple schemas or checking compatibility for multiple schema versions. - **Regional proximity**: Use Schema Registry endpoints in the same region as your Kafka clusters to minimize latency. #### Monitoring and alerting Implement comprehensive monitoring for schema-related operations: - **Schema evolution tracking**: Monitor schema registration events and compatibility check failures. - **Client compatibility**: Alert on applications using outdated or incompatible schema versions. - **Data quality monitoring**: Track schema violations and quality rule failures. Example monitoring patterns: ```none Schema Health Monitoring │ ├── Registration Events → Track new schema versions ├── Compatibility Failures → Alert on breaking changes ├── Client Versions → Monitor for outdated clients └── Data Quality → Track schema violations ``` #### Disaster recovery considerations Plan for Schema Registry availability and disaster recovery: - **Multi-region setup**: Schema Registry provides multiple regional endpoints for critical applications requiring high availability. - **Schema backup**: Implement regular backups of schema definitions and configurations for disaster recovery scenarios. See [schema deletion and management](../sr/schemas-delete-cc.md#schemaregistry-deletion) for storage optimization strategies. - **Failover procedures**: Design failover procedures that account for Schema Registry dependencies in your streaming applications. ## Stream processing and AI integration Modern applications increasingly require real-time data processing and AI capabilities. Confluent Cloud provides stream processing options designed for different architectural needs and complexity levels. ### Strategic direction for stream processing Apache Flink is Confluent’s strategic platform for new stream processing applications. It offers the most comprehensive feature set, active development, and future innovation including AI/ML integration. For new architectural decisions, Flink provides the best long-term investment protection with its enterprise-grade capabilities and comprehensive ecosystem. ### Flink integration [Stream Processing with Confluent Cloud for Apache Flink](../flink/overview.md#ccloud-flink) is a fully managed stream processing service that integrates seamlessly with Confluent Cloud, offering enterprise-grade capabilities: - **Advanced analytics**: Complex joins, aggregations, windowing, and pattern detection across multiple streams. - **AI model inference**: Native integration with machine learning models and [AI capabilities](../ai/overview.md#ai-overview). - **Batch and stream unification**: Process both real-time streams and historical data with the same engine. - **Enterprise features**: Advanced state management, exactly-once semantics, and sophisticated fault tolerance. - **Active development**: Receives primary investment for new features and capabilities. ### AI/ML architectural patterns With Flink’s [AI capabilities](../ai/overview.md#ai-overview), you can implement several AI-powered patterns: - **Streaming agents**: Create [AI workflows](../ai/streaming-agents/overview.md#ai-streaming-agents-overview) that can invoke tools and interact with external systems. - **Real-time recommendations**: Process user events and provide immediate personalized responses. - **Anomaly detection**: Monitor data streams and trigger alerts on unusual patterns. - **Content enhancement**: Automatically enrich streaming data with AI-generated insights. ### Design considerations - **Latency requirements**: Flink provides low-latency, complex processing capabilities for demanding real-time applications - **State management**: Consider how to handle stateful operations and failure recovery. - **Scaling strategy**: Plan for auto-scaling based on throughput and processing complexity. - **Cost optimization**: Balance processing power with cost using appropriate cluster types. ### Serverless architectures Serverless architectures rely extensively on either ephemeral functions reacting to events (FaaS or Lambda) or third party services that are exposed only through API calls. Applications require serverless architectures to be elastic, with usage-based pricing and zero operational burden. Confluent Cloud supports this with multiple cluster types designed for different use cases: - **Basic/Standard**: Usage-based pricing ideal for development and lightweight production workloads. - **Enterprise**: Production-ready with enhanced security and [private networking](../networking/overview.md#cloud-networking). - **Dedicated**: Isolated clusters for mission-critical workloads with predictable performance. - **Freight**: Cost-optimized for high-throughput, relaxed latency workloads. For detailed comparisons and specifications, see [Kafka Cluster Types in Confluent Cloud](../clusters/cluster-types.md#cloud-cluster-types). Other than calling an API or configuring the function, there is no user involvement in failure recovery or upgrades. Confluent Cloud is responsible for the availability, reliability, and uptime of your Kafka clusters. Confluent Cloud provides comprehensive serverless offerings for building event-driven applications: ### Core services - **Kafka clusters**: Fully managed, auto-scaling message brokers with usage-based pricing. - **Schema Registry**: Centralized [schema management](../sr/schemas-manage.md#sr-prv) and evolution. See [Data schema architecture and governance](#ccloud-architecture-schema-governance) for comprehensive architectural guidance. - **Connectors**: Simplify data integration by providing [over 120 pre-built connectors](../connectors/overview.md#kafka-connect-cloud). Architecturally, they reduce development overhead by abstracting the complexity of data integration, allowing teams to focus on core business logic instead of building and maintaining data pipelines. ### Stream processing - **Apache Flink**: [Cloud-native stream processing](../flink/overview.md#ccloud-flink) with Flink SQL for complex transformations, joins, and analytics. ### AI/ML integration - **AI model inference**: Integrate LLMs and ML models directly in stream processing workflows. - **Streaming agents**: Build [AI workflows](../ai/streaming-agents/overview.md#ai-streaming-agents-overview) that can invoke tools and interact with external systems. This serverless ecosystem enables you to build complete event-driven architectures: ingest data (extract), transform it in real-time with Flink (transform), and output to various systems (load). ### Function integration patterns Confluent Cloud provides native integration with cloud provider serverless functions for event-driven architectures: - **AWS Lambda integration**: [AWS Lambda Sink](../connectors/cc-aws-lambda-sink.md#cc-aws-lambda-sink) connector supports both synchronous and asynchronous invocation modes. - **Azure Functions integration**: [Azure Functions Sink](../connectors/cc-azure-functions-sink.md#cc-azure-functions-sink) connector for event-driven processing. - **Google Cloud Functions integration**: [Google Cloud Functions Gen 2](../connectors/cc-google-cloud-functions-gen2-sink.md#cc-google-cloud-functions-gen2-sink) connector with improved performance. ### Architectural patterns - **Event processing**: Trigger functions for real-time data validation, enrichment, or transformation. - **External integration**: Invoke APIs, update databases, or send notifications based on Kafka events. - **Microservices orchestration**: Coordinate stateless services through event-driven workflows. ### Design considerations - **Invocation mode**: Choose synchronous for immediate response requirements, asynchronous for fire-and-forget processing. - **Error handling**: Plan for function failures with dead letter queues and retry strategies. - **Cost optimization**: Leverage usage-based pricing for both Kafka and functions to align costs with actual usage. ## Stateless microservices Stateless microservices are a common architectural pattern for applications that use Confluent Cloud. This pattern involves building applications as a collection of distributed, loosely-coupled services. This architectural style is well-suited for the cloud, leveraging the distributed services offered by cloud providers. In microservices architectures with Confluent Cloud, data storage and state management are externalized to specialized services: ### Data layer - **Event store**: Kafka topics serve as the system of record for business events. - **State storage**: Use cloud provider managed database services (RDS, DynamoDB, Cloud SQL) for application state. - **Cache layer**: Redis or cloud-native caching services for frequently accessed data. ### Benefits of stateless design - **Resilience**: Service failures don’t result in data loss since state is externalized. See [cloud resilience patterns](../clusters/resilience.md#cloud-resilience). - **Scalability**: Services can scale independently based on load. - **Deployment**: Rolling updates and blue-green deployments become straightforward. - **Multi-tenancy**: Different tenants can share infrastructure while maintaining data isolation. ### Event-driven communication - Services communicate through Kafka topics rather than direct API calls. - Enables loose coupling and independent service evolution. - Supports event sourcing and CQRS patterns for complex business logic. ## Cloud-native applications Cloud-native applications built with Confluent Cloud are designed from the ground up to leverage cloud infrastructure capabilities and handle cloud environment characteristics. ### Migration patterns Organizations typically follow one of these paths to cloud-native architecture: - **Cloud-First**: New applications built directly on Confluent Cloud with cloud-native patterns. - **Hybrid Bridge**: On-premise applications connected to Confluent Cloud using [Cluster Linking](../multi-cloud/cluster-linking/index.md#cloud-cluster-linking) for gradual migration and data replication. - **Lift and Shift**: Existing applications moved to cloud infrastructure while gradually adopting cloud-native patterns. ### Cloud-native design principles #### Resilience and fault tolerance - Design for ephemeral infrastructure where resources can be terminated unexpectedly. - Implement circuit breakers and [retry policies](client-configs.md#client-producer-consumer-config-recs-cc) with exponential backoff. - Use health checks and graceful degradation when dependencies are unavailable. #### Security and compliance - Implement zero-trust networking with private endpoints where required. - Use short-lived credentials and rotate API keys regularly. - Leverage cloud provider identity services (IAM roles, service principals). - Consider data residency requirements for regulated industries. #### Observability and monitoring - Implement distributed tracing across microservices using correlation IDs. - Use structured logging with correlation IDs for request tracking. - Monitor key metrics: throughput, latency, error rates, and resource utilization. - Set up alerting for business-critical events and system failures. ### Cloud environment considerations The cloud environment presents unique challenges that your applications must handle: - **Dynamic infrastructure**: IP addresses, DNS names, and certificates change regularly. - **Network variability**: Increased latency and packet loss compared to data centers. - **Service dependencies**: Third-party services may have different availability characteristics. - **Cost management**: Usage-based pricing requires monitoring and optimization strategies. ### Best practices for Confluent Cloud applications - **Use supported clients**: Always use current, supported client libraries. See [What client and protocol versions are supported?](../faq.md#cloud-faq-supported-clients) for supported client versions. - **Connection management**: Configure appropriate timeouts and retry settings for reliable [connection lifecycle management](client-configs.md#client-producer-consumer-config-recs-cc): - **Producers**: Set `delivery.timeout.ms` (default 120000ms) to control total time for send operations. - **Consumers**: Configure `session.timeout.ms` to handle network delays and avoid false failures. - **Retry logic**: Use built-in retry mechanisms rather than custom connection handling. - **Metadata refresh**: Set `metadata.max.age.ms` appropriately for cluster changes. - **SNI connectivity**: Validate TLS SNI behavior with [TLS connectivity testing](../networking/testing.md#test-tcp-sni-connectivity). - **Data partitioning**: Design [topic partitioning](../topics/manage.md#topics-num-partitions) strategies that support your scaling requirements. - **Schema evolution**: Plan for schema changes using [Confluent Cloud Schema Registry](../sr/schemas-manage.md#sr-prv) compatibility settings. See [Data schema architecture and governance](#ccloud-architecture-schema-governance) for detailed evolution strategies. - **Multi-region**: Consider [cross-region replication](../multi-cloud/cluster-linking/index.md#cloud-cluster-linking) for disaster recovery and global applications. ## Network and security architecture Network architecture decisions significantly impact your application’s security, performance, and compliance posture. Consider these patterns early in your architecture planning. ### Networking patterns #### Public vs private networking Choose between public and private networking based on security requirements and operational complexity: - **Public networking**: Suitable for development, testing, and applications with standard security requirements. All connections use TLS 1.2+ with API key authentication. - **Private networking**: Required for regulated industries or sensitive data. Eliminates public internet exposure but requires additional network management. For detailed networking options, see [Manage Networking on Confluent Cloud](../networking/overview.md#cloud-networking). #### Private connectivity options When private networking is required, choose the appropriate pattern: - **VPC/VNet Peering**: [Direct network connection](../networking/peering/aws-peering.md#cloud-networking-peering-aws) for simple, dedicated connectivity between specific networks. - **PrivateLink/Private Link / Private Service Connect**: Service-level isolation supporting IP overlap and multiple VPCs connecting to the same cluster. See [AWS PrivateLink](../networking/private-links/aws-privatelink.md#cloud-networking-privatelink-aws), [Azure Private Link](../networking/private-links/azure-privatelink.md#cloud-networking-privatelink-azure), and [Google Cloud Private Service Connect](../networking/gcp-platt.md#cloud-networking-privatelink-gc-esku). - **Transit Gateway**: [Centralized hub-and-spoke connectivity](../networking/aws-transit-gateway.md#aws-transit-gateway) for complex multi-VPC environments. #### Multi-region considerations - **Cross-region access**: Plan for network latency and data transfer costs. - **Data residency**: Consider [regulatory requirements](../get-started/regions.md#providers-regions) for data location and review [cluster creation considerations](../clusters/create-cluster.md#cloud-create-cluster). - **Disaster recovery**: Design [failover patterns](../multi-cloud/cluster-linking/dr-failover.md#cloud-cluster-linking-dr) for business continuity. ### Authentication and access patterns #### Authentication strategies Choose authentication methods based on your environment and integration requirements: - **API keys**: Simple authentication for external applications and services. Use with [service accounts](../security/authenticate/workload-identities/service-accounts/overview.md#service-accounts) for programmatic access. - **OAuth/OIDC**: Enterprise integration with identity providers. Use [identity pools](../security/authenticate/workload-identities/identity-providers/oauth/identity-pools.md#add-oauth-identity-pools) for workload identity federation. - **mTLS**: [Certificate-based authentication](../security/authenticate/workload-identities/identity-providers/mtls/overview.md#mtls-overview) for high-security environments. Provides mutual authentication and enhanced security for Dedicated clusters. #### Access control patterns Implement layered access control using Confluent Cloud’s security mechanisms: - **RBAC**: Use predefined [roles](../security/access-control/rbac/predefined-rbac-roles.md#cloud-rbac-roles) for broad permissions at organization, environment, and cluster levels. - **ACLs**: Implement fine-grained [access control](../security/access-control/acls/overview.md#acl-overview) for specific Kafka resources (topics, consumer groups). - **IP Filtering**: Restrict access to [trusted IP ranges](../security/access-control/ip-filtering/overview.md#ip-filtering-overview) for additional network-level security. #### Service account design patterns - **One service account per application**: Provides maximum granularity for monitoring and access control. - **Shared service accounts**: Consider for applications with identical access patterns and operational requirements. - **Environment separation**: Use separate service accounts for development, staging, and production environments. ### Data protection and encryption patterns #### Client-side field level encryption (CSFLE) [CSFLE](../security/encrypt/csfle/overview.md#csfle-overview) provides the strongest data protection by encrypting sensitive fields before data leaves your applications. This approach ensures that sensitive data is never stored or transmitted in plaintext, even from Confluent Cloud administrators. #### CSFLE architectural patterns - **PII data protection**: Encrypt personally identifiable information (PII) (names, email addresses, social security numbers) while keeping non-sensitive fields searchable and accessible. - **Regulatory compliance**: Meet GDPR, HIPAA, PCI DSS, and other compliance requirements by encrypting regulated data types. - **Zero-trust data architecture**: Implement defense-in-depth where sensitive data remains protected even if infrastructure is compromised. - **Selective decryption**: Enable different consumers to access different levels of sensitive data based on their authorization. #### When to use CSFLE Choose CSFLE for applications that: - Handle highly sensitive data (financial, healthcare, government) - Require compliance with strict data protection regulations - Need to limit access to sensitive fields even from cloud providers - Support multiple consumers with different authorization levels - Process data across untrusted network boundaries #### CSFLE integration considerations - **Key management strategy**: Plan integration with cloud KMS services (AWS KMS, Azure Key Vault, Google Cloud KMS) for encryption key lifecycle. - **Schema design**: Use [Schema Registry](../sr/schemas-manage.md#sr-prv) tags to identify fields requiring encryption and define encryption rules. - **Stream processing integration**: CSFLE works with [Flink](../security/encrypt/csfle/flink-integration.md#csfle-flink-integration) for processing encrypted data streams. - **Performance impact**: Factor in encryption/decryption overhead in throughput and latency planning. #### mTLS architectural patterns [Mutual TLS](../security/authenticate/workload-identities/identity-providers/mtls/overview.md#mtls-overview) provides certificate-based authentication for the highest security environments: - **Certificate-based authentication**: Replace shared secrets with X.509 certificates for client authentication. - **Infrastructure security**: Integrate with existing PKI infrastructure and certificate management systems. - **Zero-trust networking**: Implement mutual authentication where both client and server verify each other’s identity. - **Compliance requirements**: Meet regulations requiring certificate-based authentication. #### When to use mTLS - **High-security environments**: Financial services, government, or highly regulated industries. - **Existing PKI infrastructure**: Organizations already using certificate authorities and client certificates. - **Supported cluster types**: [mTLS is available](../security/authenticate/workload-identities/identity-providers/mtls/configure.md#mtls-requirements) on Dedicated clusters for all cloud providers, and Enterprise and Freight clusters on AWS. - **Machine-to-machine communication**: Service-to-service authentication without shared passwords. #### Authentication method selection guide Choose authentication based on your security requirements: | Authentication Method | Security Level | Complexity | Use Cases | |----------------------------------------------------------------------------------------------------------|------------------|--------------|----------------------------------------------------------------| | API Keys | Standard | Low | External apps, development, basic integration | | OAuth/OIDC | High | Medium | Enterprise SSO, identity federation, user-based access | | [mTLS](../security/authenticate/workload-identities/identity-providers/mtls/configure.md#configure-mtls) | Very High | High | Regulated industries, PKI environments, machine authentication | #### Security architecture decision framework Use this decision tree to select appropriate security measures for your application architecture: ```none Security Requirements Assessment │ ├── Data Sensitivity Level? │ ├── Public/Non-sensitive → Standard encryption (TLS), API Keys │ ├── Internal/Confidential → Add RBAC, Private networking │ └── Highly Sensitive/PII → CSFLE + comprehensive controls │ ├── Compliance Requirements? │ ├── GDPR/CCPA → CSFLE for PII, Audit logging, Data lineage │ ├── HIPAA → CSFLE, BYOK, Private networking, mTLS │ ├── PCI DSS → CSFLE for payment data, Network isolation │ └── SOC 2 → RBAC, Audit logging, Service accounts │ ├── Network Environment? │ ├── Public Internet → Public clusters, IP filtering, TLS │ ├── Corporate Network → VPC Peering, PrivateLink │ └── Air-gapped → Dedicated clusters, Private networking │ ├── Authentication Pattern? │ ├── Human Users → OAuth/OIDC integration │ ├── Applications → Service accounts with API keys │ ├── High Security → mTLS certificates │ └── Mixed → Identity pools with appropriate method per use case │ └── Data Access Patterns? ├── Broad Access → Topic-level ACLs, RBAC roles ├── Field-level Control → CSFLE with selective decryption └── Time-based Access → Temporary credentials, Token rotation ``` #### Implementation priority matrix Implement security controls in order of impact and feasibility: 1. **Foundation** (Required): TLS encryption, service accounts, basic RBAC 2. **Enhancement** (Recommended): Private networking, ACLs, audit logging. 3. **Advanced** (High-security): CSFLE, mTLS, BYOK, advanced monitoring. 4. **Specialized** (Compliance-driven): Schema rules, lineage tracking, custom encryption patterns. ## Multi-tenancy and resource management Multi-tenant architectures allow multiple applications or teams to share Confluent Cloud resources while maintaining isolation and performance guarantees. ### Tenant isolation patterns #### Application-level tenancy Design tenant isolation at the application level: - **Topic-per-tenant**: Separate topics for each tenant providing strong isolation but requiring more management overhead. - **Partition-per-tenant**: Use message keys to route tenant data to specific partitions within shared topics. - **Mixed patterns**: Combine approaches based on tenant size and isolation requirements. #### Principal design for multi-tenancy Use [client quotas](../clusters/client-quotas.md#client-quotas) and principals to manage resource allocation: - **Service account per tenant**: Maximum isolation and individual quota management per tenant. - **Identity pool per business unit**: Group related applications while maintaining quota boundaries. - **Hierarchical principals**: Design principal structure to support both individual and aggregate monitoring. ### Resource management patterns #### Client quota strategies Implement [Client Quotas](../clusters/client-quotas.md#client-quotas) to prevent resource contention: - **Default quotas**: Set cluster-wide defaults to prevent any single tenant from overwhelming shared resources. - **Tenant-specific quotas**: Allocate resources based on business requirements and SLA agreements. - **Performance tiers**: Create different quota levels for different service levels (bronze, silver, gold). #### Monitoring and cost allocation - **Per-tenant metrics**: Use the [Metrics API](../monitoring/metrics-api.md#metrics-api) to track resource consumption by tenant for billing and capacity planning. - **Alerting thresholds**: Set up monitoring for quota utilization and performance anomalies. - **Cost tracking**: Implement chargeback or showback models based on actual resource consumption. ## Observability and monitoring patterns Comprehensive observability is essential for production applications, security compliance, and operational excellence. ### Operational monitoring architecture #### Metrics integration patterns Design metrics collection and analysis workflows: - **Real-time dashboards**: Use the [Metrics API](../monitoring/metrics-api.md#metrics-api) to build operational dashboards showing cluster health, throughput, and consumer lag. - **Capacity planning**: Monitor resource utilization trends to predict scaling needs and optimize costs. - **Performance monitoring**: Track key performance indicators like end-to-end latency, throughput, and error rates. #### Application-level observability - **Distributed tracing**: Implement tracing across your microservices to track requests through event-driven workflows. - **Structured logging**: Use correlation IDs (unique request identifiers) to connect logs across services and trace event processing chains. See [audit log examples](../monitoring/audit-logging/audit-log-records.md#cloud-audit-log-records) for structured JSON logging patterns. - **Circuit breaker patterns**: Monitor and implement circuit breakers for resilient streaming applications. ### Security monitoring and compliance #### Audit logging architecture Implement [audit logging](../monitoring/audit-logging/cloud-audit-log-concepts.md#cloud-audit-logs) for security monitoring and compliance: - **SIEM integration**: Stream audit logs to security information and event management systems for real-time threat detection. - **Compliance reporting**: Automate compliance report generation for regulatory requirements (SOX, GDPR, HIPAA). - **Incident response**: Design audit log queries and playbooks for security incident investigation. #### Security monitoring patterns - **Access monitoring**: Track authentication patterns, failed login attempts, and privilege escalation. - **Behavioral analysis**: Monitor for unusual access patterns, cross-region activity, and resource access anomalies. - **Compliance automation**: Implement automated checking for security policy violations and configuration drift. For detailed audit logging best practices, see [Best Practices for Audit Logs on Confluent Cloud](../monitoring/audit-logging/best-practices.md#best-practices-cloud-audit-logs).