Use self-managed encryption keys with Tableflow¶
Tableflow supports self-managed encryption keys (also known as BYOK) to enable organizations in sensitive industries to take advantage of Tableflow while meeting compliance requirements for data-at-rest encryption. With self-managed encryption key support, Tableflow can read from clusters with self-managed encryption keys and write to storage options with self-managed encryption keys, including both Confluent Managed Storage and Bring Your Own Storage (BYOS). Choose one storage configuration based on your needs; encryption key requirements and setup steps differ.
This section covers encryption behavior, limitations, and validation procedures for Tableflow with self-managed encryption keys.
Confluent Managed Storage encryption¶
When you use Confluent Managed Storage with Tableflow, the self-managed encryption key used for your Kafka cluster is automatically reused for Tableflow Confluent Managed Storage. This ensures consistent encryption across your Kafka data and Tableflow tables.
For an overview of storage behavior and capabilities, see Confluent Managed Storage.
Important
- The encryption key cannot be changed separately for Tableflow Confluent Managed Storage after it’s configured.
- Key rotations impact both Kafka data and Tableflow data stored in Confluent Managed Storage.
- If you revoke access to the encryption key, both Kafka and Tableflow data in Confluent Managed Storage become inaccessible.
User-managed Amazon S3 storage encryption¶
Organizations that want self-managed encryption keys with Tableflow typically use their own Amazon S3 buckets with AWS KMS encryption. This approach provides full control over both the storage infrastructure and encryption keys, ensuring maximum compliance for sensitive industries.
This corresponds to the Bring Your Own Storage (BYOS) option for Tableflow. You can choose between two AWS encryption modes:
- SSE-KMS (Server-Side Encryption with KMS): Standard single-layer encryption using AWS KMS keys.
- DSSE-KMS (Dual-layer Server-Side Encryption with KMS): Enhanced security with two independent layers of encryption for highly sensitive data.
For detailed information on these encryption modes, see SSE-KMS and DSSE-KMS in the AWS documentation. For setup steps and required IAM and KMS policies, see Configure Storage for Tableflow.
To configure self-managed encryption keys with user-managed S3 buckets, you must add the required KMS permissions to the KMS key used to encrypt your S3 bucket. For more information about configuring KMS permissions with provider integrations, see Create an AWS Provider Integration in Confluent Cloud.
Important
For multi-account setups, you must update both the IAM role permissions and the KMS key policy to grant the necessary KMS actions to the Confluent Cloud role.
Self-managed encryption key workflows for Tableflow¶
Tableflow supports two workflows when using self-managed encryption keys:
Confluent Managed Storage with self-managed encryption keys¶
When your Kafka cluster uses self-managed encryption keys, Tableflow with Confluent Managed Storage automatically inherits the same encryption key with no additional configuration required:
- Navigate to your Kafka topic in the Confluent Cloud Console.
- Click Enable Tableflow.
- Select Confluent Managed Storage as the destination storage.
- Complete the Tableflow setup.
All data-at-rest is automatically encrypted using the self-managed encryption key associated with your Kafka cluster.
User-managed storage with self-managed encryption¶
To set up storage with your own Amazon S3 bucket (SSE-KMS or DSSE-KMS) and apply the required IAM and KMS policies, follow the steps in Configure Storage for Tableflow. That guide covers BYOS setup and includes the KMS key policy update. This page focuses on behavior, limitations, and validation.
Validation and error handling¶
Tableflow performs validation to ensure proper access to encryption keys:
Key access validation¶
During setup, Tableflow validates that it can successfully read from and write to encrypted storage using the configured encryption keys. If validation fails, you receive error messages and should take the following actions:
- Missing KMS permissions: Verify that your IAM role has the required KMS
actions (
kms:Encrypt,kms:Decrypt,kms:ReEncrypt*,kms:GenerateDataKey*,kms:DescribeKey) and that the KMS key policy allows access from your IAM role. - Invalid or disabled encryption keys: Check that the KMS key exists, is enabled, and is in the correct AWS region.
- Network connectivity issues: Ensure your AWS account has access to KMS services and that no network policies are blocking the connection.
Security considerations for Tableflow¶
- Data caching and key revocation: Confluent internal systems may locally cache decrypted data and keys for up to six hours for performance and cost efficiency. Confluent proactively evicts cached data and keys on a best-effort basis when key revocation is detected.
- Data in transit: All data transfers use TLS 1.2 encryption by default when Tableflow writes to destination storage.
- Key rotation: Automatic key rotation is supported and applied to both new and existing Tableflow data.
- Catalog integration: When using catalogs with Tableflow and self-managed keys, you must grant the catalog access to the KMS key that encrypts the Tableflow data. This applies to both Confluent Managed Storage and Bring Your Own Storage (BYOS). For detailed instructions, see Integrate Catalogs with Tableflow in Confluent Cloud.
- Compliance: Tableflow with self-managed encryption keys enables organizations in sensitive and regulated industries to maintain compliance requirements for data-at-rest encryption while leveraging the benefits of open table formats and real-time data processing.