Process Encrypted Data with Confluent Cloud for Apache Flink

Confluent Cloud for Apache Flink® supports processing data encrypted with client-side field level encryption and client-side payload encryption. This enables you to work with sensitive data while maintaining end-to-end encryption, ensuring that your data is protected even if the server is compromised.

Diagram showing how client-side encrypted data flows through processing while maintaining end-to-end encryption

Note

Client-side field level encryption (CSFLE) and client-side payload encryption (CSPE) with Flink is a Limited Availability feature in Confluent Cloud.

For limitations during the Limited Availability phase, see Known limitations and caveats.

If you would like to participate in the Limited Availability Program, contact your Confluent account team.

Confluent Cloud for Apache Flink® supports two modes for working with data encrypted using client-side field level encryption or client-side payload encryption:

Transparent decryption

Flink automatically decrypts encrypted fields for processing when you have the required permissions. The decrypted data is available for all SQL operations. This mode requires that you share Key Encryption Keys with Confluent and grant permissions to access them.

Deterministic encryption

Flink processes encrypted data without decrypting it. This mode works with deterministic encryption only and supports limited operations like equality comparisons, grouping, and joins on encrypted values.

The mode you use depends on your security requirements, the operations you need to perform, and whether you share encryption keys with Confluent.

Known limitations and caveats

The following limitations and caveats apply to processing encrypted data with Flink during the Limited Availability phase. Review them before you create statements that read or write CSFLE- or CSPE-encrypted data. For background on how Flink processes encrypted data, see Transparent decryption mode and Deterministic encryption mode.

Schema and encryption rules are fixed at statement creation

Flink captures the full schema from Schema Registry, including encryption rules, when a Flink SQL statement is created. The statement uses this captured schema and these rules for its entire lifetime, which has the following consequences:

  • Schema evolution is not supported for running statements. If you evolve the schema in Schema Registry after the statement is running, such as by adding, removing, or changing fields, the running statement does not pick up the changes. It continues to use the schema that was captured when the statement was created.

  • Encryption rule changes are not supported for running statements. If you change CSFLE encryption rules in Schema Registry after the statement is running, the running statement does not pick up the changes. It silently continues to use the encryption rules that were captured when the statement was created.

  • You must re-create statements to use updated schemas or rules. To pick up an updated schema or updated encryption rules, delete and re-create the Flink SQL statement.

  • Topics can contain a mix of old and new records after a change. After a schema or encryption rule change, a topic can contain records written with both the previous and the updated schema or encryption configuration. A newly created statement uses only the current schema and rules and might not correctly process records written with earlier schemas or rules. Where possible, use a new topic or allow older records to expire through topic retention before you create the new statement.

If encryption rule evolution is enabled for your organization, statements that you create after it’s enabled pick up encryption rule changes when they read data. They decrypt each record with the rules that applied when the producer wrote it. For statements whose principal has access to the encryption keys, the caveat about a mix of old and new records doesn’t apply to encryption rule changes. Schema changes, such as adding, removing, or changing fields, still require re-creating the statement. For more information, see Encryption rule evolution for running statements.

Encryption rule evolution for running statements

When encryption rule evolution is enabled, Flink SQL statements that read encrypted data pick up encryption rule changes in Schema Registry without being re-created. For each record, Flink applies the encryption rules, metadata, and tags from the schema version that the producer used to write the record, instead of only the rules that Flink captured when you created the statement.

Encryption rule evolution applies only to statements that you create after it’s enabled for your organization. Statements that were already running keep the default behavior until you re-create them.

Note

Encryption rule evolution is a Limited Availability feature in Confluent Cloud.

If you would like to participate in the Limited Availability Program, contact Confluent Support or your Confluent account team to enable encryption rule evolution for your organization.

Running statements pick up the following changes when they read data:

  • Adding or removing a CSFLE or CSPE encryption rule.

  • Switching a rule to a different Key Encryption Key (KEK) by changing the encrypt.kek.name parameter. Rotating key material in your key management service (KMS) without changing the rule isn’t a rule change. For that procedure, see Rotate a KEK.

  • Adding or removing tags on fields, either inline in the schema, such as confluent:tags in Avro and JSON Schema or (confluent.field_meta).tags in Protobuf, or through tag patterns in the schema metadata.

  • Replacing a CSFLE rule with a CSPE rule, or a CSPE rule with a CSFLE rule.

Rule evolution applies to tags stored in the schema version: inline tags and metadata tag patterns. Tags that you add through the Catalog API take effect only after a new schema version includes them.

Flink still decrypts records written before a change with the rules that applied when the producer wrote them, so a topic can contain a mix of records written under old and new rules. A rule change, including switching a rule to a different KEK, affects only records that producers write with the new schema version. It doesn’t re-encrypt existing records. Producers that use an earlier schema version, for example because they cache it, keep writing with the earlier rules until they switch to the new version. Keep access to the earlier KEK as long as the topic contains records encrypted with it, even after producers switch, because Flink needs it to decrypt those records.

Encryption rule evolution works with the Avro, JSON Schema, and Protobuf formats.

Encryption rule evolution has the following limitations:

  • Writes don’t pick up rule changes. You must re-create a statement to write with updated rules. See Write with updated rules.

  • Schema evolution isn’t supported. Rule evolution applies only to encryption rules, metadata, and tags, including inline tags on existing fields. If you add, remove, or change the fields themselves, such as their names or types, you must re-create the statement.

  • Granting key access to a running statement starts decryption. This can write plaintext to an output topic that lacks matching rules. See Grant key access to running statements.

Write with updated rules

Only reads pick up rule changes. When a statement writes to an output topic, for example with INSERT INTO, it continues to use the output topic’s encryption rules that Flink captured when you created the statement. If the new or changed output rule references a KEK that the Flink principal can’t use yet, complete the steps in Prepare keys before you change rules before you replace the statement. Otherwise, the new statement fails, because Flink can’t encrypt the output without key access. If the KEK already encrypts data that running statements read, first follow Grant key access to running statements. To write data with updated rules, replace the statement:

  • For a stateless background statement that doesn’t write to an upsert sink, create the new statement with carry-over offsets that reference the running statement, then stop the running statement. Delete it only after the new statement starts, to avoid writing duplicate records. For more information, including which statements support carry-over offsets, see Carry-over Offsets in Confluent Cloud for Apache Flink.

  • For other statements, delete and re-create the statement.

Warning

After you add an encryption rule or tag a new field in an output topic’s schema, re-create the statements that write to it. Before you tag a new field in an input topic, tag the corresponding output field, make sure that an encryption rule covers that tag, and re-create the statements. Otherwise, running statements write the field as plaintext. If other statements read that output topic, repeat these steps for them, starting at the end of the pipeline.

Prepare keys before you change rules

For statements that decrypt or encrypt data, before you add a rule on a topic, replace a CSFLE rule with a CSPE rule, or switch a rule to a different KEK, make sure that the Flink principal can use the KEK that the rule references:

  1. Register the KEK in Schema Registry. If you let the client register the KEK from the rule’s encrypt.kms.type and encrypt.kms.key.id parameters instead, the KEK doesn’t exist until a producer first writes with the new rule, and the client doesn’t share it with Confluent automatically.

  2. Share the KEK with Confluent.

  3. Grant DeveloperRead on the KEK to the Flink principal.

  4. Confirm that the KMS key policy grants Confluent Cloud the permissions it needs to use the KEK.

Otherwise, Flink can’t decrypt records written with the new rule. Fields that a CSFLE rule encrypts pass through as ciphertext. A statement fails when it reads records that a CSPE rule encrypts, because it can’t open the payload to find its fields. A statement that writes with the new rule also fails, because Flink can’t encrypt the output without key access. For the risks of ciphertext passthrough, see Encrypted passthrough without key access.

If the KEK already encrypts data that a running statement reads, these steps also make the statement start decrypting that data. Before you share the KEK or grant DeveloperRead on it, follow Grant key access to running statements.

Sharing a KEK takes effect at the next Data Encryption Key refresh, which can take up to five minutes. After you share a KEK, wait at least five minutes before you change a rule to reference it.

If you use Deterministic encryption mode, rule changes affect results on the encrypted field. Switching a rule to a different KEK produces different ciphertext for the same value. Tagging a new field leaves older records with that field in plaintext, and removing a rule or a field’s tag leaves newer records with that field in plaintext. In all these cases, equality filters in WHERE clauses, GROUP BY, COUNT, and joins on the field return inaccurate results across records written under the old and new rules. For example, a filter that compares the field with a ciphertext value stops matching records written with the new KEK. If you add a CSPE rule, or replace a CSFLE rule with a CSPE rule, a deterministic-mode statement fails when it reads records written with that rule, because it can’t open the payload without key access.

Grant key access to running statements

If the Flink principal of a running statement gains access to a KEK, for example because you share the KEK with Confluent and grant DeveloperRead on it, the statement starts decrypting records encrypted with that KEK. If the output topic’s encryption rules that Flink captured when you created the statement don’t cover the newly decrypted fields, the statement writes those fields as plaintext. For more information, see Decrypted output without encryption rules.

Before you share a KEK with Confluent, or grant DeveloperRead on an already shared KEK to a running statement’s principal, check the output topics of statements that read data encrypted with that KEK. Running statements don’t pick up rules that you add to an output topic later. If a statement’s output topic had no encryption rules covering every field encrypted with that KEK when you created the statement, stop the statement before you give it key access. After you share the KEK and grant DeveloperRead on it, add the missing output rules and re-create the statement. Otherwise, the running statement starts writing the decrypted fields as plaintext to its output topic. To avoid writing duplicate records from a stateless background statement that doesn’t write to an upsert sink, create the new statement with carry-over offsets that reference the stopped statement. For more information, including which statements support carry-over offsets, see Carry-over Offsets in Confluent Cloud for Apache Flink.

Adding these output rules also tags the fields on the input of any statement that reads the output topic. Check those statements the same way, starting with the topics at the end of the pipeline and working back toward the source topic.

Gaining key access also affects statements that use Deterministic encryption mode. Equality filters in WHERE clauses, GROUP BY, COUNT, and joins on an encrypted field rely on every value of that field staying ciphertext. After the principal gains access to the KEK, new records arrive with the field decrypted, so the statement treats the plaintext and ciphertext forms of the same value as different values. For example, a count grouped by the field splits one group into two, and a filter that compares the field with a ciphertext value stops matching new records.

To avoid mixed results, stop the deterministic-mode statements that operate on fields encrypted with a KEK before you share that KEK with Confluent or grant DeveloperRead on it to their principal. After you give access, add encryption rules that cover those fields to the statements’ output topics. Rewrite filters that compare a field with a ciphertext value to compare it with the plaintext value, and then re-create the statements. For a stateless background statement that doesn’t write to an upsert sink, such as one that only filters, create the new statement with carry-over offsets that reference the stopped statement. Statements that group, count, or join are stateful, so carry-over offsets don’t apply, and a re-created statement writes its earlier results to the output topic again. Consider writing to a new output topic.

A re-created statement handles each record according to the record’s encryption rule. It decrypts records whose rule references a KEK that it can use. For records whose CSFLE rule references a KEK that it can’t use, it returns ciphertext. It fails when it reads records that a CSPE rule encrypts with such a KEK. Results on the encrypted field remain inaccurate if a topic mixes CSFLE records encrypted with a KEK that the statement can’t use and records where the field is plaintext. The field is plaintext either because the statement decrypted it or because the producer never encrypted it.

Data Encryption Key refresh

Data Encryption Keys refresh every five minutes. If you revoke permissions on a Key Encryption Key (KEK), running statements continue to decrypt data for up to five minutes.

Encrypted data stored in state

When you process CSFLE or CSPE encrypted data without decryption permission, the encrypted ciphertext is stored in Flink state. This encrypted data cannot be decrypted in the output topic unless you preserve the encryption rules in the output schema.

Decrypted output without encryption rules

If you have key access and decrypt input data but do not configure encryption rules for the output topic, Flink writes decrypted plaintext data to the output topic. Flink logs a warning but does not block the statement. Ensure your output topics have appropriate encryption rules.

Diagram showing input data being decrypted during processing and written as plaintext output when the output topic has no encryption rules

Encrypted passthrough without key access

If Flink runs without key access, encrypted fields pass through as raw ciphertext. The output can contain encrypted bytes without the encryption metadata needed to decrypt them. This is true for ciphertext with no rules and no encryption in output, and for encrypted rule-less ciphertext with encryption in output. In both cases, unusable ciphertext is written to the output topic.

Warning

This scenario can cause permanent data loss. If Flink writes encrypted bytes without the metadata needed to decrypt them, downstream consumers can never recover the original data.

Diagram showing encrypted data passing through without key access, producing output bytes that lack the metadata needed to decrypt

Tableflow and Flink snapshot query

Snapshot queries with union reads from Kafka and Tableflow can result in different encryption states for the same data. If you read from an encrypted Kafka topic and a plaintext Tableflow materialized table, the same field is encrypted in one source and plaintext in the other. Queries return a mix of ciphertext and plaintext for the same column.

Behavior for unauthorized users

If you process CSFLE-encrypted fields in Flink SQL statements without DeveloperRead permission on the encryption keys, the encrypted fields remain encrypted in the output. But you cannot decrypt these fields in the output topic because the encryption rules are not preserved.

This limitation affects users who:

  • Process CSFLE-encrypted data without key permissions

  • Want to decrypt the output data in downstream consumers

This is a known limitation that is not addressed in the Limited Availability release. To work with encrypted data and keep decryption capability in output topics, you must have DeveloperRead permission on the encryption keys.

Transparent decryption mode

Transparent decryption mode automatically decrypts encrypted fields for processing when you share Key Encryption Keys with Confluent and have the required permissions. This lets you perform any SQL operation on encrypted data as if it were plaintext, while maintaining end-to-end encryption.

Prerequisites

To use transparent decryption, you must:

  • Share Key Encryption Keys with Confluent when creating encryption rules

  • Grant permission on the encryption keys to the Flink principal: DeveloperRead is enough for both reading (decryption) and writing (encryption) data.

  • Configure encryption rules in Schema Registry for both input and output topics

For more information about permissions, see Data access layer.

How transparent decryption works

When a Flink SQL statement starts, Flink evaluates the permissions you have on the encryption keys referenced in the statement. To make a statement decrypt data that it couldn’t decrypt when you created it, grant access and then re-create the statement.

If encryption rule evolution is enabled, a statement that you create after it’s enabled picks up key access that you grant after the statement starts, such as DeveloperRead on a KEK that you switch a rule to, or a KEK that you share with Confluent later. The statement doesn’t need to be re-created, and it starts decrypting records encrypted with that KEK. A KEK that you share takes effect within five minutes, when Data Encryption Keys refresh. If the output topic’s encryption rules that Flink captured when you created the statement don’t cover the newly decrypted fields, the statement writes them as plaintext. For more information, see Encryption rule evolution for running statements.

If you have DeveloperRead permission on an encryption key, Flink transparently decrypts fields encrypted with that key. The decrypted data is available for all SQL operations including filtering, aggregation, joins, and transformations.

The decryption happens within the Flink processing environment. Output data is encrypted again if you have configured encryption rules for the output topic.

Permission-based behavior

The behavior of your Flink SQL statement depends on your permissions and the encryption type:

Input encryption

Output encryption

Permission

Behavior

CSFLE or CSPE

CSFLE or CSPE

Authorized

Input is decrypted for processing. Output is encrypted.

CSFLE or CSPE

None

Authorized

Input is decrypted for processing. Output is not encrypted. Confluent Cloud logs a warning about unencrypted output.

CSFLE

None

Unauthorized

Input remains encrypted and passes through without decryption. Output retains encrypted values. This can cause permanent data loss; see Encrypted passthrough without key access.

CSPE

Any

Unauthorized

The statement fails because Flink cannot decrypt the payload.

CSFLE

CSFLE or CSPE

Unauthorized

The statement fails because you cannot encrypt the output without key access.

None

CSFLE or CSPE

Authorized

Input is processed normally. Output is encrypted.

None

CSFLE or CSPE

Unauthorized

The statement fails because you cannot encrypt the output without key access.

In this table:

  • Authorized means you have DeveloperRead permission on the Key Encryption Keys referenced in the encryption rules

  • Unauthorized means you do not have DeveloperRead permission on the encryption keys

If encryption rule evolution is enabled, the behavior of statements that you create after it’s enabled can change while they run, because it depends on each record’s encryption rule and on your current key access. For example, a running statement starts decrypting records encrypted with a KEK when you grant its principal access to that KEK. For more information, see Encryption rule evolution for running statements.

Examples

Decrypt and process encrypted data

This example shows processing data where both input and output topics use CSFLE encryption. You have DeveloperRead permission on the encryption keys.

-- Create a table for encrypted customer data
CREATE TABLE customers (
  customer_id STRING,
  email STRING,  -- Encrypted with CSFLE
  ssn STRING,    -- Encrypted with CSFLE
  address STRING -- Encrypted with CSFLE
) WITH (
  'kafka.topic' = 'customers-encrypted',
  'value.format' = 'avro-registry'
);

-- Create output table with encryption rules
CREATE TABLE customer_summary (
  customer_id STRING,
  email STRING,       -- Will be encrypted with CSFLE
  location_count INT  -- Not encrypted
) WITH (
  'kafka.topic' = 'customer-summary-encrypted',
  'value.format' = 'avro-registry'
);

-- Query decrypts input, processes it, and encrypts output
INSERT INTO customer_summary
SELECT
  customer_id,
  email,
  COUNT(DISTINCT address) AS location_count
FROM customers
GROUP BY customer_id, email;

In this example:

  • Flink decrypts the email, ssn, and address fields for processing

  • The SQL operations work with plaintext values

  • The email field in the output is encrypted according to the output encryption rules

  • The location_count field is not encrypted

Process encrypted data without permission

This example shows what happens when you try to process CSFLE-encrypted data without permission. The encrypted data passes through without decryption.

-- Same input table as previous example; output table has no encryption rules
-- User does NOT have DeveloperRead on encryption keys

INSERT INTO customer_summary
SELECT
  customer_id,
  email,
  COUNT(DISTINCT address) AS location_count
FROM customers
GROUP BY customer_id, email;

In this example:

  • Flink does not decrypt the encrypted fields

  • The email and address fields contain encrypted ciphertext

  • The COUNT(DISTINCT address) operation counts distinct ciphertext values, not distinct addresses

  • The output contains encrypted values, but no encryption rules are applied

This scenario can produce unexpected results. See Permission-based behavior for details.

Deterministic encryption mode

Deterministic encryption mode allows Flink to process encrypted data without decrypting it. This mode works by using deterministic encryption, where encrypting the same plaintext value with the same key always produces the same ciphertext.

This mode does not require sharing encryption keys with Confluent. However, it supports only limited operations.

What is deterministic encryption?

At the core of this capability is deterministic encryption, a method where encrypting the same plaintext value with the same key always produces the exact same ciphertext. This property allows Flink to process the encrypted ciphertext directly, effectively performing equality comparisons, joins, and groupings on the original data without ever needing to decrypt it.

Supported operations on encrypted data

Flink does not perform decryption. It operates on raw bytes, which allows it to process the ciphertext produced by CSFLE. Because the encryption is deterministic, you can:

  • Process non-encrypted fields in your data stream without any limitations

  • Run SQL queries that operate directly on your encrypted fields

Here are the key operations possible on deterministically encrypted columns:

Filtering and equality

Use encrypted fields in WHERE clauses for exact matches.

Grouping and aggregation

Perform GROUP BY operations on encrypted fields. The only aggregation functions that work correctly are those based on uniqueness comparison, such as COUNT and COUNT(DISTINCT).

Joins

Join multiple streams together using an encrypted column. For example, join a stream of user activity to a stream of user profiles on an encrypted user ID.

Window functions

Use comparison-based window functions like LEAD and LAG.

Example

This example counts the number of active users, grouping by the deterministically encrypted email field:

SELECT COUNT(DISTINCT email_encrypted)
FROM users_stream
WHERE status = 'ACTIVE';

This example shows a common use case where the encrypted email_encrypted field can be grouped or filtered without being decrypted, leveraging deterministic encryption.

Important limitations and trade-offs

This capability comes with two important considerations:

  1. Limited aggregation functions: Because the data’s actual value is never revealed to Flink, mathematical operations do not produce correct results. Aggregation functions like SUM, AVG, MIN, and MAX execute but yield erroneous values.

  2. The deterministic trade-off: Deterministic encryption inherently reveals when two encrypted values are identical. This is a necessary trade-off that enables querying, but it’s a piece of information that can be analyzed. You should carefully consider this when deciding which fields to encrypt deterministically.

How deterministic encryption protects data

When you use CSFLE with deterministic encryption in Flink, the security of your data is maintained because the actual decryption only happens when the data is read from a sink like a database or materialized view by an authorized client application that holds the decryption keys. Flink processes the data without ever having access to the plaintext. This ensures that sensitive data cannot be exposed even in the event of a compromise within the processing environment.

CSPE behavior with Flink

Client-side payload encryption (CSPE) encrypts the entire record payload before it is written to Kafka. From the Flink perspective, CSPE follows the same permission model as CSFLE but with stricter behavior for unauthorized users:

  • When the Flink principal has DeveloperRead permission on the Key Encryption Key (KEK), Flink transparently decrypts the payload for processing and re-encrypts records on write according to the CSPE rules configured on the output topic.

  • When the Flink principal does not have DeveloperRead on the KEK, any Flink SQL statement that reads from a CSPE-encrypted topic fails because Flink can’t decrypt the payload.

  • Unlike CSFLE, there is no ciphertext pass-through mode for CSPE: unauthorized users can’t run queries that read from CSPE-encrypted topics.

For the full permission-based behavior matrix and examples, see Permission-based behavior.

Troubleshoot encryption errors

A Flink SQL statement that writes to a topic with encryption rules fails when it can’t access the Key Encryption Key (KEK) that a rule references, because it can’t encrypt the output. These errors are subcategories of error code 2400 (SerializationEncryptionError), and they stop the statement.

The following table lists the causes.

Error

What triggers it

How to fix it

2410 SerializationEncryptionKeyNotShared

The KEK isn’t shared with Confluent, so Schema Registry denied the principal that runs the statement access to the KEK.

Share the KEK with Confluent. Set shared=true for the KEK in Schema Registry, or enable Share key access with Confluent Cloud for the key in Confluent Cloud Console. Then grant the Confluent Cloud Data Encryption Key (DEK) Registry access to the key in your key management service (KMS), as described for 2430. For more information, see Add an encryption key.

2420 SerializationEncryptionKekRbacAccessDenied

The principal that runs the statement doesn’t have the RBAC role required to read the KEK.

Grant the principal the DeveloperRead role on the KEK. DeveloperRead is enough to encrypt data.

2430 SerializationEncryptionDekGenerationForbidden

Schema Registry couldn’t generate a DEK for the KEK because the permissions in your KMS don’t grant the Confluent Cloud DEK Registry access to the KEK. Schema Registry generates DEKs itself and uses the KEK in your KMS to encrypt and decrypt them.

Update the key policy or IAM permissions in your KMS so that they grant the Confluent Cloud DEK Registry access to the KEK, as shown in Confluent Cloud Console for the key. For more information, see Add an encryption key.

A statement that fails with one of these errors stays in the FAILED state after you fix the cause. To continue processing, resume the statement. If it fails again with the same error, re-create it. For more information, see Error handling and recovery.

Uses Google Tink

The CSFLE implementation uses the open-source Google Tink Cryptographic library to perform deterministic encryption using the AES256_SIV algorithm. For more information on Google Tink, see the following: