# Use Client-Side Field Level Encryption on Confluent Platform Client-side field level encryption (CSFLE) uses a technique called [envelope encryption](../../../_glossary.md#term-envelope-encryption), where a [key encryption key (KEK)](../../../_glossary.md#term-key-encryption-key-KEK) is used to encrypt [data encryption keys (DEKs)](../../../_glossary.md#term-data-encryption-key-DEK), which are the actual keys used to encrypt fields. The KEK is typically a key from an external Key Management System (KMS), such as AWS KMS, Azure Key Vault, or Google Cloud KMS. [Schema Registry](../../../_glossary.md#term-Schema-Registry) exposes a subcomponent called the DEK Registry, which provides APIs for managing KEKs and DEKs. The key from the KMS is registered as a KEK to the DEK Registry, and the DEK Registry will also hold the encrypted DEKs used for CSFLE. DEKs are scoped by subject. In Confluent Platform, CSFLE is fully client-managed. The client generates each DEK, uses the KEK in your KMS to encrypt the DEK, and stores the encrypted DEK in the DEK Registry. Each producer and consumer must have RBAC permission to the KEK and the appropriate KMS permissions to encrypt and decrypt DEKs. The DEK Registry also provides a `shared` KEK option, in which the DEK Registry makes direct calls to the KMS to encrypt and decrypt DEKs on behalf of clients. This option is intended for Confluent Cloud, where it lets Confluent-managed services process CSFLE-encrypted data. Sharing KEK access is not supported in Confluent Platform; use the client-managed approach described in this topic. At a high level, CSFLE requires the following steps: 1. Create a KEK in an external KMS. 2. Register the KEK to the DEK Registry. 3. Specify the tags on fields. 4. Declare encryption rules that specify which tags should be encrypted. 5. Configure the client to access the KMS. 6. Produce and consume data. ## Requirements To use client-side field level encryption (CSFLE) on Confluent Platform, the following requirements must be met. ### Confluent Platform * Confluent Enterprise 8.0 or later with the CSFLE Add-On enabled. * To use CSFLE, ensure that the following Schema Registry properties are configured before starting Schema Registry: ```text resource.extension.class=io.confluent.kafka.schemaregistry.rulehandler.RuleSetResourceExtension,io.confluent.dekregistry.DekRegistryResourceExtension confluent.license= confluent.license.addon.csfle= ``` Note that your Confluent Enterprise license key is used for the CSFLE Add-On license. ### Clients Supported clients for use with CSFLE include: #### Confluent Enterprise or Apache Kafka® Java Client Client-Side Field Level Encryption (CSFLE) enables robust data protection for sensitive fields within your Kafka topics. To use CSFLE on Confluent Platform, you must have a Confluent Enterprise license with the CSFLE Add-On enabled, which requires Confluent Enterprise 8.0 or later. Your client applications must use a version of the Apache Kafka® Java Client libraries that supports CSFLE. CSFLE relies on Confluent Schema Registry, including its DEK Registry component, for key management and serializer integration. For details about supported client versions, see [Apache Kafka Java Client](https://docs.confluent.io/kafka-clients/java/current/overview.html). #### Confluent Go Client * Requires version 2.5.0 or later * For details, see [Confluent Golang Client for Apache Kafka](https://github.com/confluentinc/confluent-kafka-go). * For examples, see [confluent-kafka-go/examples](https://github.com/confluentinc/confluent-kafka-go/tree/master/examples). #### Confluent .NET Client * Requires version 2.5.0 or later * For details, see [Confluent .NET Client for Apache Kafka](https://github.com/confluentinc/confluent-kafka-dotnet). * For examples, see [confluent-kafka-dotnet/examples](https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/examples). #### Confluent JavaScript Client for Schema Registry * Requires version 0.2.1 or later. * For details, see [Confluent JavaScript Client](https://docs.confluent.io/kafka-clients/javascript/current/overview.html). * GitHub repository: [confluent-kafka-javascript](https://github.com/confluentinc/confluent-kafka-javascript). #### Confluent Python Client for Kafka * Requires version 2.8.0 or later. * For details, see [Confluent Python Client for Apache Kafka](https://docs.confluent.io/kafka-clients/python/current/overview.html). * GitHub repository: [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python). * For examples, see [confluent-kafka-python/examples](https://github.com/confluentinc/confluent-kafka-python/tree/master/examples). * Confluent Developer course: [Python Client for Apache Kafka](https://developer.confluent.io/learn/courses/kafka-python/). ### Kafka serializers and deserializers CSFLE works with the latest versions of the existing Java [serializers and deserializers](../../../schema-registry/fundamentals/serdes-develop/index.md#serializer-and-formatter) for: * [Avro](../../../_glossary.md#term-Avro). For more information, see [Avro Schema Serializer and Deserializer for Schema Registry on Confluent Platform](../../../schema-registry/fundamentals/serdes-develop/serdes-avro.md#serdes-and-formatter-avro). * [JSON Schema](../../../_glossary.md#term-JSON-Schema). For more information, see [JSON Schema Serializer and Deserializer for Schema Registry on Confluent Platform](../../../schema-registry/fundamentals/serdes-develop/serdes-json.md#serdes-and-formatter-json). * [Protobuf](../../../_glossary.md#term-Protobuf). For more information, see [Protobuf Schema Serializer and Deserializer for Schema Registry on Confluent Platform](../../../schema-registry/fundamentals/serdes-develop/serdes-protobuf.md#serdes-and-formatter-protobuf). ### Key management service (KMS) CSFLE supports the following key management services (KMS): * [AWS Key Management Service (AWS KMS)](https://aws.amazon.com/kms/) * [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault/) * [Google Cloud Key Management Service (Cloud KMS)](https://cloud.google.com/security-key-management) * [Hashicorp Vault Transit Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/transit). * Local key (for testing only) ### Confluent Platform resources For Confluent Platform resources that are supported for CSFLE, see [Use CSFLE for Confluent Enterprise](overview.md#csfle-options). ## Configure client-side field level encryption The following steps show how to configure client-side field level encryption for use with Confluent Platform Enterprise. ### Step 1: Create a KEK in your key management service (KMS) To start using CSFLE, the first step is to create a KEK using your key management service (KMS). For the steps to create a KEK, see [Create a KEK](manage-keys.md#create-kek-csfle). ### Step 2: Register the KEK with the DEK Registry After you have a KEK, you can register it with the DEK Registry using Confluent CLI or REST APIs. Note that if the KEK is not registered beforehand, you can register it on demand by the client, assuming that the client has the appropriate permissions with the DEK Registry. For the steps to register a KEK with the DEK Registry, see [Register a KEK with the DEK Registry](manage-keys.md#register-kek-csfle). ### Step 3: Add tags to the schema fields Tags are metadata attributes that you can add to schema fields. You can use tags to specify which fields are encrypted. These tags are used to define the encryption policies that specify which fields in the data schema need to be encrypted and how they should be handled. Tags can either be inline or external. Here is an example of how tags can be added to schema fields in a Avro schema: ```json { "type":"record", "name":"MyRecord", "fields":[{ "name":"ssn", "type":"string", "confluent:tags": [ "PII", "PRIVATE" ] }] } ``` In this example, the Social Security Number (`ssn`) field includes `PII` and `PRIVATE` to indicate that the field includes personally identifiable information (PII) that is marked as private. ### Step 4: Define an encryption policy After adding the tags, you need to define an encryption policy that specifies rules for which tags use for encryption. The encryption policy is defined in a JSON file that is then uploaded to Confluent Platform. Here is an example of an encryption policy: ```json { "schema": "...", "metadata": {...}, "ruleSet": { "domainRules": [ { "name": "encryptPII", "kind": "TRANSFORM", "type": "ENCRYPT", "mode": "WRITEREAD", "tags": ["PII"], "params": { "encrypt.kek.name": "" } } ] } } ``` Note that you specified the name of the KEK in step 1. If the KEK has not yet been registered, you can optionally specify the KMS key ID and KMS type in the rule. The client automatically registers the KEK before registering any DEKs. ```json { "schema": "...", "metadata": {...}, "ruleSet": { "domainRules": [ { "name": "encryptPII", "kind": "TRANSFORM", "type": "ENCRYPT", "mode": "WRITEREAD", "tags": ["PII"], "params": { "encrypt.kek.name": "", "encrypt.kms.key.id": "", "encrypt.kms.type": "aws-kms" } } ] } } ``` During registration, if the schema is omitted, then the ruleset attaches to the latest schema in the subject. After registration, you need to include the following in the client: * `auto.register.schemas=false` * `use.latest.version=true` If you do not include these properties, the client attempts to register, or look up, a schema without any existing rules. Encryption is supported for fields of type `string` or `bytes`. Type `integer` is currently not supported. If a client does not have a rule executor for the `ENCRYPT` rule type and attempts to consume a message, then the message is not decrypted and the client receives the encrypted values. Two additional properties can be specified for DEKs in the rule parameters: | Parameter | Description | |-------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | encrypt.dek.algorithm | The encryption algorithm being used. Valid values include `AES128_GCM`,
`AES256_GCM` (default), or `AES256_SIV`. You can use `AES256_SIV`
for deterministic encryption of keys. | | encrypt.dek.expiry.days | If specified, automatic DEK rotation occurs. When a DEK is older than the
expiration period, a new DEK is generated and used for new messages, while
previous DEKs are available to decrypt older messages. There is a limit to
the number of existing DEKs (10,000) that can be retained in the DEK Registry,
so use this property cautiously. | | preserve.source.fields | For performance reasons, the fields of a message are updated during field-level
transforms. For field-level encryption, this results in the field values
being replaced with the encrypted field values. If the original field values
should be retained in the message, then set this property to `true`. | ### Step 5: Configure the KMS key encryption key For Confluent Platform, specify the credentials to access your KMS on the client. Each producer and consumer that encrypts or decrypts data must be able to reach your KMS to encrypt and decrypt DEKs. For each of the supported KMS providers, the following dependencies are required: ### AWS ```xml io.confluent kafka-schema-registry-client-encryption-aws 7.6.0 ``` ### Azure ```xml io.confluent kafka-schema-registry-client-encryption-azure 7.6.0 ``` ### Google Cloud ```xml io.confluent kafka-schema-registry-client-encryption-gcp 7.6.0 ``` ### Hashicorp Vault ```xml io.confluent kafka-schema-registry-client-encryption-hcvault 7.6.0 ``` ### Local ```xml io.confluent kafka-schema-registry-client-encryption 7.6.0 ``` Next, you need to configure the following parameters on the clients. ```text rule.executors._default_.param.access.key.id= rule.executors._default_.param.secret.access.key= ``` Alternatively, the AWS access key and AWS secret key can be passed using environment variables, named `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. Now, whenever a message is sent, the `ssn` field is automatically encrypted before serialization and decrypted after deserialization. ### Step 6: Produce with Kafka serializers To produce with Kafka serializers, add the appropriate serializer to the producer properties: * KafkaAvroSerializer * KafkaProtobufSerializer * KafkaJsonSchemaSerializer ## Access control (RBAC) for CSFLE You must set the following property to enforce RBAC for CSFLE on Confluent Enterprise: `dek.registry.rbac.enable=true`. To learn more about this configuration option, see the entry for `dek.registry.rbac.enable` in the Schema Registry Configuration Reference. #### IMPORTANT CSFLE access control on Confluent Enterprise is distinct from your external KMS (for example, AWS KMS) permissions. You must configure BOTH: - Confluent Platform RBAC role bindings to authorize principals to manage KEKs/DEKs, encryption policy (rules), and Schema Registry resources. - External KMS IAM/policies to allow the DEK Registry (or your client) to use the KMS key. Having KMS access alone does not grant Confluent Platform RBAC permissions, and vice versa. RBAC role bindings for CSFLE on Confluent Enterprise are shown below. #### KEK/DEK Role Permissions | Role Name | Create KEK/DEK | Read KEK/DEK | Update KEK/DEK | Delete KEK/DEK | |-----------------|----------------------|----------------------|----------------------|----------------------| | SystemAdmin | ✔ | ✔ | ✔ | ✔ | | UserAdmin | | | | | | ClusterAdmin | ✔ | ✔ | ✔ | ✔ | | Operator | | | | | | SecurityAdmin | | | | | | ResourceOwner | ✔ (based on scoping) | ✔ (based on scoping) | ✔ (based on scoping) | ✔ (based on scoping) | | DeveloperRead | | ✔ (based on scoping) | | | | DeveloperWrite | ✔ (based on scoping) | ✔ (based on scoping) | ✔ (based on scoping) | | | DeveloperManage | | | | | #### NOTE Blank cells indicate no permission on KEK/DEK resources for that role. `DeveloperManage` does not grant permissions on KEK/DEK resources. For CSFLE operations, bind roles on the KEK by name (`Kek:`): use `ResourceOwner` for full lifecycle, or `DeveloperWrite`/`DeveloperRead` for scoped create/read/update. ### Permissions and scoping - RBAC authorizations for CSFLE are evaluated using the KEK name. Grant access by binding roles on the `Kek:` resource in the target Schema Registry cluster. There is no separate role binding required on DEKs; DEK operations are authorized based on the KEK binding (and request scoping such as subject/version). - RBAC roles do not determine whether a DEK can be decrypted. DEK encryption and decryption is permitted by your cloud provider KMS IAM (for example, AWS KMS, Azure Key Vault, or Google Cloud KMS). Ensure the DEK Registry principal (or client) has the needed KMS permissions. - Encrypted DEKs are persisted in the DEK Registry. Without KEK access (from KMS), DEKs cannot be decrypted and data remains inaccessible. ### Typical role sets by persona - Developer working on a subject with CSFLE: - On Schema Registry subject: `DeveloperRead` (read schemas) and, if producing/updating schemas, `DeveloperWrite`. - On KEK: `DeveloperWrite` or `DeveloperRead` on the scoped `Kek:` used by that subject to allow client on-demand registration and read of DEK for encryption and decryption. - In Confluent Cloud with KEK sharing enabled: producers/consumers must still have RBAC permission on the KEK to obtain DEKs; sharing does not bypass RBAC. - Application/service owner for a subject: - `ResourceOwner` on the subject; optionally `ResourceOwner` on the KEK used by that subject to fully manage lifecycle. - Security/platform admin: - `ClusterAdmin` on Schema Registry cluster to manage DEK Registry and policies; optionally `SystemAdmin` for break-glass. For role definitions, scopes, and permissions, see [Use Predefined RBAC Roles in Confluent Platform](../../authorization/rbac/rbac-predefined-roles.md#rbac-predefined-roles). For managing KEKs and DEKs, see [Manage Encryption Keys for CSFLE on Confluent Platform](manage-keys.md#manage-encryption-keys-csfle). #### NOTE Confluent Cloud vs Confluent Platform - This page documents Confluent Enterprise (self-managed) CSFLE. In Confluent Cloud, permissions are governed by Cloud RBAC and the Confluent Cloud console/API; KEK/DEK management APIs and roles differ. - External KMS permissions (for example, AWS IAM policies that allow `Encrypt`/`Decrypt` on your KMS key) are always required regardless of Confluent Enterprise or Confluent Cloud. In Confluent Enterprise, the client principal needs these KMS permissions. In Confluent Cloud, if KEK access is delegated to the DEK Registry, the DEK Registry principal needs them; otherwise the client principal does. - In Confluent Cloud you may optionally share KEK access with Confluent so that managed services (for example, fully managed connectors or other services) can process encrypted data. In this case, only principals granted RBAC permission to the KEK can request DEKs for encryption and decryption. When KEK access is not shared with Confluent, no Confluent-managed service or operator can decrypt data under any conditions. - In Confluent Enterprise there is no KEK-access-sharing option with Confluent-managed services; CSFLE remains fully client-managed. Only your producers/consumers with RBAC permission to the KEK and with appropriate KMS permissions can encrypt and decrypt. ## Handle errors If the Schema Registry client cannot decrypt the encrypted fields, it throws an error. You can configure the client to pass the encrypted data through in spite of the error. #### WARNING **Do not use dead-letter queues (DLQs) with the current version of CSFLE** because of the potential risk that unencrypted, sensitive data might be exposed in the DLQ. This can occur if messages with fields expected to be encrypted fail to be processed and are routed to the DLQ without proper encryption. Work is in progress to address this issue. ### To pass encrypted data through To pass encrypted data through in spite of an error, on the client specify an `onFailure` property in the rule. Here is an example: ```json { "schema": "...", "metadata": {...}, "ruleSet": { "domainRules": [ { "name": "encryptPII", "kind": "TRANSFORM", "type": "ENCRYPT", "mode": "WRITEREAD", "tags": ["PII"], "params": { "encrypt.kek.name": "" }, "onFailure": "ERROR,NONE" } ] } } ``` The `onFailure` setting value of `ERROR,NONE` above includes two comma-separated values, the first for encryption and the second for decryption. * `ERROR`: If encryption fails, an error is thrown. * `NONE`: If decryption fails, the encrypted value is passed through without decryption. ## Related content * [Manage Encryption Keys for CSFLE](manage-keys.md#manage-encryption-keys-csfle) * [Client-Side Field Level Encryption Examples for Confluent Platform](code-examples.md#code-examples-csfle)