Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Secrets

Confluent Platform secrets allow you to store and manage sensitive information, such as passwords and API tokens. Compliance requirements often dictate that services should not store sensitive data as clear text in files. This sensitive data can include passwords, such as in the configuration parameters ssl.key.password, ssl.keystore.password, ssl.truststore.password, or any other sensitive data in configuration files. Secrets are administered with the confluent secret commands.

When you run confluent secret, the configuration file is modified to include code that directs the configuration resolution system to pull the configuration from a secret provider. A second file (the secrets file) that contains the encrypted secrets is also created.

Secrets use “envelope encryption”, which is a standard way to protect sensitive data with a highly secure method. A user specifies a master passphrase, which is used along with a cryptographic salt value to derive a master encryption key. The master encryption key is used to generate a separate data encryption key.

The master encryption and data encryption keys are used to encrypt the sensitive data in the configuration files. The service can later decrypt the keys. If an unauthorized user gains access to a configuration file, they cannot see the encrypted values and they have no way to decrypt them without knowing the master encryption key.

See also

For a scripted demo of how to use secrets, see the secrets demo.

Limitations

You cannot directly encrypt a JAAS file using secrets, but you can encrypt a JAAS configuration parameter that is declared in a properties file. For more information, see the Encrypt JAAS configuration parameters example.

Quick start

Prerequisite

• The Confluent CLI must be installed.

1. Create a directory for storing the security.properties file. For example:

   mkdir /usr/secrets/
   

2. Generate the master encryption key based on a passphrase.

   Typically a passphrase is much longer than a password and is easily remembered as a string of words (e.g., The streaming platform). You can specify the passphrase either in clear text on the command line, or store it in a file. A best practice is to enter this passphrase into a file and then pass it to the CLI (specified as --passphrase @<passphrase.txt>). By using a file, you can avoid the logging history which shows the passphrase in plain text.

   Choose a location for the secrets file on your local host (not a location where Confluent Platform services run). The secrets file contains encrypted secrets for the master encryption key, data encryption key, and configuration parameters, along with their metadata such as which cipher was used for encryption.

   confluent secret master-key generate \
   --local-secrets-file /usr/secrets/security.properties  \
   --passphrase @<passphrase.txt>
   

   Your output should resemble:

   Save the master key. It cannot be retrieved later.
   +------------+----------------------------------------------+
   | Master Key | xiF48SV+7gI32Yy69L6WopgRtLe6r0Plz+a4y4j6aqa= |
   +------------+----------------------------------------------+
   

3. Save the master key because it cannot be retrieved later.

4. Export the master key in the environment variable, or add the master key to a bash script.

   Important

   The subsequent confluent secret commands will fail if the environment variable is not set.

   export CONFLUENT_SECURITY_MASTER_KEY=xiF48SV+7gI32Yy69L6WopgRtLe6r0Plz+a4y4j6aqa=
   

5. Encrypt the specified configuration parameters.

   This step encrypts the properties specified by --config in the configuration file specified by --config-file. The property values are read from the configuration file, encrypted, and written to the local secrets file specified by --local-secrets-file. In place of the property values, instructions that are written into the configuration file allow the configuration resolution system to retrieve the secret values at runtime.

   The file path you specify in --remote-secrets-file is written into the configuration instructions and identifies where the resolution system can locate the secrets file at runtime. If you are running the secrets command centrally and distributing the secrets file to each node, then specify the eventual path of the secrets file in --remote-secrets-file. If you plan to run the secrets command on each node, then the remote-secrets-file should match the location specified by --local-secrets-file.

   If the --config flag is not specified, any property that contains the string password is encrypted in the configuration key. When running encrypt use a comma to specify multiple keys, for example: --config "config.storage.replication.factor,config.storage.topic". This option is not available when using the add or update commands.

   Use the following example command to encrypt the config.storage.replication.factor and config.storage.topic parameters:

   confluent secret file encrypt --config-file /etc/kafka/connect-distributed.properties \
   --local-secrets-file /usr/secrets/security.properties \
   --remote-secrets-file /usr/secrets/security.properties \
   --config "config.storage.replication.factor,config.storage.topic"
   

   You should see a similar entry in your secrets.properties file. This example shows the encrypted config.storage.replication.factor parameter.

   config.storage.replication.factor = ${securepass:/usr/secrets/security.properties:connect-distributed.properties/config.storage.replication.factor}
   

6. Decrypt the encrypted configuration parameter.

   confluent secret file decrypt \
   --local-secrets-file /usr/secrets/security.properties \
   --config-file /etc/kafka/connect-distributed.properties \
   --output-file decrypt.txt
   

   You should see the decrypted parameter. This example shows the decrypted ssl.key.password parameter.

   config.storage.replication.factor=1
   

Tip

For more information about the secrets.properties file, see the properties file reference.

Production

To operationalize this workflow, you can augment your orchestration tooling to distribute this to the destination hosts. These hosts may include Kafka brokers, Connect workers, Schema Registry instances, ksqlDB servers, Control Center, or any service using password encryption. The confluent secret commands are flexible to accommodate whatever secret distribution model you prefer. You can either do the secret generation and configuration modification on each destination host directly, or do it all on a single host and then distribute the secret data to the destination hosts. Here are the tasks to distribute the secret data:

• Export the master encryption key into the environment on every host that will have a configuration file with password protection.
• Distribute the secrets file: copy the secrets file /path/to/security.properties from the local host on which you have been working to /path/to/security.properties on the destination hosts.
• Propagate the necessary configuration file changes: update the configuration file on all hosts so that the configuration parameter now has the tuple for secrets.

Usage examples

Here are some usage examples for the confluent secret commands.

Important

The Confluent CLI must be installed.

Add remote encrypted configs from file

This command adds new encrypted configuration parameters (--config) to the specified file (--config-file). The encrypted secrets are stored in the local secrets file (--local-secrets-file).

Tip

You can specify multiple key-value pairs by separating each configuration parameter with a newline character, for example: --config "ssl.keystore.password = sslPassword \n ssl.truststore.password = password". If you include config group.id but do not include a key/value pair, or submit an empty list, you will receive an output error.

confluent secret file add --config-file /etc/kafka/connect-distributed.properties \
--local-secrets-file /usr/secrets/security.properties \
--remote-secrets-file /usr/secrets/security.properties \
--config group.id=connect-cluster

After running this command your properties file should resemble:

group.id = ${securepass:/usr/secrets/security.properties:connect-distributed.properties/group.id}

Using prefixes in secrets configurations

Secrets config.providers do not propagate to prefixes such as client.*. Thus, when using prefixes with secrets you must specify config.providers and config.providers.securepass.class:

client.config.providers=securepass
client.config.providers.securepass.class=io.confluent.kafka.security.config.provider.SecurePassConfigProvider

Encrypt JAAS configuration parameters

All Confluent Platform components support an embedded JAAS configuration, which provides a secure way to specify your JAAS configuration. Secrets also support using an embedded JAAS configuration. This configuration method allows for a more granular level of security for your secrets JAAS configuration. For example, rather than encrypting your entire secrets JAAS configuration, as you would using a properties file, when using an embedded JAAS configuration, you can encrypt the password only. In this way, your secrets configuration details will be available in logs, but not your password.

Secrets does not support using a static JAAS configuration file that is passed at runtime, as is done with brokers.

Note

• The ZooKeeper Client JAAS configuration is not supported by secrets protection.
• You cannot encrypt multiple passwords in the same JAAS configuration using the Confluent CLI secrets encryption command for encrypting the JAAS. For more information, see Known issue.

Before performing any operation on the JAAS configuration, you must first provide the configuration key in the predefined path: <entry name>/<LoginModule>/<key>.

Following is an example JAAS configuration in a property file (kafka/server.properties):

sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required /
useKeyTab=false /
adminpassword=tempPass /
useTicketCache=true /
doNotPrompt=true;

Note

You can specify the path to adminpassword as: sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword

The standard CLI syntax for encrypting, adding, or updating a secrets JAAS configuration is:

confluent secret file encrypt --config-file
--local-secrets-file
--remote-secrets-file
–-config

Of course, you must replace encrypt with add or update, depending on the configuration task you are performing.

This example shows the command and options for encrypting the password in a secrets JAAS configuration:

confluent secret file encrypt --config-file /etc/kafka/server.properties
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
–-config sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword

This example shows the command and options for adding a new secrets JAAS configuration in which the password is encrypted:

confluent secret file add --config-file /etc/kafka/server.properties
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
–-config sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword

This example shows the command and options for updating the encrypted password in the secrets JAAS configuration:

confluent secret file update --config-file /etc/kafka/server.properties
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
–-config sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword

The encrypted data that you see after executing any of the preceding commands should look similar to the following:

_metadata.master_key.0.salt = 3B++ViaAMaSwOOnxYI2bbeCtvZXRV5mxEOfb2FO3DnU=
_metadata.symmetric_key.0.created_at = 2020-03-06 14:07:34.248521 -0700 MST m=+0.011810143
_metadata.symmetric_key.0.envvar = CONFLUENT_SECURITY_MASTER_KEY
_metadata.symmetric_key.0.length = 32
_metadata.symmetric_key.0.iterations = 1000
_metadata.symmetric_key.0.salt = 4aqTrl8kdnQdVbGwkbeQHUiLA235/RKGC8zOXTHwQaI=
_metadata.symmetric_key.0.enc = ENC[AES/CBC/PKCS5Padding,data:bP93/lcsQVY5tzh4NWvD9tWO/yyTGwdAEgYHwpUokjiomma7QoH8X/jhlB7zibGd,iv:A7zk7hBwuataNy+ToT346w==,type:str]
server.properties/sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword = ENC[AES/CBC/PKCS5Padding,data:KdjkpudhWKoVe+6G35OYDw==,iv:5MgMsMT1o8d1JlXE0966Bg==,type:str]

Encrypt JSON configuration parameters

Secrets supports the encryption of JSON configuration parameters.

Before performing any operation on the JSON configuration parameters, you must first provide the configuration key in the path format:

{
    "name": "security configuration",
    "credentials": {
        "password": "password",
        "ssl.keystore.location": "/usr/ssl"
    }
}

You can specify the path to ssl.keystore.password as:

<entry-name>.<key> :
credentials.password
credentials.ssl\\.keystore\\.location

The standard CLI syntax for encrypting, adding, updating, or decrypting a secrets JSON configuration is:

confluent secret file encrypt --config-file
--local-secrets-file
--remote-secrets-file
–-config

Of course, you must replace encrypt with add, update, or decrypt, depending on the configuration task you are performing.

This example shows the command and options for encrypting the password in the secrets JSON configuration:

./confluent secret file encrypt
--config-file /etc/kafka/sample.json
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
--config credentials.password

Note

The --config key for JSON configurations must be separated by . in the path syntax.

This example shows the command and options for adding a new secrets JSON configuration in which the password is encrypted:

./confluent secret file add
--config-file /etc/kafka/sample.json
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
--config credentials.password

This example shows the command and options for updating the encrypted password in a secrets JSON configuration:

./confluent secret file update
--config-file /etc/kafka/sample.json
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
--config credentials.password

The encrypted data that you see after executing the preceding commands should look similar to the following:

_metadata.master_key.0.salt = 3B++ViaAMaSwOOnxYI2bbeCtvZXRV5mxEOfb2FO3DnU=
_metadata.symmetric_key.0.created_at = 2020-03-06 14:07:34.248521 -0700 MST m=+0.011810143
_metadata.symmetric_key.0.envvar = CONFLUENT_SECURITY_MASTER_KEY
_metadata.symmetric_key.0.length = 32
_metadata.symmetric_key.0.iterations = 1000
_metadata.symmetric_key.0.salt = 4aqTrl8kdnQdVbGwkbeQHUiLA235/RKGC8zOXTHwQaI=
_metadata.symmetric_key.0.enc = ENC[AES/CBC/PKCS5Padding,data:bP93/lcsQVY5tzh4NWvD9tWO/yyTGwdAEgYHwpUokjiomma7QoH8X/jhlB7zibGd,iv:A7zk7hBwuataNy+ToT346w==,type:str]
server.properties/sasl.jaas.config/com.sun.security.auth.module.Krb5LoginModule/adminpassword = ENC[AES/CBC/PKCS5Padding,data:KdjkpudhWKoVe+6G35OYDw==,iv:5MgMsMT1o8d1JlXE0966Bg==,type:str]
sample.json/credentials.password = ENC[AES/CBC/PKCS5Padding,data:4cCPvtf9Sgpf6amU358NDw==,iv:Aq/OmYfGIdbyw78LRe5gHQ==,type:str]

This example shows the command and options for decrypting the password in a secrets JSON configuration:

./confluent secret file decrypt
--config-file /etc/kafka/sample.json
--local-secrets-file /usr/secrets/security.properties
--remote-secrets-file /usr/secrets/security.properties
--config credentials.password

Rotate keys

This command rotates the master key or data key.

• Rotate master key (--master-key): Generates a new master key and re-encrypts the data with the new master key. The new master key is stored in an environment variable.

  confluent secret file rotate --master-key \
  --local-secrets-file /usr/secrets/security.properties \
  -–passphrase @/User/bob/secret.properties
  --passphrase-new @/User/bob/secretNew.properties
  

• Rotate data key (--data-key): Generates a new data key and re-encrypts the file with the new data key.

  confluent secret file rotate --data-key \
  --local-secrets-file /usr/secrets/security.properties \
  -–passphrase @/User/bob/secret.properties
  

Script commands

You can script the commands by using stdin or from a file:

• To pipe from stdin use dash (-), for example --passphrase -.
• To read from a file use @<path-to-file>, for example --passphrase @/User/bob/secret.properties.

confluent secret master-key generate \
--local-secrets-file /usr/demo/security.properties \
--passphrase @/Users/user.name/tmp/demo/masterkey.properties

echo -e "demoMasterKey" | confluent secret master-key generate --local-secrets-file
/usr/demo/security.properties --passphrase -

Fix corrupt master key

If your master key becomes corrupt or is lost, you must create new passwords in the properties file (e.g. ssl.keystore.password) and generate a new master key for the encrypted configuration parameters. The following example updates the ssl.keystore.password in the /etc/kafka/server.properties file.

1. Create a new password in your properties file.

   ssl.keystore.password=mynewpassword
   

2. Generate a new master key.

   confluent secret master-key generate \
   --local-secrets-file <path-to-file>/security.properties  \
   --passphrase @/User/bob/secret.properties
   

3. Encrypt the configs with new master key.

   confluent secret file encrypt --config-file /etc/kafka/server.properties  \
   --local-secrets-file <path-to-file>/security.properties \
   --remote-secrets-file <path-to-file> \
   --config ssl.keystore.password
   

Properties file reference

The secrets.properties looks like this:

_metadata.master_key.0.salt = HfCqzb0CSXSo/mEx2Oc0lUqY5hP3vGa/SayR5wxQogI=
_metadata.symmetric_key.0.created_at = 2019-07-16 16:36:05.480926 -0700 PDT m=+0.006733395
_metadata.symmetric_key.0.envvar = CONFLUENT_SECURITY_MASTER_KEY
_metadata.symmetric_key.0.length = 32
_metadata.symmetric_key.0.iterations = 1000
_metadata.symmetric_key.0.salt = hCC00OJG2VzhHhLMB6hZSuE9KBKutMK8BxFhq8OUirg=
_metadata.symmetric_key.0.enc = ENC[AES/CBC/PKCS5Padding,data:TDSUb8f6IzUtgAffkQ8jZ55QU1sn+OTbvr2+FzX1bkjnrV4d6uwqtsTxzltiG8nO,iv:1ieTVqWxOC06rDcO9XQuOQ==,type:str]
server.properties/ssl.keystore.password = ENC[AES/CBC/PKCS5Padding,data:jOGowFcgq4q1MqcJEGWCsg==,iv:3iqk+FJAbnW7MOYEiPkyFA==,type:str]

• Line 1: _metadata.master_key.0.salt is the salt used for generating the master key.
• Line 2: _metadata.symmetric_key.0.created_at is the timestamp when data key is created.
• Line 3: _metadata.symmetric_key.0.envvar is the master key environment variable.
• Line 4: _metadata.symmetric_key.0.length is the length of data key in bytes.
• Line 5: _metadata.symmetric_key.0.iterations is the number of iterations used for encryption.
• Line 6: _metadata.symmetric_key.0.salt is the salt used for generating the data key.
• Line 7: _metadata.symmetric_key.0.enc is the data key wrapped using master key in the following key:value format, where: ENC[<algorithm>,data:<data>:,iv:<initialization-vector>,type:<type>]. The algorithm format used is:
  • Symmetric encryption algorithm: Advanced Encryption Standard (AES)
  • Encryption mode: Cipher block chaining (CBC)
  • Padding scheme: PKCS5Padding

Suggested reading

Blog post: Shoulder Surfers Beware: Confluent Now Provides Cross-Platform Secret Protection