Secrets

Confluent Platform secrets allow you to store and manage sensitive information, such as passwords and API tokens. Compliance requirements often dictate that services should not store sensitive data as clear text in files. This sensitive data can include passwords, such as in the configuration parameters ssl.key.password, ssl.keystore.password, ssl.truststore.password, or any other sensitive data in configuration files. Secrets are administered with the confluent secret commands.

The secrets are encrypted within the configuration file.

Secrets use “envelope encryption”, which is a standard way to protect encrypt sensitive data in a highly secure method. The way it works is a user specifies a master passphrase which is used, along with a cryptographic salt value, to derive a master encryption key. The master encryption key is used to generate a separate data encryption key.

The master encryption and data encryption keys are used to encrypt the sensitive data in the configuration files. The service can later decrypt the keys. If an unauthorized user gains access to a configuration file, they can only see the encrypted values and they have no way to decrypt them without knowing the master encryption key.

See also

For a scripted demo of using secrets, see the secrets demo.

Limitations

You currently cannot directly encrypt a JAAS file using secrets, but you can encrypt a JAAS configuration parameter that is declared in a properties file. For more information, see the Encrypt JAAS configuration parameters example.

Quick start

Prerequisite

  1. Create a directory for storing the security.properties file. For example:

    mkdir /usr/secrets/

  2. Generate the master encryption key based on a passphrase.

    Typically a passphrase is much longer than a password and is easily remembered as a string of words (e.g., The streaming platform). You can specify the passphrase either in clear text on the command line, or store it in a file. A best practice is to enter this passphrase into a file and then pass it to the CLI (specified as --passphrase @<passphrase.txt>). By using a file, you can avoid the logging history which shows the passphrase in plain text.

    Choose a location for the secrets file on your local host (not a location where Confluent Platform services run). The secrets file contains encrypted secrets for the master encryption key, data encryption key, and configuration parameters, along with their metadata such as which cipher was used for encryption.

    confluent secret master-key generate \
--local-secrets-file /usr/secrets/security.properties  \
--passphrase @<passphrase.txt>

    Your output should resemble:

    Save the master key. It cannot be retrieved later.
+------------+----------------------------------------------+
| Master Key | xiF48SV+7gI32Yy69L6WopgRtLe6r0Plz+a4y4j6aqa= |
+------------+----------------------------------------------+

  3. Save the master key because it cannot be retrieved later.

  4. Export the master key in the environment variable, or add the master key to a bash script.

    Important

    The subsequent confluent secret commands will fail if the environment variable is not set.

    export CONFLUENT_SECURITY_MASTER_KEY=xiF48SV+7gI32Yy69L6WopgRtLe6r0Plz+a4y4j6aqa=

  5. Encrypt the specified configuration parameters.

    This step encrypts parameters (--config) from a configuration file (--config-file). The encrypted secrets are stored in the local secrets file (security.properties). The file path of the remote or local node is specified in --remote-secrets-file. This is the path where the secrets.properties file will reside on the remote node. If you plan to run the Confluent CLI on each node, the remote-secrets-file should point to the location of the security.properties on the local node. If the --config flag is not specified, any configuration that contains the string password is encrypted in the configuration key. When using encrypt you can use a comma to specify multiple keys, for example: --config "config.storage.replication.factor,config.storage.topic". This option is not available when using the add or update commands.

    For example, use this command to encrypt the config.storage.replication.factor and config.storage.topic parameters:

    confluent secret file encrypt --config-file /etc/kafka/connect-distributed.properties \
--local-secrets-file /usr/secrets/security.properties \
--remote-secrets-file /usr/secrets/security.properties \
--config "config.storage.replication.factor,config.storage.topic"

    You should see a similar entry in your secrets.properties file. This example shows the encrypted config.storage.replication.factor parameter.

    config.storage.replication.factor = ${securepass:/usr/secrets/security.properties:connect-distributed.properties/config.storage.replication.factor}

  6. Decrypt the encrypted configuration parameter.

    confluent secret file decrypt \
--local-secrets-file /usr/secrets/security.properties \
--config-file /etc/kafka/connect-distributed.properties \
--output-file decrypt.txt

    You should see the decrypted parameter. This example shows the decrypted ssl.key.password parameter.

    config.storage.replication.factor=1

Tip

For more information about the secrets.properties file, see the properties file reference.

Production

To operationalize this workflow, you can augment your orchestration tooling to distribute this to the destination hosts. These hosts may include Kafka brokers, Connect workers, Schema Registry instances, KSQL servers, Control Center, or any service using password encryption. The confluent secret commands are flexible to accommodate whatever secret distribution model you prefer. You can either do the secret generation and configuration modification on each destination host directly, or do it all on a single host and then distribute the secret data to the destination hosts. Here are the tasks to distribute the secret data:

  • Export the master encryption key into the environment on every host that will have a configuration file with password protection.
  • Distribute the secrets file: copy the secrets file /path/to/security.properties from the local host on which you have been working to /path/to/security.properties on the destination hosts.
  • Propagate the necessary configuration file changes: update the configuration file on all hosts so that the configuration parameter now has the tuple for secrets.

Usage examples

Here are some usage examples for the confluent secret commands.

Important

The Confluent CLI must be installed.

Add remote encrypted configs from file

This command adds new encrypted configuration parameters (--config) to the specified file (--config-file). The encrypted secrets are stored in the local secrets file (--local-secrets-file).

Tip

You can specify multiple key-value pairs by separating each configuration parameter with a newline character, for example: --config "ssl.keystore.password = sslPassword \n ssl.truststore.password = password". If you include config group.id but do not include a key/value pair, or submit an empty list, you will receive an output error.

confluent secret file add --config-file /etc/kafka/connect-distributed.properties \
--local-secrets-file /usr/secrets/security.properties \
--remote-secrets-file /usr/secrets/security.properties \
--config group.id=connect-cluster

After running this command your properties file should resemble:

group.id = ${securepass:/usr/secrets/security.properties:connect-distributed.properties/group.id}

Encrypt JAAS configuration parameters

This example encrypts a JAAS configuration parameter sasl.jaas.config that is declared in the server.properties file.

  1. Add the JAAS configuration parameter to /etc/kafka/server.properties.

    sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    username=RDSH7XKHLFPIIL64" /
    password="5I9s79P2zRx4di4OCCT0EG1aIvM1/ygXTegFegXEa90JC2nrhIFZZXXWqZAqYhvP";

  2. Run this command to encrypt the file.

    confluent secret file encrypt --config-file /etc/kafka/server.properties \
--local-secrets-file /usr/secrets/security.properties \
--remote-secrets-file /usr/secrets/security.properties \
--config "sasl.jaas.config"

    The resulting encrypted value in security.properties will resemble:

    server.properties/sasl.jaas.config = ENC[AES/CBC/PKCS5Padding,data:qbsO+5p1cKLrwTvUFthL+IfBUiQbBDE/
bSc5SxNlJ9ZfHb8b771iXSF2dy94nnj3dqwsJfL9fa/nBLxhJcqCqoto/jg9sLIOHysHqP7Uj2/
ww3tqrh0g2Peobv5EIj4Z,iv:k7w3XbwWD/xnG7n3p4+xVA==,type:str]

Update encrypted configuration values

This command updates the encrypted configuration parameters (--config) in the specified file (--config-file). Configuration parameters are specified as key value pairs, for example --config "ssl.keystore.password = password". The encrypted secrets are stored in the local secrets file (local-secrets-file).

Tip

You can specify multiple key-value pairs by separating each configuration parameter with a newline character, for example: --config "ssl.keystore.password = sslPassword \n ssl.truststore.password = password".

confluent secret file update --config-file /etc/kafka/connect-distributed.properties \
--local-secrets-file /usr/secrets/security.properties \
--remote-secrets-file /usr/secrets/security.properties \
--config group.id=connect-cluster

Note

When using update and add: if you include config group.id but do not include a key/value pair, or submit an empty list, you will receive an output error.

Rotate keys

This command rotates the master key or data key.

  • Rotate master key (--master-key): Generates a new master key and re-encrypts the data with the new master key. The new master key is stored in an environment variable.

    confluent secret file rotate --master-key \
--local-secrets-file /usr/secrets/security.properties \
-–passphrase @/User/bob/secret.properties
--passphrase-new @/User/bob/secretNew.properties

  • Rotate data key (--data-key): Generates a new data key and re-encrypts the file with the new data key.

    confluent secret file rotate --data-key \
--local-secrets-file /usr/secrets/security.properties \
-–passphrase @/User/bob/secret.properties

Script commands

You can script the commands by using stdin or from a file:

  • To pipe from stdin use dash (-), for example --passphrase -.
  • To read from a file use @<path-to-file>, for example --passphrase @/User/bob/secret.properties.
confluent secret master-key generate \
--local-secrets-file /usr/demo/security.properties \
--passphrase @/Users/user.name/tmp/demo/masterkey.properties

echo -e "demoMasterKey" | confluent secret master-key generate --local-secrets-file
/usr/demo/security.properties --passphrase -

Fix corrupt master key

If your master key becomes corrupt or is lost, you must create new passwords in the properties file (e.g. ssl.keystore.password) and generate a new master key for the encrypted configuration parameters. The following example updates the ssl.keystore.password in the /etc/kafka/server.properties file.

  1. Create a new password in your properties file.

    ssl.keystore.password=mynewpassword

  2. Generate a new master key.

    confluent secret master-key generate \
--local-secrets-file <path-to-file>/security.properties  \
--passphrase @/User/bob/secret.properties

  3. Encrypt the configs with new master key.

    confluent secret file encrypt --config-file /etc/kafka/server.properties  \
--local-secrets-file <path-to-file>/security.properties \
--remote-secrets-file <path-to-file> \
--config ssl.keystore.password

Properties file reference

The secrets.properties looks like this:

1
2
3
4
5
6
7
8
_metadata.master_key.0.salt = HfCqzb0CSXSo/mEx2Oc0lUqY5hP3vGa/SayR5wxQogI=
_metadata.symmetric_key.0.created_at = 2019-07-16 16:36:05.480926 -0700 PDT m=+0.006733395
_metadata.symmetric_key.0.envvar = CONFLUENT_SECURITY_MASTER_KEY
_metadata.symmetric_key.0.length = 32
_metadata.symmetric_key.0.iterations = 1000
_metadata.symmetric_key.0.salt = hCC00OJG2VzhHhLMB6hZSuE9KBKutMK8BxFhq8OUirg=
_metadata.symmetric_key.0.enc = ENC[AES/CBC/PKCS5Padding,data:TDSUb8f6IzUtgAffkQ8jZ55QU1sn+OTbvr2+FzX1bkjnrV4d6uwqtsTxzltiG8nO,iv:1ieTVqWxOC06rDcO9XQuOQ==,type:str]
server.properties/ssl.keystore.password = ENC[AES/CBC/PKCS5Padding,data:jOGowFcgq4q1MqcJEGWCsg==,iv:3iqk+FJAbnW7MOYEiPkyFA==,type:str]
  • Line 1: _metadata.master_key.0.salt is the salt used for generating the master key.
  • Line 2: _metadata.symmetric_key.0.created_at is the timestamp when data key is created.
  • Line 3: metadata.symmetric_key.0.envvar is the master key environment variable.
  • Line 4: _metadata.symmetric_key.0.length is the length of data key in bytes.
  • Line 5: _metadata.symmetric_key.0.iterations is the number of iterations used for encryption.
  • Line 6: _metadata.symmetric_key.0.salt is the salt used for generating the data key.
  • Line 7: _metadata.symmetric_key.0.enc is the data key wrapped using master key in the following key:value format, where: ENC[<algorithm>,data:<data>:,iv:<initialization-vector>,type:<type>]. The algorithm format used is:
    • Symmetric encryption algorithm: Advanced Encryption Standard (AES)
    • Encryption mode: Cipher block chaining (CBC)
    • Padding scheme: PKCS5Padding