Configure Security in ksqlDB for Confluent Platform
ksqlDB supports several combinations of encryption and authentication on its client-facing and internal endpoints. ksqlDB also supports many of the security features of the other services it communicates with, like Apache Kafka® and Schema Registry.
Securing ksqlDB for Confluent Cloud
You can use ksqlDB with a Kafka cluster in Confluent Cloud. For more information, see Connecting ksqlDB to Confluent Cloud.
Securing ksqlDB on-premises
This section covers how to secure installations of ksqlDB outside of Confluent Cloud, like on-premises installations or manual installations on other cloud platforms.
To configure security for ksqlDB, add your configuration settings to the
<path-to-confluent>/etc/ksqldb/ksql-server.properties file and then
start the ksqlDB Server with your configuration file specified.
<path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksqldb/ksql-server.properties
Tip
These instructions assume you are installing Confluent Platform by using ZIP or TAR archives. For more information, see On-Premises Deployments.
The section is split into:
Securing ksqlDB installation: Covers how to secure access to ksqlDB itself and its own internal communication.
Securing communication with other services: Covers how to secure communication with other services and access to the resources they hold.
Securing ksqlDB installation
ksqlDB supports two main deployment modes:
Securing interactive deployments: Interactive deployments are those where the ksqlDB servers accept client connections.
Securing headless deployments: headless deployments are those where the ksqlDB servers do not accept client connections. They read the SQL statements they should run from a file on-disk.
Securing interactive deployments
Securing the interactive ksqlDB installation involves securing the HTTP endpoints that the ksqlDB server is listening on.
As well as accepting connections and requests from clients, a multi-node ksqlDB cluster also requires inter-node communications. You can choose to configure the external client and internal inter-node communication separately or over a single listener:
Securing single listener setup: Ideal for single-node installations, or where the inter-node communication is over the same network interface as client communication.
Securing dual listener setup: Useful where inter-node communication is over a different network interface or requires different authentication or encryption configuration.
Securing single-listener setup
Securing a single listener for ksqlDB is appropriate when both client and inter-node communication utilize the same authentication and security configuration.
The supported options include encrypting the connection and authenticating clients:
Configuring listener for SSL encryption: Creates an encrypted connection between the client and server, as well as for inter-node communication.
Configuring listener for HTTP-BASIC authentication: Uses a username and password for authenticating to ksqlDB.
Configuring Listener for SSL encryption
ksqlDB can be configured to use HTTPS rather than the default HTTP for all communication.
If you haven’t already, you will need to create SSL key and trust stores.
Use the following settings to configure the ksqlDB server to use HTTPS:
listeners=https://hostname:port
ssl.truststore.location=/var/private/ssl/ksql.server.truststore.jks
ssl.truststore.password=zzzz
ssl.keystore.location=/var/private/ssl/ksql.server.keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
Note the use of the HTTPS protocol in the listeners config.
To enable the server to authenticate clients (2-way authentication), use the following additional setting:
ssl.client.authentication=REQUIRED
Note
The ssl.client.auth setting is deprecated.
Additional server configuration options for HTTPS
Additional settings are available for configuring ksqlDB for HTTPS.
ssl.enabled.protocols: A comma-separated list of protocols enabled
for SSL connections. Leave blank to use the default from the
Apache Kafka® SslConfigs.java file
(see DEFAULT_SSL_ENABLED_PROTOCOLS).
ssl.cipher.suites: A comma-separated list of SSL cipher suites.
Leave blank to use your JVM defaults.
Configure the CLI for HTTPS
If the ksqlDB server is configured to use HTTPS, CLI instances may need to be configured with suitable key and trust stores.
If the server’s SSL certificate is not signed by a recognized public Certificate Authority, you must configure the CLI with a trust store that trusts the server’s SSL certificate.
If you haven’t already, create SSL key and trust stores.
Use the following settings to configure the CLI server:
ssl.truststore.location=/var/private/ssl/ksql.client.truststore.jks
ssl.truststore.password=<secure-password>
If the server is performing client authentication (2-way authentication), use the following additional settings:
ssl.keystore.location=/var/private/ssl/ksql.client.keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=<another-secure-password>
# Used to find the key store entry with the given alias in the keystore
ssl.keystore.alias=<key-store-entry-alias>
Settings for the CLI can be stored in a suitable file and passed to the
CLI by using the --config-file command-line arguments, for example:
<path-to-confluent>/bin/ksql --config-file ./config/ksql-cli.properties https://localhost:8088
Securing dual-listener setup
Using dual listeners for ksqlDB is appropriate when the client and inter-node communication utilize different authentication and security configurations. This is most likely the case when ksqlDB is deployed as an IaaS service.
The supported setups are SSL-mutual auth for the internal communication combined with SSL encryption and authentication for the external client:
Configuring internal for SSL-mutual authentication: Creates secure and authenticated connections for inter-node communication, but leaves the external client API unsecured. This is most appropriate when clients are trusted, but the internal APIs are protected from use.
Configuring internal for SSL-mutual authentication and external for HTTP-BASIC authentication: Creates secure and authenticated connections for inter-node communication and uses basic authentication for the external client API. This is most likely to be used with SSL above.
Configuring internal for SSL-mutual authentication and external for SSL encryption: Creates secure and authenticated connections for inter-node communication and uses SSL for the external client API. This is most likely to be used with authentication below.
Configuring internal for SSL-mutual authentication
ksqlDB supports securing inter-node communication using SSL mutual authentication.
For more information about configuring ksql.internal.listener, see
Configuring Listeners of a ksqlDB Cluster.
Your key store must contain the key pair for your internal listener set
with ksql.internal.listener. If your internal certificate is not
signed by a recognized public Certificate Authority, a trust store is
required to contain certificates for nodes in your cluster. Below is an
example configuration:
# Contains the internal key pair for this node.
# (e.g. key pair for node-1.internal.example.com)
ssl.keystore.location=/var/private/ssl/ksql.server.keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
# Contains the certificates for nodes in the cluster.
# (e.g. certs for node-1.internal.example.com, node-2.internal.example.com)
ssl.truststore.location=/var/private/ssl/ksql.server.truststore.jks
ssl.truststore.password=zzzz
listeners=http://0.0.0.0:8088
ksql.internal.listener=https://node-1.internal.example.com:8099
# This enables mutual auth checking for the internal listener
ksql.internal.ssl.client.authentication=REQUIRED
Configuring internal for SSL-mutual authentication and external for HTTP-BASIC authentication
Client-facing basic HTTP authentication can be used alongside authentication for the internal listener. This ensures that neither the client or internal APIs can be accessed by unauthorized users.
Note
If you’re using Basic authentication, you should configure ksqlDB to use HTTPS for secure communication, because the Basic protocol passes credentials in plain text.
Below is an example configuration:
ssl.keystore.location=/var/private/ssl/ksql.server.keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
ssl.truststore.location=/var/private/ssl/ksql.server.truststore.jks
ssl.truststore.password=zzzz
listeners=http://0.0.0.0:8088
ksql.internal.listener=https://node-1.internal.example.com:8099
ksql.internal.ssl.client.authentication=REQUIRED
authentication.method=BASIC
authentication.roles=admin,developer,user,ksq-user
authentication.realm=KsqlServer-Props
For more detail on basic authentication, see Configuring internal for SSL-mutual authentication and external for HTTP-BASIC authentication.
Configuring internal for SSL-mutual authentication and external for SSL encryption
If you want to use HTTPS on listeners as well as use SSL mutual auth
for internal communication on ksql.internal.listener, you will
likely require two different key pairs, since your host’s identity to
clients may be different from its internal identity. To create such a
key store, refer below.
In such a configuration, you must specify which key pair is used for a
given listener by providing a key store alias. For example, if set,
ksql.ssl.keystore.alias.internal will be used to find the key store
entry with the given alias when setting up the internal listener.
Similarly, ksql.ssl.keystore.alias.external is used for the client
listener listeners. Below is an example configuration:
# Contains the client certificate for this node under the alias 'client'.
# (e.g. key pair for external.example.com)
# Contains the internal certificate for this node under the alias 'internal_node1'.
# (e.g. key pair for node-1.internal.example.com)
ssl.keystore.location=/var/private/ssl/ksql.server.keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
# The aliases referenced above
ksql.ssl.keystore.alias.external=client
ksql.ssl.keystore.alias.internal=internal_node1
# Contains the certificates for nodes in the cluster.
# (e.g. certs for node-1.internal.example.com, node-2.internal.example.com)
ssl.truststore.location=/var/private/ssl/ksql.server.truststore.jks
ssl.truststore.password=zzzz
listeners=https://external.example.com:8088
ksql.internal.listener=https://node-1.internal.example.com:8099
# Enable mutual auth checking for the internal listener
ksql.internal.ssl.client.authentication=REQUIRED
# Don't require external clients to authenticate via SSL
ksql.ssl.client.authentication=NONE
Setting up a Key Store and Trust Store
In order to create a keystore with multiple key pairs with aliases, follow the below examples, depending on the source of the keys.
# Generated key pairs with aliases 'client' and 'internal_node1'
keytool -genkey -alias client -keyalg RSA -keypass password -storepass password -keystore ksql.server.keystore.p12 -storetype PKCS12
keytool -genkey -alias internal_node1 -keyalg RSA -keypass password -storepass password -keystore ksql.server.keystore.p12 -storetype PKCS12
# Imported key pairs, with aliases 'client' and 'internal_node1'
keytool -importkeystore -deststorepass password -destkeystore ksql.server.keystore.p12 -deststoretype PKCS12 -destalias client -srckeystore client_api.p12 -srcstoretype PKCS12 -srcalias client
keytool -importkeystore -deststorepass password -destkeystore ksql.server.keystore.p12 -deststoretype PKCS12 -destalias internal_node1 -srckeystore internal_node1.p12 -srcstoretype PKCS12 -srcalias internal_node1
Also, extracting certificates to add to a trust store can be done with the following commands:
keytool -export -alias internal_node1 -storepass password -file node1.cer -keystore internal_node1.p12 -srcstoretype PKCS12
keytool -export -alias internal_node2 -storepass password -file node2.cer -keystore internal_node2.p12 -srcstoretype PKCS12
keytool -import -v -trustcacerts -alias internal_node1 -file node1.cer -keystore ksql.server.truststore.jks -keypass password -storepass password
keytool -import -v -trustcacerts -alias internal_node1 -file node2.cer -keystore ksql.server.truststore.jks -keypass password -storepass password
Securing headless deployments
Unlike interactive deployments, there are no exposed REST APIs, so security is greatly simplified.
You may still have to secure communication with other services.
Securing communication with other services
This section covers how to secure communications of ksqlDB with other services.
The section is split into:
Configure ksqlDB for Secured Confluent Schema Registry: Covers how to secure communication with Schema Registry.
Configure ksqlDB for Secured Apache Kafka clusters: Covers how to secure communication with Kafka.
Configure ksqlDB for Secured Confluent Schema Registry
You can configure ksqlDB to connect to Schema Registry over HTTP by setting
the ksql.schema.registry.url to the HTTPS endpoint of Schema Registry.
Depending on your security setup, you might also need to supply
additional SSL configuration. For example, a trustStore is required if
the Schema Registry SSL certificates aren’t trusted by the JVM by default. A
keyStore is required if Schema Registry requires mutual authentication.
You can configure SSL for communication with Schema Registry by using
non-prefixed names, like ssl.truststore.location, or prefixed names
like ksql.schema.registry.ssl.truststore.location. Non-prefixed
names are used for settings that are shared with other communication
channels, where the same settings are required to configure SSL
communication with both Kafka and Schema Registry. Prefixed names affect
communication with Schema Registry only and override any non-prefixed
settings of the same name.
Use the following to configure ksqlDB for communication with Schema Registry over HTTPS, where mutual authentication isn’t required and Schema Registry SSL certificates are trusted by the JVM:
ksql.schema.registry.url=https://<host-name-of-schema-registry>:<ssl-port>
Use the following settings to configure ksqlDB for communication with Schema Registry over HTTPS, with mutual authentication, with an explicit trustStore, and where the SSL configuration is shared between Kafka and Schema Registry:
ksql.schema.registry.url=https://<host-name-of-schema-registry>:<ssl-port>
ksql.schema.registry.ssl.truststore.location=/etc/kafka/secrets/ksql.truststore.jks
ksql.schema.registry.ssl.truststore.password=<your-secure-password>
ksql.schema.registry.ssl.keystore.location=/etc/kafka/secrets/ksql.keystore.jks
ksql.schema.registry.ssl.keystore.password=<your-secure-password>
ksql.schema.registry.ssl.key.password=<your-secure-password>
Use the following settings to configure ksqlDB for communication with Schema Registry over HTTP, without mutual authentication and with an explicit trustStore. These settings explicitly configure only ksqlDB to Schema Registry SSL communication.
ksql.schema.registry.url=https://<host-name-of-schema-registry>:<ssl-port>
ksql.schema.registry.ssl.truststore.location=/etc/kafka/secrets/sr.truststore.jks
ksql.schema.registry.ssl.truststore.password=<your-secure-password>
The exact settings will vary depending on the encryption and authentication mechanisms Schema Registry is using, and how your SSL certificates are signed.
You can pass authentication settings to the Schema Registry client used by ksqlDB by adding the following to your ksqlDB Server config.
ksql.schema.registry.basic.auth.credentials.source=USER_INFO
ksql.schema.registry.basic.auth.user.info=username:password
For more information, see Schema Registry Security Overview.
Configure ksqlDB for Secured Apache Kafka clusters
The following are common configuration examples.
Configuring Kafka Encrypted Communication
This configuration enables ksqlDB to connect to a Kafka cluster over SSL, with a user supplied trust store:
security.protocol=SSL
ssl.truststore.location=/etc/kafka/secrets/kafka.client.truststore.jks
ssl.truststore.password=confluent
The exact settings will vary depending on the security settings of the Kafka brokers, and how your SSL certificates are signed. For full details, and instructions on how to create suitable trust stores, please refer to the Security Guide.
To use separate trust stores for encrypted communication with
Kafka and external communication with ksqlDB clients, prefix the
SSL truststore configs with ksql.streams.:
security.protocol=SSL
ksql.streams.ssl.truststore.location=/etc/kafka/secrets/kafka.client.truststore.jks
ksql.streams.ssl.truststore.password=confluent
Configure Kafka Authentication
This configuration enables ksqlDB to connect to a secure Kafka cluster using PLAIN SASL, where the SSL certificates have been signed by a CA trusted by the default JVM trust store.
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=\
org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<ksql-user>" \
password="<password>";
The exact settings will vary depending on what SASL mechanism your Kafka cluster is using and how your SSL certificates are signed. For more information, see the Security Guide.