Configuring KSQL Server

KSQL configuration parameters can be set for KSQL server and queries as well as for the underlying Kafka Streams and Kafka Clients (producer and consumer).

Tip

These instructions assume you are installing Confluent Platform by using ZIP or TAR archives. For more information, see Install.

Setting KSQL Server Parameters

You can specify KSQL server configuration parameters by using the server configuration file (ksql-server.properties) or the KSQL_OPTS environment variable. Properties set with KSQL_OPTS take precedence over those specified in the KSQL configuration file. A recommended approach is to configure a common set of properties using the KSQL configuration file and override specific properties as needed, using the KSQL_OPTS environment variable.

KSQL Server Configuration File

By default the KSQL server configuration file is located at <path-to-confluent>/etc/ksql/ksql-server.properties. The file follows the syntax conventions of Java properties files.

<property-name>=<property-value>

For example:

bootstrap.servers=localhost:9092
listeners=http://localhost:8088

After you have updated the server configuration file, you can start the KSQL server with the configuration file specified.

$ <path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

For more information, see KSQL Configuration Parameter Reference.

KSQL_OPTS Environment Variable

You can override KSQL server configuration parameters by using the KSQL_OPTS environment variable. The properties are standard Java system properties. For example, to set ksql.streams.num.streams.threads to 1:

$ KSQL_OPTS="-Dksql.streams.num.streams.threads=1" <path-to-confluent>/bin/ksql-server-start \
  <path-to-confluent>/etc/ksql/ksql-server.properties

You can specify multiple parameters at the same time. For example, to configure ksql.streams.auto.offset.reset and ksql.streams.num.stream.threads:

$ KSQL_OPTS="-Dksql.streams.auto.offset.reset=earliest -Dksql.streams.num.stream.threads=1" <path-to-confluent>/bin/ksql-server-start \
  <path-to-confluent>/etc/ksql/ksql-server.properties

JMX Metrics

To enable JMX metrics, set JMX_PORT before starting the KSQL server:

$ export JMX_PORT=1099 && \
  <path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

Non-interactive (Headless) KSQL Usage

KSQL supports locked-down, “headless” deployment scenarios where interactive use of the KSQL cluster is disabled. For example, to allow a team of users to develop and verify their queries interactively on a shared testing KSQL cluster. But when deploying those queries in your production environment, you want to lock-down access to KSQL servers, version-control the exact queries, and store them in a .sql file. This will prevent users from interacting directly with the production KSQL cluster.

You can configure servers to exclusively run a predefined script (.sql file) via the --queries-file command line argument, or the ksql.queries.file setting in the KSQL configuration file. If a server is running a predefined script, it will automatically disable its REST endpoint and interactive use.

Tip

When both the ksql.queries.file property and the --queries-file argument are present, the --queries-file argument will take precedence.

To start the KSQL Server in headless, non-interactive configuration via the --queries-file command line argument:
  1. Create a predefined script and save as an .sql file.

  2. Start the KSQL with the predefined script specified via the --queries-file argument.

    $ <path-to-confluent>/bin/ksql-start-server <path-to-confluent>/etc/ksql/ksql-server.properties \
      --queries-file /path/to/queries.sql
    
To start the KSQL Server in headless, non-interactive configuration via the ksql.queries.file in the server configuration file:
  1. Configure the ksql-server.properties file. The bootstrap.servers and ksql.queries.file are required. For more information about configuration, see Configuring KSQL Server.

    # Inform the KSQL server where the Kafka cluster can be found:
    bootstrap.servers=localhost:9092
    
    # Define the location of the queries file to execute
    ksql.queries.file=/path/to/queries.sql
    
  2. Start the KSQL server with the configuration file specified.

    $ <path-to-confluent>/bin/ksql-start-server <path-to-confluent>/etc/ksql/ksql-server.properties