Important

You are viewing documentation for an older version of Confluent Platform. For the latest, click here.

Configuring KSQL Server

KSQL configuration parameters can be set for KSQL server and queries as well as for the underlying Kafka Streams and Kafka Clients (producer and consumer).

Tip

These instructions assume you are installing Confluent Platform by using ZIP or TAR archives. For more information, see On-Premises Deployments.

Setting KSQL Server Parameters

You can specify KSQL server configuration parameters by using the server configuration file (ksql-server.properties) or the KSQL_OPTS environment variable. Properties set with KSQL_OPTS take precedence over those specified in the KSQL configuration file. A recommended approach is to configure a common set of properties using the KSQL configuration file and override specific properties as needed, using the KSQL_OPTS environment variable.

Tip

If you deploy Confluent Platform by using Docker containers, you can specify configuration parameters as environment variables to the KSQL Server image. For more information, see Install KSQL with Docker.

KSQL Server Configuration File

By default the KSQL server configuration file is located at <path-to-confluent>/etc/ksql/ksql-server.properties. The file follows the syntax conventions of Java properties files.

<property-name>=<property-value>

For example:

bootstrap.servers=localhost:9092
listeners=http://localhost:8088

After you have updated the server configuration file, you can start the KSQL server with the configuration file specified.

$ <path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

For more information, see KSQL Configuration Parameter Reference.

KSQL_OPTS Environment Variable

You can override KSQL server configuration parameters by using the KSQL_OPTS environment variable. The properties are standard Java system properties. For example, to set ksql.streams.num.streams.threads to 1:

$ KSQL_OPTS="-Dksql.streams.num.streams.threads=1" <path-to-confluent>/bin/ksql-server-start \
  <path-to-confluent>/etc/ksql/ksql-server.properties

You can specify multiple parameters at the same time. For example, to configure ksql.streams.auto.offset.reset and ksql.streams.num.stream.threads:

$ KSQL_OPTS="-Dksql.streams.auto.offset.reset=earliest -Dksql.streams.num.stream.threads=1" <path-to-confluent>/bin/ksql-server-start \
  <path-to-confluent>/etc/ksql/ksql-server.properties

KSQL Server Runtime Environment Variables

When KSQL Server starts, it checks for shell environment variables that control the host Java Virtual Machine (JVM). Set the following environment variables to control options like heap size and Log4j configuration. These settings are applied by the ksql-run-class shell script when KSQL Server starts.

KSQL_CLASSPATH

Path to the Java deployment of KSQL Server and related Java classes. The following command shows an example KSQL_CLASSPATH setting.

export CLASSPATH=/usr/share/java/my-base/*:/usr/share/java/my-ksql-server/*:/opt/my-company/lib/ksql/*:$CLASSPATH
export KSQL_CLASSPATH="${CLASSPATH}"
KSQL_LOG4J_OPTS

Specifies KSQL Server logging options by using the Log4j configuration settings. The following example command sets the default Log4j configuration.

export KSQL_LOG4J_OPTS="-Dlog4j.configuration=file:$KSQL_CONFIG_DIR/log4j-rolling.properties"

For more information, see Log4j Configuration.

KSQL_JMX_OPTS

Specifies KSQL metrics options by using Java Management Extensions (JMX). The following example command sets the default JMX configuration.

export KSQL_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "

For more information, see Monitoring and Management Using JMX Technology.

KSQL_HEAP_OPTS

Specifies the initial size and maximum size of the JVM heap for the KSQL Server process. The following example command sets the initial size and maximum size to 15GB.

export KSQL_HEAP_OPTS="-Xms15G -Xmx15G"

For more information, see JRockit JVM Heap Size Options.

KSQL_JVM_PERFORMANCE_OPTS

Specifies performance tuning options for the JVM that runs KSQL Server. The following example command sets the default JVM configuration.

export KSQL_JVM_PERFORMANCE_OPTS="-server -XX:+UseConcMarkSweepGC -XX:+CMSClassUnload ingEnabled -XX:+CMSScavengeBeforeRemark -XX:+ExplicitGCInvokesConcurrent -XX:New Ratio=1 -Djava.awt.headless=true"

For more information, see D Command-Line Options.

JMX_PORT

Specifies the port that JMX uses to report metrics.

export JMX_PORT=1099
JAVA_HOME

Specifies the location of the java executable file.

export JAVA_HOME=<jdk-install-directory>

JMX Metrics

To enable JMX metrics, set JMX_PORT before starting the KSQL server:

export JMX_PORT=1099 && \
<path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

Run the ksql-print-metrics tool to see the available JMX metrics for KSQL.

<path-to-confluent>/bin/ksql-print-metrics

Your output should resemble:

_confluent-ksql-default_bytes-consumed-total: 926543.0
_confluent-ksql-default_num-active-queries: 4.0
_confluent-ksql-default_ksql-engine-query-stats-RUNNING-queries: 4
_confluent-ksql-default_ksql-engine-query-stats-NOT_RUNNING-queries: 0
_confluent-ksql-default_messages-consumed-min: 0.0
_confluent-ksql-default_messages-consumed-avg: 29.48784732897881
_confluent-ksql-default_num-persistent-queries: 4.0
_confluent-ksql-default_ksql-engine-query-stats-ERROR-queries: 0
_confluent-ksql-default_num-idle-queries: 0.0
_confluent-ksql-default_messages-consumed-per-sec: 105.07699698626074
_confluent-ksql-default_messages-produced-per-sec: 11.256903025105757
_confluent-ksql-default_error-rate: 0.0
_confluent-ksql-default_ksql-engine-query-stats-PENDING_SHUTDOWN-queries: 0
_confluent-ksql-default_ksql-engine-query-stats-REBALANCING-queries: 0
_confluent-ksql-default_messages-consumed-total: 10503.0
_confluent-ksql-default_ksql-engine-query-stats-CREATED-queries: 0
_confluent-ksql-default_messages-consumed-max: 100.1243737430132

The following table describes the available KSQL metrics.

JMX Metric Description
bytes-consumed-total Number of bytes consumed across all queries.
error-rate Number of messages that have been consumed but not processed across all queries.
messages-consumed-avg Average number of messages consumed by a query per second.
messages-consumed-per-sec Number of messages consumed per second across all queries.
messages-consumed-min Number of messages consumed per second for the query with the fewest messages consumed per second.
messages-consumed-max Number of messages consumed per second for the query with the most messages consumed per second.
messages-consumed-total Number of messages consumed across all queries.
messages-produced-per-sec Number of messages produced per second across all queries.
num-persistent-queries Number of persistent queries that are currently executing.
num-active-queries Number of queries that are actively processing messages.
num-idle-queries Number of queries with no messages available to process.

Non-interactive (Headless) KSQL Usage

KSQL supports locked-down, “headless” deployment scenarios where interactive use of the KSQL cluster is disabled. For example, the CLI enables a team of users to develop and verify their queries interactively on a shared testing KSQL cluster. But when you deploy these queries in your production environment, you want to lock down access to KSQL servers, version-control the exact queries, and store them in a .sql file. This prevents users from interacting directly with the production KSQL cluster. For more information, see Headless Deployment.

You can configure servers to exclusively run a predefined script (.sql file) via the --queries-file command line argument, or the ksql.queries.file setting in the KSQL configuration file. If a server is running a predefined script, it will automatically disable its REST endpoint and interactive use.

Tip

When both the ksql.queries.file property and the --queries-file argument are present, the --queries-file argument will take precedence.

To start the KSQL Server in headless, non-interactive configuration via the --queries-file command line argument:

  1. Create a predefined script and save as an .sql file.

  2. Start the KSQL with the predefined script specified via the --queries-file argument.

    <path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties \
--queries-file /path/to/queries.sql
To start the KSQL Server in headless, non-interactive configuration via the ksql.queries.file in the server configuration file:

  1. Configure the ksql-server.properties file. The bootstrap.servers and ksql.queries.file are required. For more information about configuration, see Configuring KSQL Server.

    # Inform the KSQL server where the Kafka cluster can be found:
bootstrap.servers=localhost:9092

# Define the location of the queries file to execute
ksql.queries.file=/path/to/queries.sql

  2. Start the KSQL server with the configuration file specified.

    <path-to-confluent>/bin/ksql-server-start <path-to-confluent>/etc/ksql/ksql-server.properties

Schema resolution

When you run a KSQL application that uses Avro, KSQL infers schemas from Schema Registry automatically, but the behavior after restarting KSQL Server differs between interactive and non-interactive mode.

  • Interactive mode: after KSQL Server restarts, it doesn’t contact Schema Registry again to resolve schemas, because it has previously persisted the information to the command topic.
  • Non-interactive mode: after KSQL Server restarts, it does contact Schema Registry again to resolve schemas. If schemas have changed, unexpected behavior in your ksqlDB applications may occur.

Important

If your KSQL applications use Avro and you run them in non-interactive mode, ensure that the schemas don’t change between KSQL Server restarts, or provide the schema explicitly. If the schema may evolve, it’s safer to provide the schema explicitly.