.. title:: Operate ksqlDB for Confluent Platform .. meta:: :description: Learn how to install and operate local ksqlDB Server instances. .. _ksql_operations: Operaate |ksqldb| for |cp| ========================== This topic provides guidance and links to help you install and operate your local |ksqldb| Server. ================================================ Local Development and Testing with Confluent CLI ================================================ For development and testing purposes, you can use Confluent CLI to spin up services on a single host. For more information, see the :ref:`quickstart`. .. include:: ../includes/cli.rst :start-after: cli_limitations_start :end-before: cli_limitations_end =================================== Installing and Configuring |ksqldb| =================================== You have a number of options when you set up |ksqldb| Server. For more information on installing and configuring |ksqldb|, see the following topics. - :ref:`install_overview` - :ksqldb-docs:`Configure ksqlDB Server|operate-and-deploy/installation/server-config/` - :ksqldb-docs:`ksqlDB Configuration Parameter Reference|operate-and-deploy/installation/server-config/config-reference/` ======================================= Starting and Stopping |ksqldb| Clusters ======================================= |ksqldb| provides the following start and stop scripts: ksql-server-start This script starts the |ksqldb| server. It requires a server configuration file as an argument and is located in the ``/bin`` directory of your |cp| installation. For more information, see :ref:`start_ksql-server`. ksql-server-stop This script stops the |ksqldb| server. It is located in the ``/bin`` directory of your |cp| installation. ============= Health Checks ============= - The |ksqldb| REST API supports a "server info" request at ``http://:8088/info`` and a basic server health check endpoint at ``http://:8088/healthcheck``. - Check runtime stats for the |ksqldb| server that you are connected to using ``DESCRIBE EXTENDED`` and ``EXPLAIN ``. .. _ksql-monitoring-and-metrics: ====================== Monitoring and Metrics ====================== |ksqldb| includes JMX (Java Management Extensions) metrics, which give insights into what is happening inside your |ksqldb| servers. These metrics include the number of messages, the total throughput, throughput distribution, error rate, and more. .. include:: includes/ksql-includes.rst :start-after: enable_JMX_metrics_start :end-before: enable_JMX_metrics_end For more information about Kafka Streams metrics, see :ref:`streams_monitoring`. ================= Capacity Planning ================= The :ksqldb-docs:`Capacity Planning guide|operate-and-deploy/capacity-planning/` describes how to size your |ksqldb| clusters. =============== Troubleshooting =============== ------------------------------------ SELECT query hangs and doesn’t stop? ------------------------------------ Queries in |ksqldb|, including non-persistent queries such as ``SELECT * FROM myTable EMIT CHANGES``, are continuous streaming queries. Streaming queries will not stop unless explicitly terminated. To terminate a non-persistent query in the |ksqldb| CLI you must type ``Ctrl + C``. -------------------------------------------------- No results from ``SELECT * FROM`` table or stream? -------------------------------------------------- This is typically caused by the query being configured to process only newly arriving data instead, and no new input records are being received. To fix, do one of the following: - Run ``SET 'auto.offset.reset' = 'earliest';``. For more information, see :ksqldb-docs:`Configure ksqlDB CLI|operate-and-deploy/installation/cli-config/` and :ksqldb-docs:`Configure ksqlDB Server|operate-and-deploy/installation/server-config/`. - Write new records to the input topics. ------------------------------------------------------------ Can’t create a stream from the output of windowed aggregate? ------------------------------------------------------------ |ksqldb| doesn't support structured keys, so you can't create a stream from a windowed aggregate. ---------------------------------------------- |ksqldb| doesn’t clean up its internal topics? ---------------------------------------------- Make sure that your |ak-tm| cluster is configured with ``delete.topic.enable=true``. For more information, see :platform:`deleteTopics|clients/javadocs/javadoc/org/apache/kafka/clients/admin/AdminClient.html`. ------------------------------------------------ |ksqldb| CLI doesn’t connect to |ksqldb| server? ------------------------------------------------ The following warning may occur when you start the |ksqldb| CLI. .. code:: bash **************** WARNING ****************** Remote server address may not be valid: Error issuing GET to KSQL server Caused by: java.net.SocketException: Connection reset Caused by: Connection reset ******************************************* Also, you may see a similar error when you create a |ksqldb| query by using the CLI. .. code:: bash Error issuing POST to KSQL server Caused by: java.net.SocketException: Connection reset Caused by: Connection reset In both cases, the CLI can't connect to the |ksqldb| server, which may be caused by one of the following conditions: - |ksqldb| CLI isn't connected to the correct |ksqldb| server port. - |ksqldb| server isn't running. - |ksqldb| server is running but listening on a different port. Check the port that |ksqldb| CLI is using ----------------------------------------- Ensure that the |ksqldb| CLI is configured with the correct |ksqldb| server port. By default, the server listens on port ``8088``. For more info, see :ref:`Starting the ksqlDB CLI `. Check the |ksqldb| server configuration --------------------------------------- In the |ksqldb| server configuration file, check that the list of listeners has the host address and port configured correctly. Look for the ``listeners`` setting: .. code:: bash listeners=http://0.0.0.0:8088 Or if you are running over IPv6: :: listeners=http://[::]:8088 For more info, see :ref:`Starting ksqlDB Server `. Check for a port conflict ------------------------- There may be another process running on the port that the |ksqldb| server listens on. Use the following command to check the process that's running on the port assigned to the |ksqldb| server. The following example checks the default port, which is ``8088``. .. code:: bash netstat -anv | egrep -w .*8088.*LISTEN Your output should resemble: .. code:: bash tcp4 0 0 *.8088 *.* LISTEN 131072 131072 46314 0 In this example, ``46314`` is the PID of the process that's listening on port ``8088``. Run the following command to get info on the process: .. code:: bash ps -wwwp Your output should resemble: .. code:: bash io.confluent.ksql.rest.server.KsqlServerMain ./config/ksql-server.properties If the ``KsqlServerMain`` process isn't shown, a different process has taken the port that ``KsqlServerMain`` would normally use. Check the assigned listeners in the |ksqldb| server configuration, and restart the |ksqldb| CLI with the correct port. -------------------------------------------------------- Exception when |ksqldb| CLI connects to |ksqldb| server? -------------------------------------------------------- The following exception may occur when you start the |ksqldb| CLI. :: [2023-06-02 01:38:24,295] ERROR (io.confluent.ksql.cli.Cli) java.lang.NullPointerException: at index 2 at com.google.common.collect.ObjectArrays.checkElementNotNull(ObjectArrays.java:225) ... This exception is caused by a mismatch between the |ksqldb| CLI version and the |ksqldb| server version. |ksqldb| doesn't guarantee CLI / server compatibility across versions. ------------------------------------------------ Replicated topic with Avro schema causes errors? ------------------------------------------------ Confluent Replicator renames topics during replication, and if there are associated Avro schemas, they aren't automatically matched with the renamed topics. In the |ksqldb| CLI, the ``PRINT`` statement for a replicated topic works, which shows that the Avro schema ID exists in |sr|, and |ksqldb| can deserialize the Avro message. But ``CREATE STREAM`` fails with a deserialization error: .. code:: bash CREATE STREAM pageviews_original (viewtime bigint, userid varchar, pageid varchar) WITH (kafka_topic='pageviews.replica', value_format='AVRO'); [2018-06-21 19:12:08,135] WARN task [1_6] Skipping record due to deserialization error. topic=[pageviews.replica] partition=[6] offset=[1663] (org.apache.kafka.streams.processor.internals.RecordDeserializer:86) org.apache.kafka.connect.errors.DataException: pageviews.replica at io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:97) at io.confluent.ksql.serde.connect.KsqlConnectDeserializer.deserialize(KsqlConnectDeserializer.java:48) at io.confluent.ksql.serde.connect.KsqlConnectDeserializer.deserialize(KsqlConnectDeserializer.java:27) The solution is to register schemas manually against the replicated subject name for the topic: .. code:: bash # Original topic name = pageviews # Replicated topic name = pageviews.replica curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data "{\"schema\": $(curl -s http://localhost:8081/subjects/pageviews-value/versions/latest | jq '.schema')}" http://localhost:8081/subjects/pageviews.replica-value/versions -------------------------- Check |ksqldb| server logs -------------------------- If you're still having trouble, check the |ksqldb| server logs for errors: .. code:: bash confluent log ksql-server Look for logs in the default directory at ``/usr/local/logs`` or in the ``LOG_DIR`` that you assign when you start the |ksqldb| CLI. For more info, see :ref:`Starting the ksqlDB CLI `. =============== Related content =============== - `Video: Taking KSQL to Production `__