Features & Support for Statements in Confluent Manager for Apache Flink

This section describes features and limitations of the current version of Flink SQL in Confluent Manager for Apache Flink (CMF).

Supported features

CMF provides support for the following Flink SQL features:

Kafka Cluster Integration:

Kafka clusters can be configured as databases. Each topic within a configured Kafka cluster is exposed as a Flink table and can be queried. Non-compacted topics are exposed as append-only tables, and compacted topics are exposed as upsert tables.

Table Schema Derivation:

Table schemas are derived from the topic’s schema information provided by Schema Registry. All formats supported by Schema Registry (Avro, JSON, Protobuf) are also supported. Topics without schema information are exposed as tables with raw BYTE columns for the topic’s key and value records.

Environment Catalog:

Each environment has a writable environment catalog in which you register user-defined functions and tables that use custom connectors with SQL DDL. See Manage Environment Catalogs in Confluent Manager for Apache Flink.

User-Defined Functions:

You can register scalar, table, and aggregate Java user-defined functions with CREATE FUNCTION and call them in statements. See User-Defined Functions in Confluent Manager for Apache Flink.

Custom Connectors and Formats:

In the environment catalog, you can use custom connectors and formats, such as CDC sources or filesystem sinks, by uploading their JARs as artifacts. See Custom Connectors and Formats in Confluent Manager for Apache Flink.

Supported Statement Types:

Following are the supported statement types:

  • LIST TABLES;

  • LIST DATABASES;

  • LIST CATALOGS;

  • SHOW CURRENT DATABASE;

  • SHOW CURRENT CATALOG;

  • DESCRIBE <table>;

  • CREATE TABLE ...;

  • ALTER TABLE ...;

  • DROP TABLE ...;

  • CREATE FUNCTION ...;

  • DROP FUNCTION ...;

  • SHOW USER FUNCTIONS;

  • SELECT ...; (only append-only results are supported)

  • INSERT INTO ...; (only append-only results are supported)

Limitations

The following limitations exist in the current version:

  • Unsupported Table DDL: CREATE TABLE AS statements are not supported.

  • Statement Complexity: Very large statements might not be supported because there is a limit on the size of the query execution plan produced by the optimizer. Since the plan size depends on many factors and the plan is also compressed, it is not possible to give clear guidance for the supported size or complexity of SQL statements. If you run into error messages regarding plan size, try to decompose the complex statement into multiple smaller statements.

  • Views: Views are not supported.

  • EXPLAIN Statements: EXPLAIN statements are not supported.

  • Custom Catalogs: You cannot register your own catalog implementations. CMF provides the built-in Kafka catalogs and a per-environment environment catalog.

Related content