Changelog

Version 5.0.0

KSQL 5.0 includes a number of new features, including:

  • User-Defined Functions (UDF).
  • Support for nested data via the new STRUCT data type.
  • Support for writing to existing streams using the new INSERT INTO statement.
  • Stream-Stream Joins.
  • Table-Table Joins.
  • Table Aggregations.
  • A revamped REST API that will be kept compatible from this release onwards.
  • Confluent Platform Docker images for KSQL Server and CLI
  • Java 10 support.

Detailed Changlog

  • PR-1610 - Specify a namespace for avro schemas
  • PR-1570 - DOCS-400: Add troubleshooting steps for KSQL server port
  • PR-1588 - Add example avro schemas to the ksql packaging build
  • PR-1529 - Add a schema inference test
  • PR-1511 - Fix NPE when printing null values
  • PR-1547 - Fix the bug for null check in comparison expression code gen,
  • PR-1525 - Add more unit tests for avro serializer
  • PR-1521 - Add unit tests for deserializing a comprehensive set of avro/connect types
  • PR-1519 - Extend the query translation test to support parametrized format and add avro to some cases
  • PR-1549 - Ensure that the list of interceptors is modifiable before mutating it
  • PR-1528 - Include example schemas in the packaging build
  • PR-1531 - Recommend the use of STRUCT instead of EXTRACTJSONFIELD
  • PR-1526 - Include monitoring interceptor config in streams configs
  • PR-1427 - Udf docs
  • PR-1522 - Reduce log level when loading blacklist to INFO from ERROR
  • PR-1524 - Java 10 support
  • PR-1527 - Drop all references to the KSQL UI
  • PR-1479 - Write configs to command topic and read back before building queries
  • PR-1512 - Data-gen now can create struct columns.
  • PR-1517 - Fix spacing in struct schema string
  • PR-1518 - Add documentation for structs
  • PR-1516 - Fix test not run due to incorrect imports
  • PR-1513 - fix incorrect imports
  • PR-1507 - Standardize on a single version for the avro-random-genator
  • PR-1481 - Consistent schema format between websocket endpoint and ksql endpoint
  • PR-1503 - Convert invalid avro field names before serializing
  • PR-1501 - sort udf names
  • PR-1500 - don’t bomb on non-existing plugin dir
  • PR-1495 - Pass schema registry client configs onto the schema registry client
  • PR-1482 - String masking functions
  • PR-1434 - Schema Translation Test + Nested Support in Query Validation Test
  • PR-1486 - list and describe updated to include aggregate functions
  • PR-1478 - Use ‘:’ for struct field reference.
  • PR-1494 - Output more info to help with debugging system tests.
  • PR-1467 - Support java based UDAFS
  • PR-1390 - Upgrade handling step 1: clean up KSQL configs
  • PR-1453 - Struct end to end
  • PR-1471 - Renable the build of the cp-ksql-server docker image
  • PR-1465 - support null literal in UDFs #1462
  • PR-1458 - add jar path to description of functions
  • PR-1457 - Simple Security manager for UDFS
  • PR-1449 - create module for UDF annotations to minimize dependencies
  • PR-1470 - Use a string for formatted time field in clickstream schema
  • PR-1448 - Add config for ext dir
  • PR-1461 - Fix some doc formatting issues.
  • PR-1463 - Handle rename of KafkaConsumerService to ConsumerService
  • PR-1455 - Rename SPAN to WITHIN everywhere
  • PR-1435 - Support List/Show Functions (UDFs) and Describe Function <name>
  • PR-1417 - Add new join types to KSQL
  • PR-1451 - KSQL-1093: Fix munged KSQL query code block
  • PR-1430 - Updated AST builder to handle dot notation.
  • PR-1442 - Simplified Security
  • PR-1445 - Prefixed output topics
  • PR-1433 - update packaging to include ext dir for udfs
  • PR-1426 - Throw exception when calling FunctionRegistry#getUdfFactory(name) if function doesn’t exist
  • PR-1429 - UDF tidy ups
  • PR-1423 - Collect metrics for UDFs
  • PR-1412 - DOCS-415: New topic on KSQL custom test data
  • PR-1416 - Validate that function names are valid java identifiers
  • PR-1428 - Minor style change in comment.
  • PR-1410 - Added query rewrite feature using AST.
  • PR-1414 - Add missing comma to fix broken REST API example
  • PR-1415 - Add year 2018 to the copyright info in the CLI banner
  • PR-1366 - Create and load Java based UDFs from an ext directory
  • PR-1362 - Return schema description API entities as JSON objects
  • PR-1400 - Add struct support for avro by using Connect Converter API
  • PR-1406 - #1396 Fix regression issue with string concat using ‘+’ operator.
  • PR-1407 - Avoid NPE if not function exists that handles required types.
  • PR-1401 - Switch to optional schema fields since we can have null values.
  • PR-1409 - Issue #1111: Document schemaRegistryURL arg for ksql-datagen in usage…
  • PR-1402 - add test for ws schema serde
  • PR-1408 - MINOR: Remove use of deprecated method removed in AK
  • PR-1383 - Using Connect data format in KSQL.
  • PR-1140 - Do schema inference before writing to output topic
  • PR-1370 - Wrap api query id strings in a class
  • PR-1368 - Using ExpressionTypeManager to detect the types for function look up.
  • PR-1379 - Disable server identification verification for SecureIntegrationTest
  • PR-1378 - Update with May 2018 preview release information
  • PR-1347 - CP KSQL CLI docker image
  • PR-1372 - Drop cp-ksql-server from jenkinsfile as well
  • PR-1371 - Remove cp-ksql-server from the build to unblock packaging
  • PR-1369 - Using the kafka topic name in delete topic message.
  • PR-1358 - Skip building jars for the ksql cp docker image.
  • PR-1363 - Print an error if a message fails to produce with datagen
  • PR-1360 - Fix checkstyle failures
  • PR-1356 - New geo distance function
  • PR-1359 - handle array subsrcipts when generating function args
  • PR-1358 - Skip building jars for the ksql cp docker image.
  • PR-1353 - Support functions with same name but different arguments
  • PR-1354 - HOTFIX: Bump POM version for cp-ksql-server
  • PR-1334 - Add ksql-examples to the class path when running ksql-datagen
  • PR-1322 - CP docker image for KSQL server
  • PR-1285 - Refactor WebSocket endpoints and add support for print topic
  • PR-1349 - Minor: Add test around extract JSON array field.
  • PR-1350 - Child first class loader for UDFs
  • PR-1344 - Add syntax ref for INSERT INTO
  • PR-1337 - Fix the way LIMIT clauses are handled
  • PR-1321 - Make FunctionRegistry an interface and make MetaStore implement it
  • PR-1340 - Post fix udf instance names to ensure they are unique.
  • PR-1341 - Fix system tests (ish)
  • PR-1335 - Don’t ignore leading spaces when saving history
  • PR-1331 - Follow-on updates for KSQL_OPTS
  • PR-1333 - Inherit maven-compiler-plugin definition from common
  • PR-1329 - Fix run class to work with Java 10 and use ExplicitGCInvokesConcurrent
  • PR-1330 - Make KSQL_OPTS more prominent
  • PR-1301 - Minor: Admin client leak
  • PR-1315 - Consumed imports
  • PR-1277 - Ksql 1217 optionally delete kafka topic with drop statement
  • PR-1309 - Add crosslink from ‘starting KSQL server’ section to the headless mode instructions
  • PR-1114 - Struct Data Type(Part-1): New struct type, DDL statements and Describe
  • PR-1306 - Fix the clickstream demo
  • PR-1307 - Rename KSQL HTTP API to KSQL REST API
  • PR-1305 - MINOR: Add query validation test for stringtotimestamp with double single quote
  • PR-1303 - The DESCRIBE ACL on the __consumer_offsets topic is not required.
  • PR-1091 - Add parent reference to ast nodes
  • PR-1296 - Remove deprecated punctuate
  • PR-1294 - Fix production config docs
  • PR-1167 - DOCS-397 - Replace hard-coded version references with variables
  • PR-1249 - Include function names in error message for unsupported table aggrega…
  • PR-1220 - Add external dependency on Avro Random Generator and remove its source code
  • PR-576 - Insert into implementation
  • PR-1256 - Better reporting of invalid serverAddress or connection issues in CLI
  • PR-1197 - KSQL-883: Add KafkaClientSupplier to KSQL API
  • PR-1278 - Add Preview Release information, update Latest News
  • PR-1219 - Improved error message for Stream/Table and Query relation correctness.
  • PR-1254 - Update clickstream to 5.0.0-beta1 (#1248)
  • PR-1242 - Work around a race condition in the test ZK instance.
  • PR-1247 - Update docker quickstart for 5.0.0-beta1
  • PR-1201 - Add versioning to the API (#1151)
  • PR-1240 - 5.0 API docs
  • PR-1210 - Fix some more rest api inconsistencies
  • PR-1136 - Table aggregations
  • PR-1163 - Minor: Fix lifecycle of AdminClient and KafkaTopicClientImpl instances.
  • PR-1208 - Add a Pull request template
  • PR-1159 - Fix Rest API redirects.
  • PR-1185 - Fix flaky test, caused by async topic creation.
  • PR-1216 - Clarify that KSQL timestamps are in milliseconds.
  • PR-1050 - Escape discovered avro field name if it is a ksql lexer token literal #(1043)
  • PR-1198 - text is the new binary
  • PR-1147 - SourceDescription cleanup + listing with descriptions
  • PR-637 - Small java doc improvement for KsqlAggregateFunction
  • PR-1183 - MINOR: Make README ksql blurb consistent with the one on 4.1.x
  • PR-1179 - Shutdown cleaning should there be an error on start up.
  • PR-1177 - Minor: Fix flakey KafkaTopicClient integration tests
  • PR-1173 - Fix flakey CliTest and issue with LIMIT clause not being honoured.
  • PR-1166 - Allow tests with Kafka cluster to be run more than once in IDE.
  • PR-1148 - Remove unused module types.
  • PR-1174 - Fix handling of table tombstones
  • PR-1171 - Add back key-constraints
  • PR-1160 - Remove .md documentation. Move docs-rst to docs
  • PR-838 - Rename ‘Kafka output topic’ to just ‘Kafka topic’ in describe ext (#838)
  • PR-1155 - Update README for KSQL 4.1
  • PR-1158 - Fix failing build
  • PR-1109 - Ksql 1054 better aggregation with complex expressions
  • PR-652 - Added referential integrity enforcement for streams/tables and queries
  • PR-756 - Add the ability to extract the record timestamp from a string field #646
  • PR-1130 - Return proper errors from the HTTP server endpoints
  • PR-1134 - Add logging to the CliTest
  • PR-1098 - Docs on configuring KSQL -> SR over HTTPS
  • PR-944 - Ksql 660 schema registry clean up
  • PR-1126 - Don’t depend on the internal ‘PlainSaslServer’ class from kafka
  • PR-1103 - Return more stuff from the rest API
  • PR-1122 - fix spelling curretnly -> currently
  • PR-1120 - update error messages when failing to parse an avro schema
  • PR-1108 - add websockets query endpoint
  • PR-1107 - Build clickstream docker img with 4.1 cp