# Debug Streaming Agents with Confluent Cloud Streaming Agents provide debugging capabilities that give you visibility into agent behavior. You can inspect execution traces, audit tool calls, and verify output tables to test and troubleshoot agent workflows. Debugging a streaming agent workflow, in which agents and tools are defined with the [CREATE TOOL](../../flink/reference/statements/create-tool.md#flink-sql-create-tool) and [CREATE AGENT](../../flink/reference/statements/create-agent.md#flink-sql-create-agent) statements and orchestrated by [AI_RUN_AGENT](../../flink/reference/functions/model-inference-functions.md#flink-sql-ai-run-agent-function), requires a blend of observability practices, trace inspection, and output validation. This guide describes key techniques for debugging and troubleshooting these workflows systematically. For structured, ongoing monitoring of agent execution steps, you can use system log tables to capture model and tool interactions. For more information, refer to [Monitor Streaming Agents](monitor-streaming-agents.md#monitor-streaming-agents). ## Leverage built-in tracing and auditing ### Automatic trace logging Every invocation of [AI_RUN_AGENT](../../flink/reference/functions/model-inference-functions.md#flink-sql-ai-run-agent-function) and tool calls, including UDFs and external MCP services, generates system traces. These traces log the agent name, input payload, selected tools, tool outputs, and timing. Inputs and outputs are event-driven, so you can always replay an agent’s actions, because they’re backed by a Apache Kafka® topic. Agent traces provide the interaction history, which is a trace of all interactions that are logged automatically in a Kafka topic. ### Trace auditing Access trace logs by querying the system log table configured for [AI_RUN_AGENT](../../flink/reference/functions/model-inference-functions.md#flink-sql-ai-run-agent-function), or by viewing statement logs in the Confluent Cloud Console. Look for agent session information, invocation order, response times, and any error events. For more information, see [Monitor Streaming Agents](monitor-streaming-agents.md#monitor-streaming-agents) and [Statement logging](../../flink/operate-and-deploy/monitor-statements.md#flink-sql-monitor-statements-event-logging). ### What to check - Did the agent receive the correct input? - Which tools did the agent select and invoke for each input row? - Did tool outputs match expectations for type, format, and content? - Were any calls delayed, retried, or failed? ## Inspect output tables for workflow verification ### Output table monitoring Streaming agent results are inserted into output tables specified in [CREATE AGENT](../../flink/reference/statements/create-agent.md#flink-sql-create-agent). Regularly query or export these tables to validate correctness and completeness. ### Schema and content verification Ensure the output table schema matches agent definitions and tool outputs. Look for missing fields, malformed data, or unexpected results, which are indicators of context or agent prompt issues. The following statement is an example diagnostic query. ```sql SELECT * FROM processed_claims WHERE agent_result IS NULL OR agent_result LIKE '%error%' LIMIT 100; ``` ## Enable and check application logs ### Flink job logs Use Flink’s job manager/task manager logs to catch runtime errors, exceptions, warnings, or unexpected agent/tool behavior. Look for stack traces, input/output dumps, and system messages. In Confluent Cloud, you can search, filter by time range and log level, and download these logs from the **Logs** tab in the Confluent Cloud Console. For more information, see [Statement logging](../../flink/operate-and-deploy/monitor-statements.md#flink-sql-monitor-statements-event-logging). ### Custom debug logging If you’re using the Table API or custom UDFs, integrate custom logging statements inside your function code to track variables and conditions. For more information, see [Enable Logging in a User Defined Function](../../flink/how-to-guides/enable-udf-logging.md#flink-sql-enable-udf-logging). ## Replay and isolate problematic events ### Replayability Because the system audits every event and agent/tool call, you can replay input events that led to errors or suspicious outcomes by resubmitting these records into the workflow or testing them in isolation. ### Step-by-step isolation Temporarily limit the agent’s tool array to a single tool or simplify the system prompt to narrow down issues in orchestration logic. ## Tune agent prompts and tool definitions ### Prompt adjustment Many workflow errors trace back to ambiguous or overly complex system prompts. Refine prompts to make decision logic explicit and reference tool names directly. ### Tool validation Verify that tool registration with [CREATE TOOL](../../flink/reference/statements/create-tool.md#flink-sql-create-tool) correctly lists endpoint/UDF names, access parameters, and descriptions. Unused or misnamed tools may go uncalled or produce unexpected errors. ## Monitor tool invocation metrics ### Performance metrics Use Flink’s streaming metrics and built-in auditing to monitor latency, throughput, and error rates for each tool call. The statement details page in the Confluent Cloud Console shows messages in/out per minute, consumer lag, and CFU (Confluent Flink Unit) usage for the statement running your agent. For more information, see [View and monitor statements in Cloud Console](../../flink/operate-and-deploy/monitor-statements.md#flink-sql-monitor-statements-details-view) and [Statement CFU usage](../../flink/concepts/statement-cfu-metrics.md#flink-statement-cfu). ### Error-rate analysis Regularly review metrics dashboards or log summaries to detect patterns, such as higher error rates or latency spikes, which often indicate workflow bottlenecks. To set up metrics integrations with services such as Prometheus and Datadog, see [Flink metrics integrations](../../flink/operate-and-deploy/monitor-statements.md#flink-sql-monitor-metrics). ## Best practices ### Replay strategy > - Start with small, representative datasets > - Test edge cases and error conditions > - Compare multiple agent versions systematically > - Document replay results and decisions ### Debugging workflow > 1. Analyze step-by-step execution > 2. Identify root causes of issues > 3. Test fixes with replay > 4. Deploy and monitor ### Performance optimization > - Monitor tool call latencies > - Identify frequently used tools > - Optimize tool implementations > - Cache frequently accessed data > - Use appropriate timeouts ### Security and privacy > - Sanitize sensitive data in logs > - Control access to replay data > - Implement data retention policies > - Monitor for data leaks ## Troubleshooting Replay failing : - Verify session data exists - Check agent version compatibility - Ensure all dependencies are available - Check for data corruption Performance issues during replay : - Use smaller session sets - Optimize queries - Check resource availability - Consider parallel replay ## Debugging tips - Use step-by-step analysis to understand agent behavior - Compare successful and failed sessions - Look for patterns in tool call failures - Monitor resource usage during replay - Use visualization tools for complex workflows ## Related content - [Monitor Streaming Agents](monitor-streaming-agents.md#monitor-streaming-agents) - [Agent Runtime Guide](agent-runtime-guide.md#streaming-agents-runtime) - [Build and Run Streaming Agents](create-and-run-streaming-agents.md#streaming-agents-build-and-run) - [Flink AI/ML Functions](../../flink/reference/functions/model-inference-functions.md#flink-sql-model-inference-functions) - [Streaming Agents Overview](overview.md#ai-streaming-agents-overview) #### NOTE This website includes content developed at the [Apache Software Foundation](https://www.apache.org/) under the terms of the [Apache License v2](https://www.apache.org/licenses/LICENSE-2.0.html).