Authorization Revoked¶
Using the Confluent CLI, revoke the producer’s authorization to write to the topic.
This scenario will look at Confluent Cloud metrics from the Metrics API and client metrics from the client application’s MBean object kafka.producer:type=producer-metrics,client-id=producer-1
.
Introduce failure scenario¶
Create an ACL that denies the service account permission to write to any topic, inserting your service account ID instead of
sa-123456
:confluent kafka acl create --service-account sa-123456 --operation write --topic '*' --deny
Diagnose the problem¶
From your web browser, navigate to the Grafana dashboard at http://localhost:3000 and login with the username
admin
and passwordpassword
.Navigate to the
Producer Client Metrics
dashboard. Wait 2 minutes and then observe:- The top level panel with
Record error rate
(record-error-rate
) should turn red, a major indication something is wrong. - Throughput, e.g.
Outgoing byte rate
(outgoing-byte-rate
), shows the producer is successfully sending messages to the broker. This is technically correct: the producer _is_ sending the batch of records to the cluster but they are not being written to the broker’s log because of lack of authorization.
- The top level panel with
Check the status of the Confluent Cloud cluster, specifically that it is accepting requests. Navigate to the
Confluent Cloud
dashboard.In the
Confluent Cloud
dashboard, look at the top panels, they should all be green which means the cluster is operating safely within its resources.Change the topics filter to show only
demo-topic-1
. Observe:Topic received bytes
(io.confluent.kafka.server/received_bytes
) is still high because it Confluent Cloud is still receiving the records and using network bandwidth, before they get denied due to authorization errors.Topic retained bytes
(io.confluent.kafka.server/retained_bytes
) has flattened because the records sent by the producer are not getting written to the log.Topic sent bytes
(io.confluent.kafka.server/sent_bytes
), which are the records sent to the consumer, has dropped to zero because there are no new records to send.
Check the producer logs for more information about what is going wrong. Use the following docker command to get the producer logs:
docker compose logs producer
Verify that you see log messages similar to what is shown below:
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [demo-topic-1]
Note that the logs provide a clear picture of what is going on–
org.apache.kafka.common.errors.TopicAuthorizationException
. This was expected because the failure scenario we introduced removed the ACL that permitted the service account to write to the topic.View the source code that catches this exception, ccloud-observability/src, using a
Callback()
.producer.send(new ProducerRecord<String, PageviewRecord>(topic, key, record), new Callback() { @Override public void onCompletion(RecordMetadata m, Exception e) { if (e != null) { e.printStackTrace(); } else { System.out.printf("Produced record to topic %s%n", topic); } } });
Resolve failure scenario¶
Delete the ACL created above that denied the service account permission to write to any topic. Insert your service account ID instead of
sa-123456
:confluent kafka acl delete --service-account sa-123456 --operation write --topic '*' --deny
Verify that the
org.apache.kafka.common.errors.TopicAuthorizationException
log messages stopped in theproducer
container.docker compose logs producer