Configure AWS IAM OAuth for Python Client

AWS IAM-based authentication uses the OAUTHBEARER SASL mechanism and is available as an optional install extra. When running on AWS compute (EC2, EKS, ECS, Fargate, Lambda) with an IAM role, the client automatically mints and refreshes short-lived JSON Web Tokens (JWT) tokens through AWS Security Token Service (STS).

Note

The confluent-kafka[oauthbearer-aws] package is available in Python Client version 2.15 or later.

Installation

Install the AWS IAM authentication extra:

pip install 'confluent-kafka[oauthbearer-aws]==2.15.0'

This adds boto3 and related dependencies. Without the extra, no AWS dependencies are imported.

Minimal configuration

The following example shows the minimal configuration for a consumer:

from confluent_kafka import Consumer

consumer = Consumer({
    'bootstrap.servers': 'pkc-xxxx.aws.confluent.cloud:9092',
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'OAUTHBEARER',

    'sasl.oauthbearer.method': 'oidc',
    'sasl.oauthbearer.metadata.authentication.type': 'aws_iam',
    'sasl.oauthbearer.config': 'region=us-east-1,audience=https://confluent.cloud/oidc',

    'group.id': 'my-group',
    'auto.offset.reset': 'earliest',
})

The same settings apply to Producer and AdminClient.

Complete configuration example

conf = {
    'bootstrap.servers': 'pkc-xxxx.aws.confluent.cloud:9092',
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'OAUTHBEARER',

    'sasl.oauthbearer.method': 'oidc',
    'sasl.oauthbearer.metadata.authentication.type': 'aws_iam',
    'sasl.oauthbearer.config': (
        'region=us-east-1,'
        'audience=https://confluent.cloud/oidc,'
        'duration_seconds=900,'
        'signing_algorithm=ES384,'
        'sts_endpoint=https://sts-fips.us-east-1.amazonaws.com,'
        'aws_debug=console,'
        'tag_team=platform,'
        'tag_environment=prod'
    ),
    'sasl.oauthbearer.extensions': (
        'logicalCluster=lkc-abc,'
        'identityPoolId=pool-xyz'
    ),
}

Configuration keys

Required parameters

The following parameters are required in sasl.oauthbearer.config:

Key	Description
`region`	AWS region for STS calls (for example, `us-east-1`)
`audience`	OIDC audience matching broker or IAM role configuration

Optional parameters

The following parameters are optional in sasl.oauthbearer.config:

Key	Default	Description
`duration_seconds`	`300`	Token lifetime: 60-3,600 seconds. Auto-refreshes at approximately 80%.
`signing_algorithm`	`ES384`	JWT signature: `ES384` is recommended. `RS256` is also supported.
`sts_endpoint`	SDK default	Override STS endpoint. For example, FIPS or VPC.
`aws_debug`	`none`	SDK logging: `none` or `console`
`tag_<name>`		Custom claim, up to 50 total. For example, `tag_team=platform` adds `team` claim.

Configuration is a comma-separated string of key=value pairs. Whitespace around pairs is ignored, but do not add spaces around the equals sign (=). Values can contain equals signs. Escape commas with \,.

SASL extensions

Use the separate sasl.oauthbearer.extensions key:

'sasl.oauthbearer.extensions': 'logicalCluster=lkc-abc,identityPoolId=pool-xyz'

Do not place extensions in sasl.oauthbearer.config.

Prerequisites

The application runs with AWS credentials resolvable by boto3’s default chain.
The IAM role permits STS GetWebIdentityToken for the requested audience.
The audience matches the IAM role trust relationship and broker configuration.

Common issues

The following are common configuration issues:

Missing sasl.oauthbearer.method: sasl.oauthbearer.method=oidc is mandatory. Raises ValueError if omitted.
Missing SASL settings: Set both security.protocol=SASL_SSL and sasl.mechanisms=OAUTHBEARER.
Empty sasl.oauthbearer.config: sasl.oauthbearer.config must be non-empty and include the required region and audience parameters.
Missing extra: Raises ImportError if the authentication type is set but the extra is not installed.
Audience mismatch: GetWebIdentityToken is rejected at STS if the audience does not match.
Missing boto3[crt]: Might be needed for aws login credential sessions.