Amazon Managed Streaming for Apache Kafka (Amazon MSK)

Amazon MSK is a fully managed Apache Kafka service hosted by AWS. Hub backend instance can be set easily to use AWS MSK by defining the standard spring kafka properties. See the sample properties in following sections.

Access MSK with no authentication and no encryption

If MSK is provisioned without any authentication and encryption, by default the access protocol is defined as plain-text. In such case, it's enough to set only bootstrap servers in application.yml as below.

application.yml

spring:
  kafka:
    bootstrap-servers: b-1.test.kafka.ap-east-1.amazonaws.com:9092,b-2.test.kafka.ap-east-1.amazonaws.com:9092

Access MSK with IAM role-based authentication and encryption

If MSK is provisioned with IAM role-based authentication and encryption (within the cluster and between clients and brokers), use the properties below for accessing the service. Make sure the IAM role which is assigned to the backend instance container tasks
has sufficient MSK permissions as stated here in IAM access control.

Additionally, trust relationships should be defined as part of the same permission set for the specific instance. See the examples below.

application.yml

spring:
  kafka:
    bootstrap-servers: b-1.test.kafka.ap-east-1.amazonaws.com:9098,b-2.test.kafka.ap-east-1.amazonaws.comm:9098
    security.protocol: 'SASL_SSL'
    ssl:
      trust-store-location: 'file:/security/cacerts-zenoo.jks'
      trust-store-password: '***'
    properties:
      sasl:
        jaas.config: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
        mechanism: 'AWS_MSK_IAM'
        client.callback.handler.class: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'

You may need to specify the IAM role explicitly within the JAAS config. Example value with debug mode:

spring:
  kafka:
    bootstrap-servers: b-1.test.kafka.ap-east-1.amazonaws.com:9098,b-2.test.kafka.ap-east-1.amazonaws.comm:9098
    security.protocol: 'SASL_SSL'
    ssl:
      trust-store-location: 'file:/security/cacerts-zenoo.jks'
      trust-store-password: '***'
    properties:
      sasl:
        jaas.config: 'software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::123456:role/ecsTaskExecutionRole" awsRoleSessionName="producer" awsStsRegion="eu-west-1" awsDebugCreds=true;'
        mechanism: 'AWS_MSK_IAM'
        client.callback.handler.class: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'

Sample policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DescribeTransactionalId",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:ReadData",
                "kafka-cluster:WriteDataIdempotently",
                "kafka-cluster:WriteData",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:Connect"
            ],
            "Resource": "arn:aws:kafka:eu-west-1:123456:cluster/test-eu-west-1-msk/*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTransactionalId",
                "kafka-cluster:AlterTransactionalId"
            ],
            "Resource": "arn:aws:kafka:eu-west-1:123456:transactional-id/test-eu-west-1-msk/*/*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:*Topic",
                "kafka-cluster:ReadData",
                "kafka-cluster:WriteData"
            ],
            "Resource": "arn:aws:kafka:eu-west-1:123456:topic/test-eu-west-1-msk/*"
        },
        {
            "Sid": "VisualEditor3",
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Resource": "arn:aws:kafka:eu-west-1:123456:group/test-eu-west-1-msk/*"
        }
    ]
}

Sample trust relationship configuration

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:sts::917319201960:assumed-role/ecsInstanceRole/i-0e6c0393f595d39e0"
                ],
                "Service": [
                    "ecs-tasks.amazonaws.com",
                    "ec2.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}