April 12, 2019

Using Kiam to access AWS resources from Kubernetes pods

Applications running on EC2 instances typically use the role associated with the instance (the instance profile) to interact with AWS APIs. Using the instance’s role instead of providing credentials to the application is beneficial because role credentials are temporary and automatically rotated. When it comes to Kubernetes, however, multiple pods are typically running on the same node, which means that different applications must share the same role. Using the same role for different workloads is problematic because it does not let you scope the permissions down to the requirements of each application.

Enter Kiam

Kiam is a tool that helps solve this problem. Using Kiam, you can consume AWS services from pods running in Kubernetes while following the security principle of least privilege. After installing Kiam, pods in your cluster can use pod-specific IAM roles to interact with AWS services.

Kiam’s duties are split across two components: a server and a node agent. The server component is responsible for requesting credentials and thus needs to run on an EC2 instance that has access to the AWS STS API. Given that the server can assume any IAM role, it should run on a node that does not run user workloads (such as a Control Plane node) to reduce the attack surface of the infrastructure. The node agent, on the other hand, intercepts requests destined for the AWS metadata API and redirects them to a local HTTP server for processing. The following sequence diagram depicts the overall process.

Kiam Sequence Diagram

Using Kiam

The official documentation has a great walkthrough about deploying Kiam on Kubernetes. The one thing that can trip you up (as it did me) is the IAM configuration that needs to be done ahead of time. To use Kiam, you should create a dedicated role for the Kiam server (called kiam-server, for example). This role must 1) have permissions to call STS:AssumeRole and 2) be assumable by the EC2 instance running the Kiam server. On the flipside, the EC2 instance running the Kiam server must have permissions to assume the kiam-server role via a call to STS:AssumeRole.

Once Kiam is up and running, you need to explicitly allow pods in a namespace to assume a specific set of IAM roles. You achieve this by annotating the namespace with a regular expression:

kind: Namespace
metadata:
  name: my-application
  annotations:
    iam.amazonaws.com/permitted: "my-application-.*"

After configuring the namespace, pods in the namespace can assume the roles that have been allowed. To assign a role to a pod, you use an annotation:

kind: Pod
metadata:
  name: user-checkout-server
  namespace: my-application
  annotations:
    iam.amazonaws.com/role: my-application-user-checkout-db

When running Kubernetes on AWS, consider deploying a solution such as Kiam to provide application-specific IAM credentials to your pods. In this way, you can create roles for each application and thus follow the principle of least privilege. If you are an EKS user, you can use Kiam, but you should be aware that AWS is working on an official solution that takes a different approach. Perhaps I’ll explore it in a future post.