Demo Blog Post
Overview & Defensive Context#
In our morning threat analysis (AWS EKS Pod Identity: Deconstructing Workload Token Interception), we dissected the internal architecture of Amazon EKS Pod Identity and the underlying eks-pod-identity-agent DaemonSet. By replacing the public OpenID Connect (OIDC) federation used in legacy IAM Roles for Service Accounts (IRSA) with an in-cluster link-local HTTP broker listening at 169.254.170.23:80, AWS delivered significant developer ergonomics.
However, this transition introduced a concentrated attack surface at the cluster network layer:
- Link-Local Broker Probing: An unauthenticated HTTP endpoint on every worker node brokering AWS Security Token Service (STS) credentials via
eks-auth:AssumeRoleForPodIdentity. - Projected Token Replay: Static JSON Web Tokens (JWTs) mounted at
/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-tokenthat can be exfiltrated via Local File Inclusion (LFI) or path traversal and replayed against169.254.170.23:80. - The Missing Condition Trap: IAM role trust policies delegating authority to
pods.eks.amazonaws.comwithout strictaws:SourceArnconstraints, exposing roles to cross-cluster and cross-account assumption. - Namespace Multi-Tenancy Breakout: Loose Kubernetes RBAC permissions enabling unprivileged cluster tenants to schedule workloads bound to high-privilege ServiceAccounts.
graph TD
subgraph "THE EKS WORKLOAD IDENTITY THREAT MODEL"
direction TB
item_1["<b>Attack Vector</b><br/>Projected Token Theft & Link-Local Broker Replay"]
item_2["<b>Broker Endpoint</b><br/>http://169.254.170.23:80/v1/credentials"]
item_3["<b>Cloud Visibility</b><br/>'Legitimate' AssumeRoleForPodIdentity in CloudTrail"]
item_4["<b>Blast Radius</b><br/>Full Cloud IAM Role Takeover from Kubernetes Pod"]
item_1 --> item_2
item_2 --> item_3
item_3 --> item_4
endWhy Default Kubernetes & Cloud Perimeter Controls Fail#
Standard enterprise security configurations frequently fail to prevent workload credential theft in EKS environments due to four common architectural blind spots:
Default-Allow Kubernetes Egress: By default, Kubernetes network implementations (including the Amazon VPC CNI plugin) enforce a default-allow policy for outbound container traffic. Every container running on a worker node has direct, uninhibited network access to host link-local addresses—including EC2 Instance Metadata (
169.254.169.254) and EKS Pod Identity (169.254.170.23). Unprivileged pods that require zero AWS access can freely probe the credential broker.Permissive Token File System Permissions: When the EKS admission webhook projects the
eks-pod-identity-tokenvolume into a pod, it applies default file permissions of0644(defaultMode: 420). Any non-root user or compromised web application worker process (e.g.,www-data,nginx,node) can read the token directly from disk without requiring container root escalation.CloudTrail Telemetry Indistinguishability: When an adversary replays a stolen token to
169.254.170.23, the link-local agent on the worker node forwards the request to the regionaleks-authendpoint. AWS CloudTrail logs a successfulAssumeRoleForPodIdentityevent originating from the node's Elastic Network Interface (ENI) IP address. Because the request originates from an authentic cluster node using a valid cryptographic token, standard SIEM rules perceive the assumption as routine microservice activity.Host Network Sharing (
hostNetwork: true): Pods configured withhostNetwork: truebypass the container network namespace entirely, sharing the host network stack with theeks-pod-identity-agentDaemonSet. A compromised host-network pod can bind to node interfaces, sniff unencrypted HTTP traffic directed to169.254.170.23:80, and capture bearer tokens passed by sibling workloads.
[!WARNING] Securing EKS Pod Identity requires synchronizing controls across both AWS IAM and the Kubernetes control plane. Hardening IAM trust policies without enforcing Kubernetes NetworkPolicies leaves link-local endpoints vulnerable to in-cluster token replay.
Architecture Hardening#
A resilient defense against EKS Pod Identity exploitation requires a four-tiered defense-in-depth model: IAM Trust Policy Hardening, Link-Local NetworkPolicy Isolation, Kubernetes Admission Control, and Cross-Layer Telemetry Auditing.
Defense-in-Depth Architecture Flowchart#
flowchart TD
classDef pod fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#93c5fd
classDef gate fill:#2d1b06,stroke:#f59e0b,stroke-width:2px,color:#fde68a
classDef block fill:#450a0a,stroke:#dc2626,stroke-width:2px,color:#fca5a5
classDef pass fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#a7f3d0
classDef aws fill:#1e1b4b,stroke:#8b5cf6,stroke-width:2px,color:#ddd6fe
ClientPod[Application Pod in Kubernetes]:::pod --> NetPolCheck{Tier 1: Kubernetes NetworkPolicy Egress Check}:::gate
NetPolCheck -- Pod Lacks Approved Pod-Identity Label --x Drop1[Action: Egress Packet to 169.254.170.23 Dropped]:::block
NetPolCheck -- Authorized Workload with Label --> LinkLocalAgent[Tier 2: Node Agent 169.254.170.23:80]:::pass
LinkLocalAgent --> TokenValidation{Tier 3: EKS Auth Token Verification}:::gate
TokenValidation -- Token Forged / Expired / Invalid Audience --x Drop2[Action: HTTP 403 Forbidden Returned]:::block
TokenValidation -- Valid Token Signed by Cluster JWKS --> IAMTrustEval{Tier 4: AWS IAM Trust Policy Evaluation}:::aws
IAMTrustEval -- Missing or Mismatched aws:SourceArn --x Drop3[Action: STS Access Denied 403]:::block
IAMTrustEval -- Explicit Match on Association ARN --> STSIssuance[Tier 5: STS Issues 15-Min Scoped Credentials]:::pass
STSIssuance --> CloudResource[Access Allowed: S3 / DynamoDB / Secrets Manager]:::aws
LinkLocalAgent -. System Call Telemetry .-> FalcoEngine[Tier 6: Falco eBPF Runtime Detection Engine]:::gate
FalcoEngine --> SOCAlert[SOC Alert: Anomalous Process Querying 169.254.170.23]:::blockLayer 1: Cryptographic IAM Trust Policy Hardening#
The primary defense against cross-cluster and cross-workload privilege escalation is enforcing strict condition keys in the IAM role's trust policy.
Under EKS Pod Identity, trust policies delegate assumption authority to the AWS service principal pods.eks.amazonaws.com. Without conditions, any EKS cluster in the AWS account can assume the role.
Hardened IAM Trust Policy Spec
Every IAM role configured for EKS Pod Identity must enforce:
aws:SourceArn: Bound strictly to the exact Amazon Resource Name (ARN) of the specificpodidentityassociation.aws:SourceAccount: Restricting assumption to the originating AWS Account ID.- Capped Session Duration: Limiting
DurationSecondsto 900 seconds (15 minutes) to minimize the replay window of exfiltrated credentials.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnforceStrictEksPodIdentityConstraints",
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Condition": {
"StringEquals": {
"aws:SourceAccount": "123456789012"
},
"ArnEquals": {
"aws:SourceArn": "arn:aws:eks:us-east-1:123456789012:podidentityassociation/prod-cluster/a-1b2c3d4e5f6g7h8i9"
}
}
}
]
}[!IMPORTANT] Never use wildcard asterisks (
*) inaws:SourceArnfor high-privilege roles. Binding the IAM role to the unique ID of the Pod Identity Association (a-1b2c3d4e5f6g7h8i9) ensures that even if a developer creates a matching ServiceAccount name in another cluster or namespace, the role cannot be assumed.
Layer 2: Kubernetes NetworkPolicy Link-Local Isolation#
To prevent unauthorized pods or compromised containers from probing 169.254.170.23:80, deploy egress-filtering Kubernetes NetworkPolicies using Calico, Cilium, or the Amazon VPC CNI.
Cluster-Wide Default-Deny Link-Local Policy
Apply a namespace-level NetworkPolicy that drops all outbound traffic to link-local IP addresses (169.254.0.0/16), explicitly blocking both EC2 IMDS (169.254.169.254) and EKS Pod Identity (169.254.170.23), unless the pod carries an approved label:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: isolate-eks-link-local-endpoints
namespace: production
spec:
podSelector:
matchExpressions:
- key: eks.amazonaws.com/pod-identity
operator: NotIn
values: ["allowed"]
policyTypes:
- Egress
egress:
# Allow all standard traffic EXCEPT link-local IP space
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 169.254.169.254/32 # Block EC2 IMDS
- 169.254.170.23/32 # Block EKS Pod Identity Agent
- fd00:ec2::23/128 # Block EKS Pod Identity IPv6
# Ensure DNS resolution remains functional
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53With this policy active, any unprivileged pod or web application compromised via SSRF cannot establish a TCP connection to 169.254.170.23:80.
Layer 3: Admission Control & Host Network Restrictions#
Because the eks-pod-identity-agent runs as a DaemonSet with hostNetwork: true, any container sharing the host network namespace can snoop or hijack link-local traffic.
Kyverno Policy: Ban hostNetwork on User Workloads
Deploy an admission policy using Kyverno or OPA Gatekeeper that rejects pod creation requests specifying hostNetwork: true, except for approved infrastructure DaemonSets:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: disallow-host-network-namespaces
annotations:
policies.kyverno.io/category: Pod Security Standards
policies.kyverno.io/severity: high
spec:
validationFailureAction: Enforce
background: true
rules:
- name: validate-host-network
match:
any:
- resources:
kinds:
- Pod
exclude:
any:
- resources:
namespaces:
- kube-system
validate:
message: "Pods are prohibited from sharing the host network namespace (hostNetwork: true)."
pattern:
spec:
=(hostNetwork): falseLayer 4: Hardening Token Volume Permissions#
Ensure that application manifests restrict the permissions of projected volumes, preventing arbitrary non-root users from reading the projected JWT token:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 10001
fsGroup: 10001
containers:
- name: application
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/run/secrets/pods.eks.amazonaws.com/serviceaccount
name: eks-pod-identity-token
readOnly: true
volumes:
- name: eks-pod-identity-token
projected:
defaultMode: 256 # POSIX 0400: Read-only strictly for container owner
sources:
- serviceAccountToken:
audience: "pods.eks.amazonaws.com"
expirationSeconds: 3600
path: eks-pod-identity-tokenSetting defaultMode: 256 (0400) ensures that only the process running with UID 10001 can read the token file, blocking unprivileged helper processes.
Production Detection Queries#
Security Operations Centers (SOCs) and cloud threat hunters must deploy multi-layered detection rules across AWS CloudTrail management events, Kubernetes audit logs, and runtime eBPF sensors.
1. Production Sigma Rule: Anomalous EKS Pod Identity Role Assumption (CloudTrail)#
This Sigma rule detects unexpected AssumeRoleForPodIdentity API calls in AWS CloudTrail, focusing on assumptions originating outside known cluster node IP ranges or targeting sensitive administrative roles:
title: Suspicious AWS EKS Pod Identity Role Assumption
id: b4f198c2-3e7a-4901-8b2e-9d3f1a8c7042
status: production
description: |
Detects anomalous invocations of the eks-auth:AssumeRoleForPodIdentity API in AWS CloudTrail.
Flags role assumptions originating from unexpected IP addresses, anomalous user agents,
or targeting high-privilege administrative IAM roles.
references:
- https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
- https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/
tags:
- attack.credential_access
- attack.t1552
- attack.privilege_escalation
- attack.t1078.004
logsource:
product: aws
service: cloudtrail
detection:
selection_event:
eventSource: 'eks-auth.amazonaws.com'
eventName: 'AssumeRoleForPodIdentity'
selection_high_priv:
requestParameters.roleArn|contains:
- 'AdministratorAccess'
- 'PowerUser'
- 'ClusterAdmin'
- 'OrganizationAccountAccessRole'
filter_known_subnets:
# Whitelist your known EKS worker node VPC CIDR blocks
sourceIPAddress|cidr:
- '10.0.0.0/16'
- '172.20.0.0/16'
condition: selection_event and (selection_high_priv or not filter_known_subnets)
fields:
- eventTime
- sourceIPAddress
- userAgent
- requestParameters.clusterName
- requestParameters.roleArn
- responseElements.assumedRoleUser.assumedRoleId
falsepositives:
- Authorized automated deployments utilizing administrative roles during initial cluster bootstrapping.
level: high2. Falco Behavioral Runtime Rule: Unauthorized Connection to Link-Local Credential Agent#
Using Falco with the eBPF kernel driver, detect when a container that lacks Pod Identity credentials attempts to establish an outbound TCP connection to 169.254.170.23:80:
- rule: Unauthorized Connection to EKS Pod Identity Agent
desc: Detects outbound network connections to link-local IP 169.254.170.23 from unauthorized containers
condition: >
evt.type = connect and evt.dir = < and
fd.rip = "169.254.170.23" and fd.rport = 80 and
not k8s.pod.label["eks.amazonaws.com/pod-identity"] = "allowed" and
not proc.pname in (eks-pod-identity-agent, aws-k8s-agent)
output: >
ALERT: Unauthorized container connection to EKS Pod Identity Agent
(pod=%k8s.pod.name ns=%k8s.ns.name command=%proc.cmdline pid=%proc.pid
remote_ip=%fd.rip remote_port=%fd.rport container_id=%container.id)
priority: CRITICAL
tags: [k8s, aws, pod_identity, network, mitre_credential_access]3. Microsoft Sentinel KQL Rule: Temporary Pod Identity Credentials Used Outside EKS Cluster#
When temporary STS credentials issued to an EKS pod are exfiltrated, adversaries execute AWS CLI calls from their external machines or non-EKS cloud infrastructure. This KQL query detects STS sessions generated by AssumeRoleForPodIdentity that execute cloud actions from external IP addresses:
// Microsoft Sentinel KQL: Detect Stolen EKS Pod Identity Credentials Used Outside Cluster
let Lookback = 24h;
// 1. Gather all AssumeRoleForPodIdentity issuance events
let PodIdentityAssumptions = AWSCloudTrail
| where TimeGenerated >= ago(Lookback)
| where EventSource == "eks-auth.amazonaws.com" and EventName == "AssumeRoleForPodIdentity"
| extend AssumedRoleId = tostring(parse_json(ResponseElements).assumedRoleUser.assumedRoleId)
| extend ClusterName = tostring(parse_json(RequestParameters).clusterName)
| extend NodeIP = SourceIpAddress
| project IssuanceTime = TimeGenerated, AssumedRoleId, ClusterName, NodeIP, RoleArn = tostring(parse_json(RequestParameters).roleArn);
// 2. Identify cloud activity executed by the assumed role sessions
AWSCloudTrail
| where TimeGenerated >= ago(Lookback)
| where EventSource != "eks-auth.amazonaws.com"
| extend SessionId = tostring(split(UserIdentityArn, "/")[1])
| join kind=inner (
PodIdentityAssumptions
) on $left.SessionId == $right.AssumedRoleId
// 3. Flag activity where SourceIpAddress does not match any known EKS node IP
| where SourceIpAddress != NodeIP and SourceIpAddress !startswith "10." and SourceIpAddress !startswith "172.20."
| project ActivityTime = TimeGenerated,
SessionId,
RoleArn,
EventSource,
EventName,
ExternalAttackerIP = SourceIpAddress,
UserAgent,
ClusterName,
NodeIP,
AlertReason = "CRITICAL: EKS Pod Identity temporary credentials used from an external, non-node IP address (Stolen Token Replay)"Enterprise Mitigation Matrix#
When evaluating security engineering controls for EKS Pod Identity, security architects must weigh operational deployment effort against residual risk:
| Remediation Strategy | Implementation Effort | Blast Radius & Operational Risk | Detection & Prevention Efficacy | Performance Overhead | Architectural Longevity |
|---|---|---|---|---|---|
| Workaround: Kubernetes NetworkPolicy Egress Block | 1 - 2 Hours (Apply namespace NetworkPolicy blocking 169.254.170.23/32) | Low Unassigned pods blocked from link-local; authorized pods require explicit label. | High (90%) Prevents unauthorized pods from reaching the node agent over HTTP. | Zero Enforced in-kernel via eBPF / iptables. | Interim Barrier Essential defense while auditing IAM role trust policies. |
Hotfix: Restrict IAM Trust Policies (aws:SourceArn) | 2 - 4 Hours (Update IAM role trust policies with exact association ARNs) | Low - Moderate If association ARN is misconfigured, pod role assumption will fail. | Absolute (100%) Eliminates cross-cluster and cross-account role assumption entirely. | Zero Evaluated in AWS STS control plane. | Mandatory Standard Aligns with AWS IAM Well-Architected security best practices. |
| Architecture Fix: Kyverno Admission Control & Seccomp | 1 - 2 Weeks (Enforce hostNetwork: false, read-only token permissions, seccomp) | Moderate Requires validation across third-party Helm charts and DaemonSets. | Comprehensive (100%) Eliminates local sniffing, token file harvesting, and container breakouts. | Zero Native container runtime and Linux kernel enforcement. | Permanent State Establishes a hardened, zero-trust Kubernetes workload identity tier. |
[!IMPORTANT] The single most critical step in securing EKS Pod Identity is adding the
aws:SourceArncondition to the IAM role trust policy. Omitting this condition allows any workload across any cluster in your AWS organization to assume the role if they discover its ARN.
Incident Response & Verification Playbook#
If telemetry indicates that an EKS pod was compromised or an alert flags anomalous AssumeRoleForPodIdentity activity, execute the following four-phase incident response playbook immediately:
flowchart TD
classDef phase fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#93c5fd
classDef action fill:#450a0a,stroke:#dc2626,stroke-width:2px,color:#fca5a5
classDef verify fill:#064e3b,stroke:#10b981,stroke-width:2px,color:#6ee7b7
classDef warn fill:#2d1b06,stroke:#f59e0b,stroke-width:2px,color:#fde68a
P1["Phase 1: CloudTrail Triage & Session Identification"]:::phase --> P1_Action["Identify Stolen AssumedRoleId, RoleArn & External Source IP"]:::warn
P1_Action --> P2["Phase 2: Immediate Containment & Credential Revocation"]:::phase
P2 --> P2_Step1["1. Apply RevokeOlderSessions Inline Policy to IAM Role"]:::action
P2_Step1 --> P2_Step2["2. Delete Compromised Pod Identity Association in EKS"]:::action
P2_Step2 --> P2_Step3["3. Quarantine Compromised Kubernetes Pod / Worker Node"]:::action
P2_Step3 --> P3["Phase 3: Network Isolation & Token Invalidation"]:::phase
P3 --> P3_Step1["1. Deploy Default-Deny NetworkPolicy for 169.254.170.23/32"]:::action
P3_Step1 --> P3_Step2["2. Terminate Pod to Force New Projected Token Generation"]:::action
P3_Step2 --> P4["Phase 4: Post-Remediation Verification & Hardening"]:::phase
P4 --> P4_Verify["Confirm Revocation Active, Audit IAM Trust Policy & Test Egress Block"]:::verifyPhase 1: CloudTrail Scoping & Session Identification#
- Identify the Compromised STS Session: Query AWS CloudTrail to locate the exact session ID and assumed role ARN:
# Search CloudTrail for the recent AssumeRoleForPodIdentity event
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleForPodIdentity \
--max-results 20 | jq '.Events[] | {EventTime, Username, CloudTrailEvent: (.CloudTrailEvent | fromjson)}'- Extract Active Session Identifiers: Record the
AccessKeyId(starting withASIA...) and theassumedRoleIdfrom the event response.
Phase 2: Immediate Containment & Credential Invalidation#
- Revoke Active STS Sessions Immediately: AWS STS credentials cannot be explicitly deleted, but their permissions can be revoked instantly by attaching an inline policy to the IAM role denying all actions for sessions issued before the current timestamp:
# Attach an immediate session revocation policy to the compromised IAM role
CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
cat << EOF > revoke-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"DateLessThan": {
"aws:TokenIssueTime": "${CURRENT_TIME}"
}
}
}
]
}
EOF
aws iam put-role-policy \
--role-name CompromisedPodRole \
--policy-name RevokeActivePodSessions \
--policy-document file://revoke-policy.json
echo "[+] Successfully revoked all active STS sessions issued before ${CURRENT_TIME}."- Delete the EKS Pod Identity Association: Sever the link between the Kubernetes ServiceAccount and the IAM role:
# Delete the specific pod identity association
aws eks delete-pod-identity-association \
--cluster-name prod-cluster \
--association-id a-1b2c3d4e5f6g7h8i9Phase 3: Workload Quarantine & Network Isolation#
- Isolate the Compromised Pod: Apply network isolation labels to the pod to drop all ingress and egress traffic:
# Label pod for quarantine network policy
kubectl label pod payment-processor-78b9c-4x92m -n production quarantine=true --overwrite- Apply Default-Deny Link-Local NetworkPolicy: Ensure no further pods in the namespace can access
169.254.170.23/32or169.254.169.254/32.
Phase 4: Post-Remediation Verification & Health Checklist#
Before returning the cluster and IAM roles to operational status:
- Verify STS Invalidation: Test an API call using the exfiltrated
ASIA...access key and verify that AWS returnsExplicitDenybyRevokeActivePodSessions. - Audit IAM Role Trust Policies: Inspect all roles trusting
pods.eks.amazonaws.comand verify that each role contains an explicitaws:SourceArncondition pointing to a valid association ARN:
aws iam get-role --role-name CompromisedPodRole --query 'Role.AssumeRolePolicyDocument' | jq .- Verify NetworkPolicy Egress Dropping: Deploy a test pod without the
allowedlabel in the namespace and verify thatcurl -m 3 http://169.254.170.23/v1/credentialstimes out. - Verify Kyverno Admission Enforcement: Attempt to deploy a test pod with
hostNetwork: truein user namespaces and confirm that the admission controller rejects the request. - Confirm Falco & Sentinel Alerting: Verify that the Sentinel KQL query and Falco eBPF rules are active in production.
Authoritative References#
- Amazon Web Services (AWS) Security Guidance: Amazon EKS Pod Identity: a new way for applications on EKS to obtain IAM credentials — AWS Containers Blog
- Datadog Security Labs: Deep dive into the new Amazon EKS Pod Identity feature — Datadog Security Labs Research
- AWS Official Documentation: Pod Identity IAM Role Trust Policy Conditions — AWS EKS User Guide
- Wiz Research: EKS Access Entries and Pod Identity Create New Privilege Escalation Paths — Wiz Academy
- Trend Micro Research: An Investigation of AWS Credential Exposure via Overprivileged Containers and Link-Local APIs — Trend Micro Research
Comments
Post a Comment