0%
AWS in Practice: Services, Limits, and Failure Modes
Compute
Storage
Databases
Networking
Delivery and Visibility
Limits, Cost and Recovery
STS, Roles and Workload Identity
The previous lesson treated the principal as a given. This one is about how a piece of software comes to be that principal in the first place, which is the question of credentials. A process running on an instance, in a container, in a Lambda function or on a CI runner has to present something to AWS that proves who it is. For most of AWS's history the default answer was an access key, a long lived pair of strings that never expired unless someone rotated it. The modern answer is a role session, a set of credentials that the AWS Security Token Service issues on demand and that stop working on their own, usually within an hour.
The difference between the two is not a detail of implementation. It decides what happens on the worst day, when a credential ends up somewhere it should not be.
What an access key really is
An IAM user access key is two strings: an access key ID beginning with AKIA and a forty character secret access key. The SDK uses the secret to sign each request with Signature Version 4, and AWS verifies the signature against its copy. Nothing about the key expires. A key created in 2017 and pasted into a configuration file still works today, unless someone deactivated it. Each user can have at most two keys, which exists so that rotation can overlap: create the second key, move every consumer to it, then deactivate and delete the first.
In practice rotation rarely happens on schedule, because the consumers of a key are scattered across laptops, build servers, third party tools and old scripts, and nobody has a list. Keys therefore tend to live for years, accumulate permissions as new use cases are bolted on, and leak in predictable ways: committed to a public repository, printed in a build log, included in a container image layer, copied into a ticket, or stolen from a developer laptop. Automated scanners watch public GitHub pushes for the AKIA prefix and test any key they find within minutes. AWS runs its own scanning too and attaches a quarantine policy, AWSCompromisedKeyQuarantineV2, to users whose keys it finds exposed, but that is a safety net rather than a control you design around.
What a session credential is
A session credential has three parts rather than two: an access key ID beginning with ASIA, a secret, and a session token, plus an expiry time. All three must be presented together, and once the expiry passes AWS rejects requests signed with them. The Security Token Service issues them, and every way of becoming a role ends in an STS call: AssumeRole, AssumeRoleWithWebIdentity, AssumeRoleWithSAML, or a service such as EC2 or Lambda calling STS for you behind the scenes.
The expiry is what changes the economics of a leak. If a session credential is copied out of a build log, the attacker has until the credential expires, by default one hour, and nothing longer. The credential also carries a session name and, if used, a source identity, so CloudTrail shows exactly which session was abused. Compare that with a leaked access key, which works until a human notices and deactivates it, and which identifies only the user it belongs to, not which of its dozen consumers leaked it.
The credential provider chain
Software rarely loads credentials explicitly. Every AWS SDK has a credential provider chain, an ordered list of places to look, and it uses the first one that yields credentials. The exact order differs slightly between SDKs, but for most it is: explicit credentials passed in code, environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN), web identity token settings (AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN), the shared configuration and credentials files with any profile, credential_process or IAM Identity Center settings in them, the container credentials endpoint used by ECS and EKS Pod Identity, and finally the EC2 instance metadata service.
Two consequences follow. First, the most specific source wins, so a stray AWS_ACCESS_KEY_ID left in a container's environment overrides the task role the platform team carefully configured, and the process silently runs as whatever user that key belongs to. Second, application code should not need to know where its credentials come from. Code that constructs a client with no explicit credentials runs correctly on a laptop with an Identity Center profile, on EC2 with an instance profile, in Lambda with an execution role and in a pod with Pod Identity. Code that reads keys from its own configuration file works in exactly one place and leaks in all the others.
The SDK also handles refresh. Session credentials from the instance metadata service, the container endpoint or a web identity token are refreshed automatically before they expire, typically five to fifteen minutes before. Applications that cache a client for the lifetime of the process are fine. Applications that extract the raw credentials once at startup and pass the strings around break when the first set expires, which is a common failure in code ported from the access key era.
Checking who you are
Every credential source eventually answers the same question, and there is one API call that asks it directly. sts:GetCallerIdentity returns the account, the ARN and the unique ID of whatever principal signed the request, and it needs no permission at all, so it cannot be denied by any policy. Running aws sts get-caller-identity is the first step in debugging almost any credential problem, because it tells you whether the chain found what you expected before you start reading policies. Applications can log the result once at startup, which turns the silent wrong identity failure described above into a line in the logs that says exactly which principal the process is using.
The same property makes it the first call an attacker makes with a stolen credential, which is why a burst of GetCallerIdentity calls from an unfamiliar address is a useful detection signal, as the last section of this lesson shows.
What is left for long lived keys
A short list of cases still has no clean alternative to a long lived credential, and it keeps getting shorter. Some third party SaaS tools only accept access keys, although most now support assuming a role with an external ID. Some on premises workloads cannot reach an identity provider, which IAM Roles Anywhere now addresses with X.509 certificates. And break glass access, the emergency path when the identity provider itself is down, may use an IAM user with hardware MFA locked in a safe. Outside those, an organisation can and should aim for zero IAM user access keys, enforced by an SCP that denies iam:CreateAccessKey except for a named exception path.
Signing, clocks and the session token size
Two practical details trip up teams moving to session credentials. Signature Version 4 includes a timestamp, and AWS rejects requests whose timestamp is more than five minutes from its own clock, so hosts with drifting clocks see sudden SignatureDoesNotMatch or RequestExpired errors that look like credential problems. Keep NTP running, which on EC2 means the Amazon Time Sync Service at 169.254.169.123. And session tokens are long, often over a thousand characters, and longer still with session tags and session policies. Older tools that store credentials in fixed width fields, or proxies that cap header sizes, truncate them and produce baffling authentication failures.
None of this means session credentials are harmless when they leak. An hour is plenty of time for an automated attacker to copy data or create a persistent foothold, such as a new IAM user with its own key, if the role allows it. Expiry bounds the window, the role's permissions bound what can happen within it, and detection decides whether anyone notices. All three matter, and the rest of this lesson takes each in turn.
Rotation policies, secret scanners and quarantine policies all exist to shorten the life of a credential after it leaks. A role session does that by construction. Every workload that moves from an access key to a role session turns an unbounded exposure into one measured in minutes, and gains a session name in CloudTrail that tells you which workload it was.