Rate Limiting

Rate limiting sets an upper bound on how many requests can be handled within a set period of time. If more requests come in than that, they get blocked or pushed back for later.

Think of a turnstile at an amusement park. It's built to let people through one at a time, so no matter how many people show up at once, entry only happens at a fixed pace. Computer systems work the same way. If a single pathway that requests pass through is given a cap on how many it can process per second or per minute, anything beyond that cap simply doesn't get through—regardless of why the requests piled up in the first place.

What makes this useful is that it prevents damage without needing to know the cause. For instance, if a program ends up flooding a service with requests overnight because it keeps retrying a failed task, capping the rate right at the entrance can still stop the resulting cost spike or system overload.

You'll see it used in sentences like "AWS added rate limiting to the gateway in Bedrock AgentCore." One thing that's easy to misunderstand: rate limiting doesn't check whether each request's content is valid. It only looks at how often requests come in, regardless of what's in them, and throttles that frequency—so even legitimate requests can get caught if they come in too fast.