Throttling, Traffic Shaping, Traffic Shedding and Circuit Breakers in System Design

Brownout refers to the failure of a system under high load or unanticipated circumstances. Throttling, traffic shaping, traffic shedding and circuit breakers are techniques for maintaining system stability and averting these brownouts.

Throttling controls the rate or bandwidth at which an application processes requests, like a water tap controlling the rate of flow. By limiting the number of requests, throttling prevents system overloads, ensuring resource fairness and prioritization among users. However, when not configured correctly, throttling can lead to significant service response delays and system failures.

Traffic shaping not only controls the rate of requests but also manages the allocation of available resources. It's like a traffic officer managing vehicle flow on a busy road. Traffic shaping might involve buffering (storing) excessive requests or delaying packets to smooth out traffic bursts, ensuring a steady flow of requests that the system can handle. This, however, may result in increased latency and service degradation.

Traffic shedding is essentially rejecting incoming requests when the system is under extreme load or imminent danger of failure. It is the last resort when all other defensive mechanisms fail or are inadequate to prevent system overload. Traffic shedding helps to maintain the system's availability by discarding non-critical tasks or low-priority requests, thereby freeing up resources for more critical operations. However, traffic shedding requires a sophisticated understanding of the system's priorities to minimize the impact on essential operations and user experience.

The Circuit Breaker pattern is a type of design pattern in software development used for detecting service failure and providing an alternate service or failing gracefully instead of waiting for a response. When a system encounters repeated service failures, the circuit breaker 'trips,' 'opens' the circuit, and the system will either return an error or redirect to a fallback service without delay. The circuit breaker helps prevent the system from continually attempting a likely-to-fail action, safeguarding the failing service from further overload by incoming requests.

Throttling and traffic shaping are more preventive, controlling and managing the flow of incoming requests, ensuring the system remains stable under varying loads. In contrast, traffic shedding and the circuit breaker pattern are more reactive, coming into play when the system is under extreme pressure or when a service becomes unresponsive or fails. Throttling and traffic shaping are the first lines of defense. If the load continues to increase and the system nears its limit, traffic shedding can discard less important requests. And if a service fails or becomes unresponsive, the circuit breaker steps in to prevent this failure from affecting the system's overall performance.