Circuit Breakers

Circuit breaking options allow the enforcement of limits on the number of connections and requests that a cluster of upstream hosts can receive. Circuit breakers allow a quick and early detection of failures and prevents wasting resources waiting for timed out requests or retrying requests endlessly on a failing system. Setting these limits can protect clusters against sudden surges of traffic. Grey Matter supports these limits through Envoy's circuit breaking options set on the cluster object.

Configuration Options

The circuit breaking configuration options are:

Circuit breaker	Description	Type	Default
max_connections	Maximum number of connections that will be established to all instances in a cluster	int	1024
max_pending_requests	Maximum number of pending requests that Envoy will allow to the upstream cluster	int	1024
max_requests	Maximum number of parallel requests that Envoy will make to the upstream cluster	int	1024
max_retries	Maximum number of parallel retries that Envoy will allow to the upstream cluster	int	3
max_connection_pools	Maximum number of connection pools per cluster that Envoy will concurrently support at once	int	unlimited
track_remaining	If true, publishes stats that expose the number of resources remaining until the circuit breakers open	bool	false

By default, no circuit breakers are set on the cluster. The default values shown above are set when "circuit_breakers" in the cluster object is not equal to null. I.e., for a default circuit breaker use "circuit_breakers": {}.

Grey Matter Configuration

There is an option to configure up to two sets of circuit breaking thresholds on each cluster. The first threshold is the default threshold, which will automatically be configured for normal traffic to the cluster. The second is the option to configure a set of thresholds for routes with "high" routing priority.

Default

To configure a single set of circuit breaker options for the cluster, the circuit_breakers object set on the cluster should look like the below:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": null
  }
}

High priority

Grey Matter also provides the option to tailor circuit breakers for high priority traffic. To enable this feature, the high field in the circuit_breakers object must be configured. If left at the default ("high": null) all traffic will use the default circuit breaker. A properly configured circuit breaker for high priority traffic will look something like the following:

{
  "circuit_breakers": {
    "max_connections": 1,
    "max_pending_requests": 1,
    "max_retries": 1,
    "max_requests": 1,
    "max_connection_pools": 1,
    "track_remaining": true,
    "high": {
      "max_connections": 5,
      "max_pending_requests": 5,
      "max_retries": 5,
      "max_requests": 5,
      "max_connection_pools": 1,
      "track_remaining": true
    }
  }
}

With this configuration of a set of high priority circuit breakers, two different circuit breaking thresholds will be created. The first will apply to traffic with default priority, and the second will only apply to priority: HIGH traffic. To specify which routes to the cluster should be using the default threshold or the high priority threshold, specify "high_priority": true on the Grey Matter Route object.