# Mesh-Wide Configuration

These examples show how you can apply mesh-wide policy changes using the Grey Matter CLI and Control API.

## Prerequisites

1. [jq](https://stedolan.github.io/jq/) command-line JSON processor
2. [greymatter](https://greymatter.gitbook.io/grey-matter-documentation/1.7-beta/installation/commands-cli) setup with a running Fabric mesh.

## Mesh-Wide Configuration Changes

To apply a mesh-wide configuration change, we need to loop through each Grey Matter object you wish to change and reapply a new configuration. This can be done by using the `greymatter list $object` and `greymatter edit $object_key` in tandem.

### Example: Setting Circuit Breakers

This example will set circuit breaker `max_connections` to `500` for all clusters in the service mesh. Create a file named `update.json` with the following contents.

```javascript
{
  "circuit_breakers": {
    "max_connections": 500
  }
}
```

This update will be merged in with the current values of each cluster object when run in the following snippet. It's best to run these commands a few times individually first, to make sure the `update.json` creates the expected change. In the below example, my cluster is `cluster-slo-service`.

```bash
greymatter get cluster cluster-slo-service > cluster-slo-service.json
jq -s '.[0] * .[1]' cluster-slo-service.json update.json > merged.json
greymatter edit cluster cluster-slo-service < merged.json
```

If you're satisfied, apply the change to the entire mesh.

```bash
#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  jq -s '.[0] * .[1]' $key.json update.json > merged.json
  greymatter edit cluster $key < merged.json
  rm $key.json merged.json
done
```

#### Example: Enable Grey Matter Observables on All Proxies

Similar to the above example, create a file `update.json` with the proposed update to all `proxy` objects.

```javascript
{
  "active_proxy_filters": [
    "gm.metrics",
    "gm.observables"
  ],
  "proxy_filters": {
    "gm_observables": {
      "emitFullResponse": true,
      "useKafka": true,
      "eventTopic": "observables",
      "enforceAudit": false,
      "kafkaZKDiscover": false,
      "topic": "__REPLACE_WITH_TOPIC_NAME__",
      "kafkaServerConnection": "kafka-default.fabric.svc:9092"
    }
  }
}
```

Note that we must also supply any other currently active proxy filters (in this case just `"gm.metrics"`, since the entirety of `"active_proxy_filters"` will be overridden. Also note that we have an attribute `__REPLACE_WITH_TOPIC_NAME__`. The kafka topic is specific to each proxy, so we must replace this with the proxy name for each proxy we change.

Apply these changes with a similar script as above.

```bash
#!/bin/sh

for key in $(greymatter list proxy | jq -r '.[] | .proxy_key'); do
  greymatter get proxy $key > $key.json
  # fill in topic name with the name of the proxy
  name=$(cat $key.json | jq -r '.name')
  sed 's/__REPLACE_WITH_TOPIC_NAME__/'"$name"'/g' update.json > update-$key.json
  jq -s '.[0] * .[1]' $key.json update-$key.json > merged.json
  greymatter edit proxy $key < merged.json
  rm $key.json merged.json update-$key.json
done
```

#### Example: Configuring Objects With Specific Attributes

It's often useful to be able to update objects in the mesh only with specific attributes. We can do this by modifying our `shell` snippet above to use [jq conditionals](https://stedolan.github.io/jq/manual/#ConditionalsandComparisons). *Note: all the shell snippets in this doc can be changed to only update on specific attributes by adding the below `if` block*.

```bash
#!/bin/sh

for key in $(greymatter list proxy | jq -r '.[] | .proxy_key'); do
  greymatter get proxy $key > $key.json
  matches=$(cat $key.json | jq '.proxy_filters.gm_observables.eventTopic == "fabric"')
  if [ $matches = "true" ]; then
    jq -s '.[0] * .[1]' $key.json update.json > merged.json
    greymatter edit proxy $key < merged.json
  fi
  rm -rf $key.json merged.json
done
```

In the above example, we update only the proxies that have `proxy_filters.gm_observables.eventTopic` equal to `"fabric"`.

## Health Checks And Outlier Detection

Health checks and outlier detection help determine if an endpoint is healthy and our services are configured correctly. A common problem this can fix is incorrectly configured or unresponsive hosts. See the full [Health Checks guide](https://greymatter.gitbook.io/grey-matter-documentation/1.7-beta/guides/fabric-guides/health-check) or dive into the [interactive Envoy lab](https://www.envoyproxy.io/try/detecting-service-health-with-healthchecks) for more details.

Health checks can be added the same way we updated the clusters above, as all we need to update is the configuration options for `health_checks` on each `cluster`. Change `update.json` to enable a basic health check.

```javascript
{
  "health_checks": [
    {
      "timeout_msec": 1000,
      "interval_msec": 60000,
      "interval_jitter_msec": 1000,
      "unhealthy_threshold": 3,
      "healthy_threshold": 3
    }
  ]
}
```

Change `update.json` for basic outlier detection.

```javascript
{
  "outlier_detection": {
    "consecutive_5xx": 3,
    "base_ejection_time_msec": 30000
  }
}
```

Apply this update to each `cluster` by running the below snippet.

```bash
#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  jq -s '.[0] * .[1]' $key.json update.json > $key-merged.json
  greymatter edit cluster $key < $key-merged.json
  rm $key.json $key-merged.json
done
```

You should now see that health checks are running. Follow the logs from any given sidecar to see that health checks have been correctly enabled.

## Detect Unneeded Objects

Often there are objects "floating around" in the mesh from previous deployments or experiments. These objects can be partially detected by looping through each one and ensuring that each one was created with a name, zone\_key, or other basic properties.

```bash
#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  GREYMATTER_CONSOLE_LEVEL="none"
  possibleOutlier=$(cat $key.json | jq '.name == "" or .zone_key == "" or .instances == []')
  if [ $possibleOutlier = "true" ]; then
    echo "------ POSSIBLY UNNEEDED ------ $key "
  else
    echo "--------------OK -------------- $key "
  fi
  rm -rf $key.json merged.json
done
```

The above snippet will trigger if a `zone_key`, `cluster_key`, or `instances` attributes in any cluster is not assigned.