Set Mesh-wide Configuration

These examples show how you can apply mesh-wide policy changes using the greymatter cli and gm-control-api.

Prerequisites

1. jq command-line JSON processor

2. greymatter setup with a running Fabric mesh.

Mesh-Wide Configuration Changes

To apply a mesh-wide configuration change, we need to loop through each Grey Matter Object you wish to change and reapply a new configuration. This can be done by using the greymatter list $object and greymatter edit $object_key in tandem.

Example: Setting Circuit Breakers

This example will set circuit breaker max_connections to 500 for all clusters in the service mesh. Create a file named update.json

{
  "circuit_breakers": {
    "max_connections": 500
  }
}

This update will be merged in with the current values of each cluster object when run in the following snippet. It's best to run these commands a few times individually first, to make sure the update.json creates the expected change. In the below example, my cluster is cluster-slo-service.

greymatter get cluster cluster-slo-service > cluster-slo-service.json
jq -s '.[0] * .[1]' cluster-slo-service.json update.json > merged.json
# this is what will be appplied
cat merged.json
...
# try applying the change
greymatter edit cluster cluster-slo-service < merged.json

If you're satisfied, apply the change to the entire mesh:

#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  jq -s '.[0] * .[1]' $key.json update.json > merged.json
  greymatter edit cluster $key < merged.json
  rm $key.json merged.json
done

Example: Enable Grey Matter Observables on All Proxies

Similar to the above example, create a file update.json with the proposed update to all proxy objects:

{
  "active_proxy_filters": [
    "gm.metrics",
    "gm.observables"
  ],
  "proxy_filters": {
    "gm_observables": {
      "emitFullResponse": true,
      "useKafka": true,
      "eventTopic": "observables",
      "enforceAudit": false,
      "kafkaZKDiscover": false,
      "topic": "__REPLACE_WITH_TOPIC_NAME__",
      "kafkaServerConnection": "kafka-default.fabric.svc:9092"
    }
  }
}

Note that we must also supply any other currently active proxy filters (in this case just "gm.metrics", since the entirety of "active_proxy_filters" will be overridden. Also note that we have an attribute __REPLACE_WITH_TOPIC_NAME__. The kafka topic is specific to each proxy, so we must replace this with the proxy name for each proxy we change.

Apply these changes with a similar script as above.

#!/bin/sh

for key in $(greymatter list proxy | jq -r '.[] | .proxy_key'); do
  greymatter get proxy $key > $key.json
  # fill in topic name with the name of the proxy
  name=$(cat $key.json | jq -r '.name')
  sed 's/__REPLACE_WITH_TOPIC_NAME__/'"$name"'/g' update.json > update-$key.json
  jq -s '.[0] * .[1]' $key.json update-$key.json > merged.json
  greymatter edit proxy $key < merged.json
  rm $key.json merged.json update-$key.json
done

Example: Configuring Objects With Specific Attributes

It's often useful to be able to update objects in the mesh only with specific attributes. We can do this by modifying our shell snippet above to use jq conditionals. Note: all the shell snippets in this doc can be changed to only update on specific attributes by adding the below if block.

#!/bin/sh

for key in $(greymatter list proxy | jq -r '.[] | .proxy_key'); do
  greymatter get proxy $key > $key.json
  matches=$(cat $key.json | jq '.proxy_filters.gm_observables.eventTopic == "fabric"')
  if [ $matches = "true" ]; then
 jq -s '.[0] * .[1]' $key.json update.json > merged.json
 greymatter edit proxy $key < merged.json
  fi
  rm -rf $key.json merged.json
done

In the above example, we update only the proxies that have proxy_filters.gm_observables.eventTopic equal to "fabric".

Health Checks And Outlier Detection

Health checks and outlier detection help determine if an endpoint is healthy and our services are configured correctly. A common problem this can fix is incorrectly configured or unresponsive hosts. See envoy docs for available configurations and source code for api definition.

Health checks can be added the same way we updated the clusters above, as all we need to update is the configuration options for health_checks on each cluster. Change update.json to enable a basic health check.

{
  "health_checks": [
    {
      "timeout_msec": 1000,
      "interval_msec": 60000,
      "interval_jitter_msec": 1000,
      "unhealthy_threshold": 3,
      "healthy_threshold": 3
    }
  ]
}

Change update.json for basic outlier detection.

{
  "outlier_detection": {
    "consecutive_5xx": 3,
    "base_ejection_time_msec": 30000
  }
}

Apply this update to each cluster by running the snippet

#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  jq -s '.[0] * .[1]' $key.json update.json > $key-merged.json
  greymatter edit cluster $key < $key-merged.json
  rm $key.json $key-merged.json
done

You should now see that health checks are running. Follow the logs from any given sidecar to see that health checks have been correctly enabled.

Detect Unneeded Objects

Often there are objects "floating around" in the mesh from previous deployments or experiments. These objects can be partially detected by looping through each one and ensuring that each one was created with a name, zone_key, or other basic properties.

#!/bin/sh

for key in $(greymatter list cluster | jq -r '.[] | .cluster_key'); do
  greymatter get cluster $key > $key.json
  GREYMATTER_CONSOLE_LEVEL="none"
  possibleOutlier=$(cat $key.json | jq '.name == "" or .zone_key == "" or .instances == []')
  if [ $possibleOutlier = "true" ]; then
 echo "------ POSSIBLY UNNEEDED ------ $key "
  else
   echo "--------------OK -------------- $key "
  fi
  rm -rf $key.json merged.json
done

The above snippet will trigger if a zone_key, cluster_key, or instances attributes in any cluster is not assigned.