# Catalog Grey Matter Catalog provides additional metadata about services to the [Grey Matter Dashboard](https://greymatter.gitbook.io/grey-matter-documentation/setup/sense/intelligence-360). It connects to [Grey Matter Control](https://greymatter.gitbook.io/grey-matter-documentation/setup/fabric/gm-control)) to retrieve discovered services and combines them with user driven configuration to create summarized metadata about services running within a service mesh. Although it was designed to power the Grey Matter Intelligence 360 Application, this service can be leveraged as an API to develop further visualizations of a service mesh. See our guide on [how to use Grey Matter Catalog](https://github.com/DecipherNow/gm-gitbook-sync/tree/fdbad23611cffcdb0b6cb5d9fadb49bf146ffcaa/guides/using-the-catalog-api.md) for how to interact with this service when it's up and running. ## Server Configuration Options ### Environment Variables | Name | Type | Default | Description | | --------------------------- | ------ | --------- | ---------------------------------------------------------------------------------------------------------------------------- | | `HOST` | String | "0.0.0.0" | Hostname or IP address. | | `PORT` | Number | 8080 | TCP port to listen on. | | `AUTHORIZED_USERS` | String | "" | List of PKI Distinguished Names (DNs) approved to access metrics views in the dashboard. An empty string permits all access. | | `INSTANCE_POLLING_INTERVAL` | String | "5s" | The interval at which health checks are made to service instances. | | `INSTANCE_MAX_SILENCE` | String | "15s" | The wait time before a non-responsive service instance is considered down. | | `METRICS_MAX_RETRIES` | Number | 3 | The maximum number of retries made before stopping requests to a service instance's metrics endpoint. | | `METRICS_RETRY_DELAY` | String | "10s" | The wait period between attempts to connect to a service instance's metrics endpoint. | | `METRICS_REQUEST_TIMEOUT` | String | "15s" | The wait time to indicate a request timeout from a service instance's metrics endpoint. | ### Control The Catalog service discovers service instances from Grey Matter Control and up to ten instances of Control may be configured. Each environment variable for a specific control instance should have its own index number by replacing ``. For example, `CONTROL_SERVER_0_ADDRESS`, `CONTROL_SERVER_1_ADDRESS`, etc. | Name | Type | Default | Description | | ----------------------------------------- | ------ | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CONTROL_SERVER__ADDRESS` | String | "localhost:50000" | Address for a number of Control servers. | | `CONTROL_SERVER__ZONE_NAME` | String | "region-1" | Federated zone name used for tracking, retrieving, and updating a mesh in the Catalog. Each mesh represents a collection of service clusters. | | `CONTROL_SERVER__REQUEST_CLUSTER_NAME` | String | "edge" | The cluster used in the request made to Control to retrieve discovered service instances (i.e. the edge node proxy). | | `CONTROL_SERVER_RETRY_DELAY` | String | "5s" | For each Control server, a "retry delay" can be specified. If an error occurs during creation of a client connection, establishing a stream, or sending a DiscoveryRequest, Catalog will restart the connection process. This occurs indefinitely and can only be stopped by sending a DELETE request to Control's `/zones` endpoint with the configured server's specified `zoneName`. | ### TLS In Grey Matter service mesh deployments, Transport Layer Security (TLS) is typically handled by a [Grey Matter Sidecar](https://greymatter.gitbook.io/grey-matter-documentation/setup/fabric/grey-matter-proxy) sidecar in front of the Catalog Service. In other words, the Catalog Service is treated like any other service in the service mesh, whereby security is offloaded to the sidecar proxy. However, this service can run standalone, and in these cases the following environment variables can be set to configure TLS. | Name | Type | Default | Description | | ------------------ | ------- | ------- | --------------------------------------- | | `USE_TLS` | Boolean | false | Enables TLS | | `CA_CERT_PATH` | String | "" | Path to the certificate authority file. | | `SERVER_CERT_PATH` | String | "" | Path to the server certificate. | | `SERVER_KEY_PATH` | String | "" | Path to the server certificate key. | ### Persistence | Name | Type | Default | Description | | ------------------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CONFIG_SOURCE` | String | "env" | Configures where to store and read service configuration from. Set value to `gmdata` to store in Grey Matter Data. Set value to `env` to use environment variables. | | `CONFIG_POLLING_INTERVAL` | String | "3s" | Interval for checking the store for configuration changes. Only applicable when `CONFIG_SOURCE=gmdata`. | As indicated above, there are two options for configuring services known by the Catalog service - `gmdata` and `env`. Think about your deployment model before choosing either of these options, which are highlighted below. #### Configure Persistence Grey Matter Data When the Grey Matter service mesh is deployed via our [Helm charts](https://github.com/greymatter-io/helm-charts), Catalog will already be configured to persist service configuration in an internal instance of Grey Matter Data. This is our recommended approach for production deployments because of the added security and change history provided by Grey Matter Data. Changes to services stored in Grey Matter Data are made possible via the Catalog RESTful API. Below is a list of environment variables required to use Grey Matter Data as the persistent store. Note, that these are automatically set when deploying with our [Helm charts](https://github.com/greymatter-io/helm-charts). | Name | Type | Default | Description | | ---------------------------------- | ------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `CLIENT_ADDRESS` | String | "" | Hostname or IP address of Grey Matter Data. | | `CLIENT_PORT` | String | "" | TCP port exposed by Grey Matter Data. | | `CLIENT_PREFIX` | String | "" | Grey Matter Data RESTful API prefix. | | `CLIENT_IDENTITY` | String | "" | Distinguished Name (DN) of JWT user associated with Catalog, see [Helm Charts](https://github.com/greymatter-io/helm-charts/greymatter.yaml) under `internal-jwt-users` for details. | | `CLIENT_EMAIL` | String | "" | Email address used as the directory name where services JSON will be persisted. | | `CLIENT_USE_TLS` | Boolean | false | Enable TLS. | | `CLIENT_CERT` | String | "" | base64 encoded certificate for TLS. | | `CLIENT_KEY` | String | "" | base64 encoded key for TLS. | | `CLIENT_TRUST` | String | "" | base64 encoded certificate for TLS. | | `INSECURE_TLS` | Boolean | false | | | `MONGO_USE_TLS` | Boolean | false | | | `GMDATA_STARTUP_DELAY` | String | "5s" | Delay at startup to allow all instances to write to Grey Matter Data. | | `GMDATA_ROOT_EVENT_NAME` | String | "world" | Name of the top level folder in Grey Matter Data where `services.json` will be persisted. | | `GMDATA_MAX_EVENTS` | Number | 1000 | | | `GMDATA_POLLING_INTERVAL` | String | "2s" | The interval at which Catalog will poll Grey Matter Data for updated services.json. | | `GMDATA_MAX_RETRIES` | Number | 10 | | | `GMDATA_RETRY_DELAY` | String | "1s" | The delay for retry attempts for the initial Grey Matter Data connection. | | `GMDATA_OBJECT_POLICY_TEMPLATE` | String | `(if (contains email %s)(yield-all)(yield R X)))` | Grey Matter Data object policy template to enforce permissions on `servics.json` folder. Default policy allows read and execute for the Catalog service. | | `GMDATA_POLICY_LABEL` | String | "localuser owned" | | | `GMDATA_TOP_LEVEL_FOLDER_SECURITY` | String | `{"label":"PUBLIC//EVERYBODY","foreground":"#FFFFFF","background":"#008800"}` | | | `GMDATA_SERVICES_FILE_SECURITY` | String | `{"label":"PUBLIC//EVERYBODY","foreground":"#FFFFFF","background":"#008800"` | | #### Configure Persistence with Environment Variables The second configuration option is to statically define the metadata associated with each service via environment variables. This is a quick way to get Catalog running without having to setup Grey Matter Data but there are some pitfalls with this approach. {% hint style="danger" %} **Pitfalls of Configuring Persistence with Environment Variables** If you switch to using Grey Matter Data after using environment variables then the metadata will be stored in Grey Matter Data and subsequent changes to the environment variables won't take effect. You can end up creating a disconnect between configurations if you switch back and forth. If you use the Catalog RESTful API to make configuration changes and then switch back to using environment variables, you will need to keep environment variables in sync manually with those changes. {% endhint %} The following environment variables define the additional metadata associated with each service known by Catalog. Each environment variable for a specific service instance should have its own index number by replacing ``. For example, `SERVICE_0_CLUSTER_NAME`, `SERVICE_1_CLUSTER_NAME`, etc. | Name | Type | Default | Description | | --------------------------------------- | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SERVICE__CLUSTER_NAME` | String | "" | The name of the cluster (logical group name of service instances), provided by gm-control. | | `SERVICE__ZONE_NAME` | String | "" | The name of the zone associated with the cluster, provided by gm-control. This must match the value of `CONTROL_SERVER__ZONE_NAME` defined in Control's environment variables. | | `SERVICE__NAME` | String | "" | The name of the service to be displayed in the Intelligence 360 Dashboard (make this as human friendly as possible). | | `SERVICE__VERSION` | String | "" | The semver version of the service. | | `SERVICE__OWNER` | String | "" | The name of the service owner like "Decipher" or "Cool Customer". Used to sort by Owner in the Intelligence 360 Dashboard. | | `SERVICE__CAPABILITY` | String | "" | The name of the service capability like "Storage" or "Security". Used to sort by Capability in the Intelligence 360 Dashboard. | | `SERVICE__DOCUMENTATION` | String | "" | A URL path to documentation relative to the root URL of the service or a full path depending on your needs. | | `SERVICE__PROMETHEUSJOB` | String | "" | The name of the Prometheus Job used to store and query time series data associated with this service. This **must** match the cluster name. | | `SERVICE__MIN_INSTANCES` | Number | 1 | The minimum number of instances this service should scale to. If below this number, the service will be in a warning state in the Intelligence 360 Dashboard. | | `SERVICE__MAX_INSTANCES` | Number | 1 | The maximum number of instances this service should scale to. If above this number, the service will be in a warning state in the Intelligence 360 Dashboard. | | `SERVICE__ENABLE_INSTANCE_METRICS` | Boolean | true | Enable the instance metrics view in the Intelligence 360 Dashboard. | | `SERVICE__ENABLE_HISTORICAL_METRICS` | Boolean | false | Enable the historical metrics view in the Intelligence 360 Dashboard. | | `SERVICE__METRICS_TEMPLATE` | String | "" | URL template for constructing the service's instance metrics endpoint e.g. `http://{{host}}:{{port}}/metrics`. | | `SERVICE__METRICS_PORT` | Number | 8081 | TCP port serving up service instance metrics. | ### Configuration File {% hint style="info" %} **Note:** the following is not a typical deployment model. {% endhint %} If you're running the Catalog executable binary manually you can also provide a configuration file with key/value pairs.*\*\** ``` ./gm-catalog --config=settings.toml ``` If there are conflicting configurations between this file and environment variables, then the environment variables will take precedence. ## Questions {% hint style="success" %} **Need help configuring Grey Matter Catalog?** Contact us at . {% endhint %}