# Observables The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This [JSON payload](#example-payload) contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc. Observable publishing defaults to `stdout` but can also be published to a Kafka topic or location on disk. ## Filter Configuration Options | Name | Type | Default | Description | | ----------------------- | ------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `logLevel` | String | Info | Verbosity of logging: Warn, Info, Debug | | `fileName` | String | blank | Name of file to write events to. (stdout if blank) | | `emitFullResponse` | Boolean | false | \*Show response body in the observable object | | `topic` | String | "" | Sets the `eventType` field of the observable; used to sort and group messages by service/region/environment/etc in later analysis. | | `useKafka` | Boolean | false | Publish observable message to a Kafka topic | | `enforceAudit` | Boolean | false | Block requests until an observable has been successfully published to Kafka (Only applies if `useKafke=true`) | | `encryptionAlgorithm` | String | "" | Type of encryption. Must be 'aes' or blank | | `encryptionKey` | String | "" | Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32. | | `encryptionKeyID` | uint32 | 0 | User supplied number to identify the key used in encryption | | `eventTopic` | String | "" | The Kafka topic that will hold the published observable messages | | `kafkaZKDiscover` | Boolean | false | Kafka will be discovered through a zookeeper node | | `kafkaServerConnection` | String | "" | Comma delimited list of Kafka addresses, or if `kafkaZKDiscover` is `true`, a list of ZooKeeper addresses | | `useKafkaTLS` | Boolean | false | Enable TLS communication to the supplied kafka brokers | | `kafkaCAs` | String | "" | List of file URLs that point to trusts to be used when connecting to kafka | | `kafkaCertificate` | String | "" | File URL pointing to certificate to use when connecting to kafka over TLS | | `kafkaCertificateKey` | String | "" | File URL pointing to certificate key to use when connecting to kafka | | `kafkaServerName` | String | "" | Certificate server name to use when connecting to kafka | ### Payloads {% hint style="warning" %} **\*Note on Full Response Payloads** Turning on full response payloads (`emitFullResponse=true`) can cause significant amounts of data to be written to the observable. **The observables filter will attempt to write the entire response**, whether it's a: * Large data file * Full-frame JPEG * Even a multi gigabyte MP4 video This can use significant storage resources. Make sure your provisioned infrastructure can support this filter. {% endhint %} #### Key ID and Dynamic Key Rollover Users can roll over the encryption key dynamically by changing the Observables configuration in the Proxy. To enable convenient decryption, each key should be assigned a unique key ID. #### Example Configuration ```yaml http_filters: - name: gm.observables config: emitFullResponse: true useKafka: false enforceAudit: false logLevel: debug encryptionAlgorithm: aes encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ= encryptionKeyID: 1 ``` ### Frames {% hint style="info" %} **About Frames** When published in a file (or `stdout`), the (possibly encrypted) JSON payload is packaged in `Frames`. {% endhint %} | Version | Size | Key ID | Payload | | ------- | ------- | ------- | ------------ | | 1 byte | 7 bytes | 8 bytes | (Size) bytes | {% hint style="info" %} #### Example * Version = 1 * Size = 0x00033e = 830 * Key ID = 0x0000002a = 42 ```bash hexdump -C example_events example.json 00000000 01 00 03 3e 00 00 00 2a 4e 7a 7e ea 06 0d 4a 37 |...>...*Nz~...J7| 00000010 81 b4 fa 37 f6 54 23 cb 39 1a 9b d7 5a c3 9d b6 |...7.T#.9...Z...| 00000020 b6 bd d2 c3 fa ff 07 b2 2a 0f 5b 92 2f bb 58 2d |........*.[./.X-| ``` {% endhint %} #### Kafka Headers We normally write the key id to Kafka Record Headers. Such headers are only available after Kafka Version 0.11. {% hint style="info" %} **Have an older version of Kafka?** Avoid errors by using a `key id` of zero. This means you cannot roll over keys dynamically. {% endhint %} #### Example Payload ```javascript { "eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a", "eventChain": [ "01fad8f2-eb80-11e9-9e69-0a580a82033a" ], "schemaVersion": "1.0", "originatorToken": null, "eventType": "example-service", "timestamp": 1570727063, "systemIp": "10.130.3.58", "action": "GET", "payload": { "isSuccessful": true, "request": { "endpoint": "/user/me", "headers": { ":authority": "localhost:8443", ":method": "GET", ":path": "/user/me", "content-length": "0", "user-agent": "Wget", "x-envoy-original-path": "/services/example-service/1.0/user/me", "x-forwarded-proto": "http", "x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482" } }, "response": { "code": 200, "headers": { ":status": "200", "content-length": "4", "content-type": "text/plain; charset=utf-8", "date": "Thu, 10 Oct 2019 17:04:23 GMT", "x-envoy-upstream-service-time": "1" }, "body": "{user: sample-user}\n" } }, "event_mapping": { "type": "EventAccess", "action": "ACCESS" }, "time_audited": "20191010T170423.317408" } ``` ### Per-Route configuration ```javascript { "emitFullResponse": } ``` See [Routing](https://greymatter.gitbook.io/grey-matter-documentation/1.3/usage/traffic_control/routing). ## Questions {% hint style="success" %} **Need help configuring your observables filter?** Contact our team at: [Grey Matter Support](https://support.greymatter.io/support/home). {% endhint %}