Observables

The Observables Filter configures the Proxy to emit a JSON payload with every request made to the microservice. This JSON payload contains a variety of different information about the request being made, as well as the user/system issuing the request. These Observables can then be aggregated to perform analysis like: audits, user-experience tracking, etc.

Observable publishing defaults to stdout but can also be published to a Kafka topic or location on disk.

Filter Configuration Options

Name	Type	Default	Description
logLevel	String	Info	Verbosity of logging: Warn, Info, Debug
fileName	String	blank	Name of file to write events to. (stdout if blank)
emitFullResponse	Boolean	false	*Show response body in the observable object
topic	String	""	Sets the eventType field of the observable; used to sort and group messages by service/region/environment/etc in later analysis.
useKafka	Boolean	false	Publish observable message to a Kafka topic
enforceAudit	Boolean	false	Block requests until an observable has been successfully published to Kafka (Only applies if useKafka=true)
encryptionAlgorithm	String	""	Type of encryption. Must be 'aes' or blank
encryptionKey	String	""	Must be blank or base 64 encoded string of 16, 24, or 32 bytes. We recommend 32.
encryptionKeyID	uint32	0	User supplied number to identify the key used in encryption
eventTopic	String	""	The Kafka topic that will hold the published observable messages
kafkaZKDiscover	Boolean	false	Kafka will be discovered through a zookeeper node
kafkaServerConnection	String	""	Comma delimited list of Kafka addresses, or if kafkaZKDiscover is true, a list of ZooKeeper addresses
useKafkaTLS	Boolean	false	Enable TLS communication to the supplied kafka brokers
kafkaCAs	String	""	List of file URLs that point to trusts to be used when connecting to kafka
kafkaCertificate	String	""	File URL pointing to certificate to use when connecting to kafka over TLS
kafkaCertificateKey	String	""	File URL pointing to certificate key to use when connecting to kafka
kafkaServerName	String	""	Certificate server name to use when connecting to kafka

Payloads

*Note on Full Response Payloads

Turning on full response payloads (emitFullResponse=true) can cause significant amounts of data to be written to the observable.

The observables filter will attempt to write the entire response, whether it's a:

• Large data file

• Full-frame JPEG

• Even a multi gigabyte MP4 video

This can use significant storage resources. Make sure your provisioned infrastructure can support this filter.

Key ID and Dynamic Key Rollover

Users can roll over the encryption key dynamically by changing the Observables configuration in the Proxy.

To enable convenient decryption, each key should be assigned a unique key ID.

Example Configuration

http_filters:
- name: gm.observables
  config:
    emitFullResponse: true
    useKafka: false
    enforceAudit: false
    logLevel: debug
    encryptionAlgorithm: aes
    encryptionKey: kvTujluRwliCWBWQvvvIxQr2Fxw3tY4cNCfkdlEobtQ=
    encryptionKeyID: 1

Frames

About Frames

When published in a file (or stdout), the (possibly encrypted) JSON payload is packaged in Frames.

Version	Size	Key ID	Payload
1 byte	7 bytes	8 bytes	(Size) bytes

Example

• Version = 1

• Size = 0x00033e = 830

• Key ID = 0x0000002a = 42

hexdump -C example_events example.json
00000000  01 00 03 3e 00 00 00 2a  4e 7a 7e ea 06 0d 4a 37  |...>...*Nz~...J7|
00000010  81 b4 fa 37 f6 54 23 cb  39 1a 9b d7 5a c3 9d b6  |...7.T#.9...Z...|
00000020  b6 bd d2 c3 fa ff 07 b2  2a 0f 5b 92 2f bb 58 2d  |........*.[./.X-|

Kafka Headers

We normally write the key id to Kafka Record Headers. Such headers are only available after Kafka Version 0.11.

Have an older version of Kafka? Avoid errors by using a key id of zero. This means you cannot roll over keys dynamically.

Example Payload

{
  "eventId": "01fad8f2-eb80-11e9-9e69-0a580a82033a",
  "eventChain": [
    "01fad8f2-eb80-11e9-9e69-0a580a82033a"
  ],
  "schemaVersion": "1.0",
  "originatorToken": null,
  "eventType": "example-service",
  "timestamp": 1570727063,
  "systemIp": "10.130.3.58",
  "action": "GET",
  "payload": {
    "isSuccessful": true,
    "request": {
      "endpoint": "/user/me",
      "headers": {
        ":authority": "localhost:8443",
        ":method": "GET",
        ":path": "/user/me",
        "content-length": "0",
        "user-agent": "Wget",
        "x-envoy-original-path": "/services/example-service/1.0/user/me",
        "x-forwarded-proto": "http",
        "x-request-id": "e63a82a7-e3b8-4ead-a8c5-af45d5a65482"
      }
    },
    "response": {
      "code": 200,
      "headers": {
        ":status": "200",
        "content-length": "4",
        "content-type": "text/plain; charset=utf-8",
        "date": "Thu, 10 Oct 2019 17:04:23 GMT",
        "x-envoy-upstream-service-time": "1"
      },
      "body": "{user: sample-user}\n"
    }
  },
  "event_mapping": {
    "type": "EventAccess",
    "action": "ACCESS"
  },
  "time_audited": "20191010T170423.317408"
}

Questions

Need help configuring your observables filter? Contact our team at: Grey Matter Support.