Configure Audits

Configure audits in Grey Matter Fabric.

The auditing capability of the Grey Matter Sidecar enables observability for all events within Grey Matter Fabric. This tutorial will guide you through a few easy steps to add an audit trail for your Fabric service.

Prerequisites

To complete this tutorial, you’ll need an understanding of, and local access to the following environments and tools:

• Unix/Linux setup

• Microservices and mesh architecture

• Grey Matter Sidecar - v0.7.2 +

• Grey Matter xDS

• Docker (https://docs.docker.com/install/) - v17.03 and newer

• Docker Compose (https://docs.docker.com/compose/install/)

• Kafka - v2.12-0 - 10.2.1

Step 1: Add Kafka to the Sidecar

Since the Sidecar will emit events into Kafka to be collected as the user wants, you will need to set up Kafka in Fabric. To emit a full GEM payload into Kafka, add the following environment variables to the hello-service-proxy section of the docker-compose.yml file.

  - EMIT_EVENTS=true
      - EMIT_FULL_RESPONSE=true
      - USE_KAFKA=true
      - ENFORCE_AUDIT=true
      - KAFKA_TOPIC="hello-service-tests"
      - KAFKA_ENABLED=true
      - OBS_ENFORCE=true
      - OBS_ENABLED=true
      - OBS_FULL_RESPONSE=true
      - KAFKA_ZK_DISCOVER=true
      - INHEADERS_ENABLED=true

Once you have made these changes, proceed to step 2.

Step 2: Add Kafka to Fabric

You'll need to add Kafka to Fabric so Kafka can start tracking audits and push them to Fabric. To add Kafka to Fabric, add the following code to the docker-compose-yml file:

kafka:
    hostname: kafka
    image: wurstmeister/kafka:0.10.2.1
    networks:
      - mesh
    environment:
      - KAFKA_HEAP_OPTS="-Xmx1G -Xms500M"
      - KAFKA_ADVERTISED_HOST=kafka
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zk
      - KAFKA_CREATE_TOPICS=hello-service-tests
    ports:
      - "22181:2181"
      - "29092:9092"
      - "9092:9092"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zk

Once you have added Kafka to Fabric, proceed to Step 3.

Step 3: Test Audit Event Results

Now verify that the audit trails work. Anytime someone visits a route that goes to the hello-service, the hello-service-proxy will emit an audit event in Kafka that says that something happened.

The event looks something like this:

{
    "action": "GET",
    "eventChain": [
        "9308cf66-8218-11e9-a159-0242ac1c0005"
    ],
    "eventId": "9308cf66-8218-11e9-a159-0242ac1c0005",
    "eventType": "",
    "originatorToken": null,
    "payload": {
        "isSuccessful": true,
        "request": {
            "endpoint": "/services/hello-service/0.1/",
            "headers": {
                ":authority": "localhost:8080",
                ":method": "GET",
                ":path": "/services/hello-service/0.1/",
                "accept": "*/*",
                "user-agent": "curl/7.54.0",
                "x-envoy-internal": "true",
                "x-forwarded-for": "172.28.0.1",
                "x-forwarded-proto": "https",
                "x-request-id": "d9e69795-3fc8-41c3-a0f2-7775822340c5"
            }
        },
        "response": {
            "body": "Hello World!",
            "code": 200,
            "headers": {
                ":status": "200",
                "content-length": "12",
                "content-type": "text/html; charset=utf-8",
                "date": "Wed, 29 May 2019 13:49:26 GMT",
                "server": "envoy",
                "x-envoy-upstream-service-time": "7"
            }
        }
    },
    "schemaVersion": "1.0",
    "systemIp": "172.28.0.5",
    "timestamp": 1559137766,
    "xForwardedForIp": "172.28.0.1"
}

View Observables in Kafka

To view exactly what is put into Kafka, enter the following command into the Kafka CLI located here: https://kafka.apache.org/quickstart#quickstart_consume

kafka-console-consumer --bootstrap-server localhost:9092 --topic hello-service-tests --from-beginning

Sample Output from Kafka

The output should look like this:

{
   "eventId":"a83bd73a-afc2-11e9-bf98-0242ac130006",
   "eventChain":[
      "a83bd73a-afc2-11e9-bf98-0242ac130006"
   ],
   "schemaVersion":"1.0",
   "originatorToken":[
      "CN=localuser,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
      "",
      "CN=localuser,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US"
   ],
   "eventType":"",
   "timestamp":1564158619,
   "xForwardedForIp":"172.19.0.1,172.19.0.1,172.19.0.6",
   "systemIp":"172.19.0.6",
   "action":"GET",
   "payload":{
      "isSuccessful":true,
      "request":{
         "endpoint":"/",
         "headers":{
            ":authority":"localhost:8080",
            ":method":"GET",
            ":path":"/",
            "accept":"*/*",
            "content-length":"0",
            "external_sys_dn":"",
            "ssl_client_s_dn":"CN=localuser,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
            "user-agent":"curl/7.54.0",
            "user_dn":"CN=localuser,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
            "x-envoy-external-address":"172.19.0.6",
            "x-envoy-original-path":"/services/hello-service/0.1/",
            "x-forwarded-for":"172.19.0.1,172.19.0.1,172.19.0.6",
            "x-forwarded-proto":"https",
            "x-real-ip":"172.19.0.1",
            "x-request-id":"9bbada13-3916-43c2-a59d-f1076a373a19"
         }
      },
      "response":{
         "code":200,
         "headers":{
            ":status":"200",
            "content-length":"12",
            "content-type":"text/html; charset=utf-8",
            "date":"Fri, 26 Jul 2019 16:30:19 GMT",
            "server":"Werkzeug/0.15.5 Python/3.6.6",
            "x-envoy-upstream-service-time":"6"
         },
         "body":"Hello World!"
      }
   }
}

What's Next?

Have your audits configured? Take the next step and learn how to visualize your audit data.

Visualize Audits

Questions?

Need help configuring audits?

Create an account at Grey Matter Support to reach our team.