Audits

Running the Grey Matter sidecar alongside a service allows for the auditing of request traffic and security information.

Audits

Key Definition

Audits are a security-relevant event within Grey Matter. An audit event, or simply event, can be either or both of the following:

• Change to the security state of the system

• Attempted or actual violation of the system access control or accountability security policies

An audit event report includes the following information:

• Name of the event

• Success or failure of the event

• Additional event-specific information that is related to security auditing

1. The Grey Matter Sidecar emits audit data to a Kafka topic for easy observability.

2. If Fabric is set up with an Edge, it pulls audit data from the PKI certificate, the IP address of the originating request, etc.

3. This audit data--also called events or observables--allows for detailed event auditing of ingress and egress traffic, and process resource use.

Grey Matter Configuration

Auditing is configured on a sidecar via the Grey Matter observables filter. The information can be emitted to an Apache Kafka cluster and visualized in a Kibana dashboard or can simply be emitted to stdout in the sidecar logs.

For a step by step to configure and visualize auditing in Grey Matter, see the Audits and Observables guides.

For more info on visualizing audits, see the Audit Proxy Observable Consumer section

What about Splunk?

While the Grey Matter Sidecar does not support direct emission of events into Splunk, you can create or modify a consumer to provide that capability. Learn more.

Indexing

Grey Matter does not index audit events directly into Elasticsearch. Instead, Grey Matter contains a Kafka consumer that reads Kafka observables. This consumer transforms and indexes them to use with Elasticsearch.

Use Kibana to Visualize Observables

Kibana is an open source Elasticsearch plugin that takes observables from Grey Matter and visualizes them in a graphical dashboard.

Kibana simplifies the creation of visualizations to explore, search, view, and interact with audit data stored in Elasticsearch indices. Kibana helps you analyze and visualize individual events and trends such as:

• Total requests

• Number of requests by individual users

• Geographic locations of requests made in Fabric

• What individual users are doing

• Timing of user requests

• What user are looking at

• user DNs (Authenticated user names)

• Geographic location of IP addresses

• Requests per hour by user

• Response codes

• Paths

• Service vs. userDN

• Services

• Response bodies

• User agents

Observable Information

The list information in a Grey Matter observables can be found here.

The following observable information was captured from a user accessing an event through a Sidecar operating within Grey Matter Fabric:

{
 "_index": "audit",
 "_type": "_doc",
 "_id": "FvUJ2GsBQetsYfWuW1Ab",
 "_score": 1,
 "_source": {
   "eventId": "00f4b3e4-a279-11e9-b433-0a580a82025d",
   "eventChain": [
     "00f4b3e4-a279-11e9-b433-0a580a82025d"
   ],
   "schemaVersion": "1.0",
   "originatorToken": [
     "cn=minos.kepheus, dc=hellas, dc=com",
     "CN=*.greymatter.svc.cluster.local,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
     "CN=*.greymatter.svc.cluster.local,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US"
   ],
   "eventType": "fibonacci",
   "timestamp": 1562697620,
   "xForwardedForIp": "15.188.27.135,10.129.2.140",
   "systemIp": "10.130.2.93",
   "action": "GET",
   "payload": {
     "isSuccessful": true,
     "request": {
       "endpoint": "/fibonacci/18",
       "headers": {
         ":authority": "demo-oauth.production.deciphernow.com",
         ":method": "GET",
         ":path": "/fibonacci/18",
         "accept-encoding": "gzip",
         "content-length": "0",
         "cookie": "OauthExpires=1562757619; OauthSignature=0OgHLzHBxSUdNk557aKWeYW9jrg; OauthUserDN=cn%3Dminos.kepheus%2C+dc%3Dhellas%2C+dc%3Dcom",
         "external_sys_dn": "CN=*.greymatter.svc.cluster.local,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
         "forwarded": "for=15.188.27.135;host=demo-oauth.production.deciphernow.com;proto=https;proto-version=",
         "ssl_client_s_dn": "CN=*.greymatter.svc.cluster.local,OU=Engineering,O=Decipher Technology Studios,=Alexandria,=Virginia,C=US",
         "user-agent": "Go-http-client/1.1",
         "user_dn": "cn=minos.kepheus, dc=hellas, dc=com",
         "x-envoy-original-path": "/services/fibonacci/1.0.0/fibonacci/18",
         "x-forwarded-for": "15.188.27.135,10.129.2.140",
         "x-forwarded-host": "demo-oauth.production.deciphernow.com",
         "x-forwarded-port": "443",
         "x-forwarded-proto": "https",
         "x-real-ip": "10.129.2.140",
         "x-request-id": "234aff6f-e376-41d5-89b8-1aa6dd0bbf4f"
       }
     },
     "response": {
       "code": 200,
       "headers": {
         ":status": "200",
         "content-length": "5",
         "content-type": "text/plain; charset=utf-8",
         "date": "Tue, 09 Jul 2019 18:40:20 GMT",
         "x-envoy-upstream-service-time": "0"
       },
       "body": "2584\n"
     }
   },
   "event_mapping": {
     "type": "EventAccess",
     "action": "ACCESS"
   },
   "time_audited": "20190709T184020.249380",
   "geo_ip": {
     "accuracy_radius": 1000,
     "latitude": 48.8607,
     "longitude": 2.3281,
     "time_zone": "Europe/Paris"
   },
   "location": {
     "lat": 48.8607,
     "lon": 2.3281
   }
 },
 "fields": {
   "payload.response.headers.date": [
     "2019-07-09T18:40:20.000Z"
   ],
   "time_audited": [
     "2019-07-09T18:40:20.249Z"
   ]
 }
}

Audit Proxy Observable Consumer (APOC)

The Audit Proxy Observable Consumer (APOC) reads from Kafka and submits the information to Elasticsearch asynchronously for maximum throughput. When deploying with the ELK stack, Logstash operates as the APOC.

APOC's goals include:

• Listening to given Kafka topic(s)

• Taking every event that is emitted into Kafka

• Transforming those events

• Submitting them to a given Elasticsearch index

Message Transformation

When a message is transformed by the APOC, event mappings are also added. The message will default to the following HTTP types found below:

{
  "GET": "ACCESS",
  "POST": "CREATE",
  "DELETE": "REMOVE",
  "PUT": "MODIFY"
}

The message can also be changed for individual routes by adding the following settings:

EVENT_TYPE_MAPPINGS:
 GET:
   - uri: "/activities"
     eventType: "EventSearchQry"
 POST:
   - uri: "/analyses/pos"
     eventType: "EventAccess"

During the transformation action_locations are added to the audit event. These consist of an IP address identifier based on the x-forwarded-for in the header of the request.

action_targets is added consisting of {x=forwarded-proto}://{:authority}{x-envoy-original-path}.

Determine Value of Creator

A creator is then added to the auditing payload, and follows these steps to determine its value.

• Allow for configuring an override default to that in the environment variable or settings.yaml or JSON.

• If there is a request header named application , use it.

• See if there is a request header named service , use it.

• Extrapolate the name if the request url contains /apps/{value}/ or /services/{value}/ x-envoy-original-path.

• Hard code a default. For instance, you could use service if none of the previous parameters are set.

In this configuration, the transformation will try to find user information in the request header and set action_initiator to be the userdn from the request. time_audited is also added to the audit information. This is the system time at which the audit transformation is completed by the APOC. Finally, if a GEO_DATABASE is supplied, the transformation will try to find the geographical location of the request based on the last IP address in the x_forwarded_for header field. The full information is added to the audit information as geo_ip, but basic location (lat and long) is added as location.

Questions

Create an account at Grey Matter Support to reach our team.