Access Log Service Stats

gm-control can be configured to parse Envoy access logs and send them to a number of stats backends, including:

• statsd

• dogstatsd

• prometheus

• wavefront

For every request, both upstream and downstream, gm-control will send to each configured stats backend:

• the upstream request and upstream response, including:

  • timestamp

  • zone

  • proxy name (of the proxy the request was sent to)

  • proxy version

  • node name

  • HTTP method

  • domain

  • route key

  • rule

  • shared rule

  • header constraint

  • upstream cluster (that the request will be sent to)

  • instance of gm-control

  • response code (for responses)

• the upstream latency, including all previously listed information

• the request event, if the stats backend supports it

For example, below is a screenshot of all upstream requests in the prometheus dashboard:

prom_dashboard

Example Configuration

Setting:

  - GM_CONTROL_STATS_BACKENDS=prometheus
  - GM_CONTROL_STATS_PROMETHETHEUS_ADDR=localhost:9102

will set up a Prometheus client on localhost:9102 that Prometheus can be configured to scrape from.

Log Access Logs

Additionally, gm-control can be configured to log access logs to stdout before they are processed by the ALS. To enable, set

  - GM_CONTROL_XDS_LOG_ACCESS_LOGS=true

The resulting logs (in gm-control) will look like this:

{
  "level": "info",
  "service": "gm-control",
  "access_log": {
    "LogEntries": {
      "HttpLogs": {
        "log_entry": [
          {
            "common_properties": {
              "downstream_remote_address": {
                "Address": {
                  "SocketAddress": {
                    "address": "127.0.0.1",
                    "PortSpecifier": {
                      "PortValue": 33438
                    }
                  }
                }
              },
              "downstream_local_address": {
                "Address": {
                  "SocketAddress": {
                    "address": "127.0.0.1",
                    "PortSpecifier": {
                      "PortValue": 10909
                    }
                  }
                }
              },
              "start_time": {
                "seconds": 1594050210,
                "nanos": 459432400
              },
              "time_to_last_rx_byte": {
                "nanos": 84200
              },
              "time_to_first_upstream_tx_byte": {
                "nanos": 271100
              },
              "time_to_last_upstream_tx_byte": {
                "nanos": 528200
              },
              "time_to_first_upstream_rx_byte": {
                "nanos": 6089400
              },
              "time_to_last_upstream_rx_byte": {
                "nanos": 6575800
              },
              "time_to_first_downstream_tx_byte": {
                "nanos": 6268700
              },
              "time_to_last_downstream_tx_byte": {
                "nanos": 6729100
              },
              "upstream_remote_address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.42.1.38",
                    "PortSpecifier": {
                      "PortValue": 10808
                    }
                  }
                }
              },
              "upstream_local_address": {
                "Address": {
                  "SocketAddress": {
                    "address": "10.42.2.45",
                    "PortSpecifier": {
                      "PortValue": 51226
                    }
                  }
                }
              },
              "upstream_cluster": "internal-jwt-security",
              "downstream_direct_remote_address": {
                "Address": {
                  "SocketAddress": {
                    "address": "127.0.0.1",
                    "PortSpecifier": {
                      "PortValue": 33438
                    }
                  }
                }
              }
            },
            "protocol_version": 2,
            "request": {
              "request_method": 1,
              "scheme": "https",
              "authority": "localhost:10909",
              "path": "/policies",
              "user_agent": "Go-http-client/1.1",
              "request_id": "63c767ba-9d6f-4050-8ecc-63473e13364f",
              "original_path": "/jwt/policies",
              "request_headers_bytes": 510,
              "request_headers": {
                "x-gm-domain": "*:10909",
                "x-gm-route": "route-data-internal-internal-jwt-slash",
                "x-gm-rule": "DEFAULT",
                "x-gm-shared-rules": "data-internal"
              }
            },
            "response": {
              "response_code": {
                "value": 200
              },
              "response_headers_bytes": 232,
              "response_body_bytes": 502,
              "response_code_details": "via_upstream"
            }
          }
        ]
      }
    }
  },
  "time": "2020-07-06T15:43:30Z"
}

See the Envoy docs for more information on the log format.

Configuration Options

See the gm-control command options for full configuration options.