The Grey Matter Metrics Filter sets up a local metrics server to gather and report real-time statistics for the sidecar, microservice, and host system.
Gathered Metrics
Total Stats
metrics version
total requests
total HTTP
total HTTPs
total RPC
total RPC/TLS
total requests
total 200
total 2xx
latency (avg)
latency (count)
latency max
latency min
latency sum
latency p50
latency p90
latency p95
latency p99
latency p9990
latency p9999
number of errors
incoming throughput
outgoing throughput
Route Stats
For each route that is addressed, the following stats will be computed and reported.
total requests
total 200
total 2xx
latency (avg)
latency (count)
latency max
latency min
latency sum
latency p50
latency p90
latency p95
latency p99
latency p9990
latency p9999
number of errors
incoming throughput
outgoing throughput
Host Stats
num goroutines
start time
CPU percent used
CPU cores on system
os
os Architecture
memory available
memory used
memory used %
process memory used
Prometheus
AWS CloudWatch
The metrics filter can also push the compiled statistics directly to AWS Cloudwatch. This allows the Grey Matter Proxy metrics to be directly used to trigger things like AutoScale actions or just for tighter monitoring directly in AWS.
Filter Configuration Option
Name
Type
Default
Description
metrics_port
Integer
8081
Port the metrics server listens on
metrics_host
String
0.0.0.0
Host the metrics server listens on
metrics_dashboard_uri_path
String
/metrics
The HTTP path to query JSON metrics data
metrics_prometheus_uri_path
String
/prometheus
The HTTP path to be scraped by Prometheus
prometheus_system_metrics_interval_seconds
Integer
15
metrics_ring_buffer_size
Integer
4096
Size of the cache of active metrics data
metrics_key_function
String
""
Function to provide internal rollup of URL paths when reporting metrics
metrics_key_depth
String
"1"
Truncate URLs to the first path section
use_metrics_tls
Boolean
false
If true, metrics server
uses TLS
server_ca_cert_path
String
SSL Trust file to use when serving metrics over TLS
server_cert_path
String
SSL Certificate to use when serving metrics over TLS
server_key_path
String
SSL Private Key file to use when serving metrics over TLS
Optionally, this filter can serve the computed statistics in a form suitable for scraping by . The prometheus endpoint will be hosted at {METRICS_HOST}:{METRICS_PORT}{METRICS_PROMETHEUS_URI_PATH}, which can then be scraped directly through the supported Prometheus mechanisms.
