The Challenge

Microservice Monitoring Is Complex

A service mesh lacks monitoring and visibility. Without mesh operations visibility, SREs and DevOps engineers struggle to find root causes for problems that can happen anywhere: on the network, inside containers, or between services.

Audit Trail and Compliance Reporting

‌Decentralized systems already have reporting challenges that grow exponentially with new services, and cloud environments. To make matters worse, failure to comply with privacy protection policies could bring legal trouble, massive fines, and loss of accreditation.

Digital Twin Modeling

Digital twins are a cost-effective way to help decision-makers across a number of business use-cases, from technical operations to marketing and sales. They require systems capable of securely replicating, interacting with, and, as necessary, accurately modifying, massive historical data volumes at rapid speed. In addition, human user and customer data is subject to several PII policy compliance requirements that may require the anonymization or the outright removal of key user data immediately upon user request.

Security Attribution Replay (Chain of Evidence)

Service meshes, microservices, serverless, and containers are key elements of Mesh Application and Service Architecture (MASA) implementation. However, these capabilities present significant service-to-service communication, discovery mechanism, security layer, and audit observability challenges at scale. The volume of data generated by mesh operations coupled with the complexity of hybrid and multi-mesh operations can overwhelm traditional compliance tracking efforts.

The Solution

Instant Visibility and Discovery

Grey Matter's Intelligence 360 helps visualize service health, data-flows, and microservice interactions so you can solve performance issues fast. Intelligence 360 provides full-scope health observability and contextual awareness for the mesh.

The combination of Fabric orchestration and Intelligence 360 dynamic control lets your team check service health, making changes on the fly fixing an anomaly impacting service performance. Grey Matter also enables granular audit and policy compliance as well as rapid dependency identification, for real-time tracking of aggregate, route, and service level SLOs for Memory Utilization, CPU Utilization, Percentile Latencies, Error Rate, and Request Rate in an easy-to-understand manner.

Grey Matter’s omnidirectional mesh telemetry capture and analysis capabilities enable deep mesh audit and observability without the need for special instrumentation. Grey Matter employs a normalized and repeatable in-depth audit capture of every event on the mesh which can be used to establish a baseline for compliance reporting. Employing Kibana as a third-party dashboard, Grey Matter enables data lineage and provenance oversight via fully observable audit controls for rapid forensic detection and diagnosis of anomalies and intrusions.

Scale with Confidence

Grey Matter supports various containers, microservices, and serverless architectures so you can adopt cloud-native technologies without worry.

Isolate the Root Cause

Grey Matter records and displays every activity within the mesh for in-depth audit control, policy compliance reporting, rapid forensic detection and diagnosis of anomalies and intrusions. You can cut through the noise to find performance outliers throughout your mesh and pinpoint the root cause of your problem within seconds.

Distributed Tracing

Grey Matter offers distributed tracing to monitor service availability and performance. This data helps teams troubleshoot the mesh and improve mesh performance.

Omnidirectional Mesh Telemetry Data

Grey Matter offers in-depth observability and policy compliance management for your complex multi-environment networks. Grey Matter sidecars run alongside each microservice within the mesh, creating a controlled and trusted service edge fleet guided by dynamically established policy guidelines. These services are responsible for managing network requirements such as scaling, access control, policy compliance, audit, and service-to-service intercommunication.

Grey Matter’s omnidirectional mesh telemetry capture and analysis capabilities enable deep mesh audit and observability without the need for special instrumentation. Network micro-segmentation and policy enforcement ensure zero-trust managed security compliance.

Mesh telemetry data powers architecture-wide service level objective (SLO) policy compliance and fine-grained hybrid mesh operational control. With Grey Matter, users can dynamically set SLOs governing each policy action atop the mesh. If one approaches warning or violation, Grey Matter alerts via an intuitive single-pane-of-glass interface for rapid mitigation.

Zero-Trust Security

Grey Matter is designed to operate using a zero-trust threat model to ensure each service and transaction running within a Grey Matter enabled hybrid mesh is appropriately protected. This is supported by the normalized and repeatable in-depth audit capture of every event on the mesh which can be used to establish a baseline for compliance reporting. The concept of zero-trust is centered on a belief that enterprises do not automatically trust systems or services inside or outside its perimeters. In the case of Grey Matter, everything attempting to connect is verified before granting access. Every action is recorded and can be played back for forensic analysis.

Grey Matter records and displays the lineage and provenance of every activity conducted by every user on every object atop the mesh throughout its life-cycle. The platform makes this information fully observable and accessible for in-depth audit control, policy compliance reporting, and rapid forensic detection and diagnosis of anomalies and intrusions.

Grey Matter is purposefully designed to capture and analyze the massive volumes of telemetry, user, and operational data generated by mesh operations. With Grey Matter, every piece of network traffic, user activity, or policy-driven system action can be brought together to create digital twins of systems, networks, and even individual users or group for analysis and testing. Because Grey Matter is built with zero-trust control and audit tracing in mind, PII information can be anonymized or selectively removed in keeping with legislative requirements such as the EU General Data Protection Regulation (GDPR).

Questions?

Artifacts

Grey Matter v1.2 artifacts are now available. Artifacts can be found in the staging repositories:

• Images

1.2.2

Server versions (if changed from 1.2.1):

Fabric

• gm-proxy:1.4.5

Fixed

• Fixed intermittent segfault when using websockets

1.2.1

Server versions (if changed from 1.2.0):

Fabric

• gm-proxy:1.4.4

• gm-jwt-security-gov:1.1.2

• gm-cli:1.4.2

Sense

• gm-dashboard:3.4.2

• gm-slo:1.1.5

Changed

• JWT-Security filter cache now respects token expiration and internally limits the max cache size

• Sidecar now supports FIPS-compliant builds

• Sidecar base build updated to Envoy 1.13.3

1.2

Server versions:

Fabric

• gm-proxy:1.4.2

• gm-control:1.4.2

• gm-control-api:1.4.4

Sense

• gm-dashboard:3.4.1

• gm-slo:1.1.4

• gm-catalog:1.0.7

Platform Services

• gm-data:1.1.1

Release Notes

Added

• Allow setting Envoy HTTP filters

• Allow setting Envoy Network filters

• Allow setting Network and HTTP filters on Grey Matter Listener object

Changed

• Sidecar base Envoy build updated to 1.13.1

• gm.inheaders filter will now return 403 if certificates are not present

• Update trace defaults to v2 APIs

Removed

• None

Fixed

• Control EDS resolution of instances now properly causes an update to be sent to the Sidecar

• Change internal protocol selection to USE_DOWNSTREM_PROTOCOL to fully support HTTP2

• Control server no longer overwrites defined static resources for the data plane

Known Issues

• The Catalog API requires that cluster name be unique, if you have two services with the same name and version values. Failure to do so will lead to a mismatch in the Sense Dashboard and you will not see one of the services. If the versions are unique then you can use the same name value.

• The proxy requires a route to be configured on the domain/listener in order for observables to be enabled.

Common Questions

If you're familiar with microservices, you've probably faced the following questions.

• How do I know what services are running in my infrastructure?

• How do I enable secure hybrid methodologies across multiple cloud and platform-as-a-service (PaaS) environments while leveraging existing on-prem investments?

• How do I deploy and enforce common security models?

• How do I avoid core function duplication (i.e. discoverability, scalability, observability, security, service level management, multinetwork fabric and data protection policies)?

• How do I understand how my new and existing IT investments are being used?

Explore Grey Matter's solutions to these common use cases below.

Microservice instances have dynamically assigned network locations, and these locations change due to autoscaling, failures, and upgrades.

Grey Matter's manage complex network traffic and data. By removing network communication complexities and infrastructure concerns, your team can focus on writing next-gen services in a polyglot, cloud-agnostic environment.

A is an infrastructure layer built into your application. It controls how parts of your app share data, and tracks these interactions so you can optimize communication and avoid downtime as your app grows. Unique service-level insight and control let you closely measure and manage business objectives.

Use a service mesh like Grey Matter to maximize network efficiency, free developers to innovate, and harness data to enable AIOps.

Modernize your applications and .

Grey Matter supports the scaling of containers, microservices, and serverless architectures, and the dashboard will help you optimize infrastructure, application performance and business outcomes. Grey Matter takes advantage of your network's data through the innovative application of neural net AI designed to enhance traditional network and application performance. Enhance your diagnostic capabilities, and enable predictive network operational monitoring and automated response.

While traditional security models rely on location-based trust, provides trust for all access requests regardless of location. It enforces adaptive controls and verifies trust on an ongoing basis.

Trust levels adapt to your evolving business. Our zero-trust approach helps prevent unauthorized access, contain breaches, and helps reduce the risk of an attacker's lateral movement.

Questions?

The Challenge

It's hard to improve performance and operations in a dynamic microservice environment.

The Solution

Dynamic Routing and Security

Grey Matter is a platform agnostic service mesh that simplifies network management. The mesh is made of --components that work in unison to optimize decentralized microservice performance on the network. Grey Matter's service policies and configurations enable dynamic routing and security based on service identity. These policies scale without IP-based rules or networking middleware.

Benefits of a Service Mesh

Grey Matter takes complexity from a single microservice and puts it into a "sidecar" proxy. This sidecar proxy works with its dedicated service to provide the following benefits:

• It gives its service behaviors the service needs to perform well in a microservice architecture, and

• It lets its dedicated service perform its business-specific tasks.

The following table summarizes the benefits a service mesh provides.

Features

Real-Time Performance Metrics

If you’re building microservices, you're anticipating the ability to scale, since a microservices architecture will look very different a year out. A new service introduces failure points, and microservices make it hard to find the root of failures without a mesh. A service mesh captures all communications as performance metrics. These metrics translate to more reliable service requests and a more secure way to scale.

Secure and Reliable Decision-Making

Grey Matter separates decision-making from data-gathering with its data and control planes to improve the performance of each of these important activities.

The data plane is a collection of sidecar proxies: one proxy for each service. The data plane manages traffic from one application to another and includes routing, forwarding, load balancing, even authentication and authorization.

The control plane connects data planes and serves as the policy and management layer of the service mesh. It collects telemetry data and makes decisions about configurations.

Hybrid Cloud, Traffic Management, and Observability

Grey Matter lets you set policies you can enforce across cloud instantiations. Its single abstraction layer hides details of the underlying cloud.

Service-to-service communication can be managed centrally, enabling advanced traffic management patterns such as service failover, path-based routing, and traffic shifting that can be applied across public and private clouds, platforms, and networks.

Centrally-managed service observability includes detailed metrics on all service-to-service communication such as connections, bytes transferred, retries, timeouts, open circuits, and request rates, response codes.

Secure Services Across Any Runtime Platform

Grey Matter offers secure communication between legacy and modern workloads. Sidecar proxies allow applications to be integrated without code changes and Layer 4 support provides nearly universal protocol compatibility.

Certificate-Based Service Identity and Encrypted Communications

Grey Matter uses TLS certificates to identify services and secure communications. Using TLS provides a strong guarantee of the identity of services communicating, and ensures all data in transit is encrypted. These certificates use the SPIFFE format for interoperability with other platforms. Grey Matter can be a certificate authority to simplify deployment, or integrate with external signing authorities like Vault. All traffic between services is encrypted and authenticated with mutual TLS.

Questions?

Grey Matter is a software platform that provides zero-trust policy compliance, operational insight, and infrastructure management. More concretely, it's a cloud-native infrastructure layer that improves microservice performance and simplifies service mesh operations.

Core Components

Grey Matter is composed of . Each simplifies the technical challenges associated with microservice management, such as service announcement, service discovery, instrumentation, logging, tracing, troubleshooting, encryption, and access control.

Technical Overviews on Grey Matter

Grey Matter's core features include:

• Zero-trust security

• Cloud, language, and vendor-agnostic implementations

• Automated service-level observability and management

You can use Grey Matter to improve service discovery, build or manage your service mesh, gain business insights, and add zero-trust security to your environment. You can also use Grey Matter to extend your unique use cases.

Read more about our use cases to learn what Grey Matter does.

Grey Matter is composed of Fabric, Data, and Sense. Learn how these components simplify microservice management in Architecture.

Why Grey Matter?

Grey Matter uses mesh-generated telemetry data to improve network insight and systems performance from local development, to production-scale container and hybrid cloud implementations.

Flexibility and Compatibility‌

We take our design cues from open architecture. Our implementations don't require predetermined downstream investments for existing infrastructure. Grey Matter is cloud, language, and platform agnostic.

Precision, Depth, and Security

Fabric, the platform’s core Envoy proxy-based mesh technology, generates and captures telemetry and audit data. This data powers dynamic enterprise policy compliance, and SLO creation and monitoring.

‌Grey Matter feeds each piece of telemetry and audit data generated throughout the lifetime of a service instance to Grey Matter’s Sense SLO monitoring overlay service. Because Grey Matter maintains the entire history of every event tied to the lifespan of every service on the mesh, the value of SLO monitoring for enterprise IT only grows over time.

Backed by the full lineage of service telemetry data, your team can:

• Establish historical patterns of use

• Refine SLOs to best reflect real-world performance

Focus on Zero-Trust

Grey Matter is designed with in mind. Our platform manages the activities, policies, security, auditing and data of every microservice in a multi-tenant mesh across any enterprise environment. These zero-trust enforcement features include:

• Enabling service-to-service mTLS connections

• Scheduled or on-demand key rotation

• Service cryptographic identifiers

Audit logs compliant with ICS 500-27 standards are automatically provided to your security team. Grey Matter maintains a historical pedigree of every action occurring on the system throughout its lifespan. This data can be provided to auditors and/or security analysts to assist with risk management, data loss, or breach impact mitigation.

Learn more about zero-trust , or learn how to set up zero-trust for your mesh.

Elegance and Accessibility‌‌

Intelligence 360 is Grey Matter’s single pane of glass for mesh service health monitoring, access, and control. Intelligence 360 enables service level objective (SLO) observability and dynamic control for every service within Grey Matter Fabric. Intelligence 360 visualizes aggregates, routes, and service-level SLOs for the following infrastructure data:

• Memory Utilization

• CPU Utilization

• Percentile Latencies

Enterprise Business Value

Grey Matter's telemetry data gives you insight into microservice performance and control over your system's resource use. It offers the following key telemetry backed value-add benefits.

Improve Data and Communications Management

is the control and data plane responsible for managing the mesh environment. Envoy-proxy backed communications and traffic management, policy enforcement, audit, and data governance take place atop Fabric.

Grey Matter Data‌

is a data store agnostic API designed to manage access control and zero-trust data security policy compliance. Data enables secure global data sharing backed by complex content tagging, object tracking, and security policy. Data is store agnostic, and capable of supporting multiple storage backends to include Amazon S3, CEPH, Gluster, local disk, and more.

​Improve Operations and Network Management

Grey Matter includes several open-source tools that improve quality and security, and offer low-cost and flexible ways to meet goals. With Grey Matter, network orchestration and policy management do not require expensive third-party tools and licensing fees.

Questions?

Welcome to Grey Matter documentation. This documentation covers all available features and options for Grey Matter, the intelligent hybrid mesh platform.

The Challenge

All Network Traffic Is Untrusted

Grey Matter Fabric

Fabric is the master control plane for Grey Matter, and helps microservices focus on specific tasks.

Explore how Fabric works within Grey Matter's architecture, or configure Fabric now.

Oversight and Visibility

Fabric's user-operated dashboard helps visualize and manage the service mesh. The dashboard provides the following features:

• Status, owner, capability, and business value sortability

• Fine-grained search targeting every microservice on the mesh

• Aggregate service reporting for all service instances spawned atop the mesh

Learn more about Grey Matter's service discovery features.

Network Operations Management and Business Intelligence

Grey Matter offers the following network operations and business intelligence activities:

• User-defined service and route level management and objective capture for:

  • Aggregated Service Level Objectives (SLOs)

    • memory, CPU, latency, error and request rates

Read more about Grey Matter's business insight capabilities here.

Security and Access Control

Each microservice requires its own data access point and generates its own data, creating a lot of complexity in the network. Grey Matter governs data access management with fine-grained access policy creation and management.

Fabric's sidecar proxies run alongside each microservice to manage network requirements such as scaling, access control, and intercommunication. Its security and access control features include:

• Service-to-service and end-user authentication

  • Mutual TLS and Fine-Grained Access Policies

  • FIPS 140-2 (BoringSSL can be built in a )

Learn more about Grey Matter's security policies here.

Compatibility and Interoperability

Grey Matter is compatible with the following technologies:

• Istio Compatible Sidecar

  • Beta features (Istio Control Plane, Citadel, Pilot, Mixer, etc.)

• Consul Compatible Sidecar

Grey Matter Data

Data and Content Transfer and Distribution

Grey Matter Data is a Data Distribution Network (DDN) that provides data and content capture, store, sync, cache, move and share of any kind, to and from consumers and services anywhere.

Data's Features

Data's features include:

• Immutable, timestamped fact database

• JWT-based authentication and authorization

• High-performance, user-defined security policy

Grey Matter Sense

Finally, the direct and atmospheric telemetry generated by this constant flow of data is fed to Sense, our experimental machine learning-enabled AI layer. We designed Sense to automate and optimize enterprise network operations, cutting resource expenditures at machine speed.

Sense's Features

Sense's features include:

• Catalog

• Service Level Objectives

• Intelligence 360

All Together

Check out our architecture section for visual representations of Grey Matter's features.

Questions?

Simple Deployment Model

In the simple deployment model, there is a single point of ingress between the Client and the Grey Matter Edge. The Edge routes traffic to appropriate Grey Matter Sidecars.

Multi-Mesh Deployment Model

The multi-mesh deployment model extends the simple deployment model by allowing each mesh egress to communicate with another mesh ingress using mTLS, as shown below.

Questions?

Grey Matter is a zero-trust hybrid mesh platform built using open architecture and mesh app and service architecture (MASA) principles.

Each microservice within Grey Matter runs and scales independently to improve secure interoperations, resiliency, continuity of operations, and insight for your business. Our omnichannel support provides rich, fluid, and dynamic connections between people, content, devices, processes, and services.

Core Architectural Principles

Our core architecture principles support the following business needs.

Security

Designed to operate using a zero-trust threat model to ensure each service running within a Grey Matter enabled hybrid mesh is appropriately secured, observed, and managed.

Portability

Enable on-premise, multi-cloud, and multi-platform as a service (PaaS) runtime environments.

Cloud-native

Built with elasticity, high availability, and cloud computing models in mind - provides a unified mesh platform to build applications as microservices, utilize container management solutions, and dynamically orchestrate workloads across hybrid enterprise.

Web-scale

Provides a solid foundation to scale with the growth of your business. Enabling modern architectural patterns supporting rapid increase or decrease in traffic volume, maintaining business insight for effectiveness and efficiency, and aiding in the reduction of bottlenecks when the time matters.

Modular

Modular service delivery - enabling loosely coupled systems and services developed independent of each other, taking advantage of continuous delivery to achieve reliability and faster time to market.

Interoperability

Creates a secure unified zero-trust network fabric, allowing systems to interchangeably serve or receive services from other systems providing enterprises the ability to perform multi-environment segmentation and observe traffic flows between environments. Managed through a runtime environment agnostic Grey Matter Control API.

Adaptive

Able to react to digital business changes, providing a pathway enabling business insight, security, and connectivity across multiple environments reducing complexities while increasing and facilitating a business's digital transformation journey.

Intelligent

Use of artificial intelligence (AI) techniques simplifying and assisting a user experience while providing business insight and fleet wide management across Grey Matter connected resources.

Automation

Integration into any ecosystems and end-to-end automation through the lifecycle of the mesh app and service architecture.

Questions?

Grey Matter Control performs service discovery and distribution of configuration in the Grey Matter service mesh. Control provides policy and configuration for all running sidecars within the Grey Matter platform.

The Control server works in conjunction with the Grey Matter Control API to manage, maintain, operate, and govern the Grey Matter hybrid mesh platform.

xDS

The Control server performs service discovery in the mesh and acts as an xDS server to which all proxies connect.

xDS is the generic name for the following:

• Endpoints (EDS)

• Clusters (CDS)

• Routes (RDS)

Service Discovery

Service Discovery is the way Grey Matter dynamically adds and removes instances of each microservice. Discovery adds the initial instances that come online, and modifies the mesh to react to any scaling actions that happen. To keep flexibility in the Grey Matter platform, the Control server supports a number of different service discovery options and platforms.

The ability to automatically populate instances of a particular microservice comes from the clusterobject. In particular, the name field in the cluster object determines which nodes will be pulled out of the mesh and populated in the instances array. In the example below, the name is catalog. This means that all services that announce as catalog in service discovery, will be found and populated into the instances array after creation.‌

Create the following object:

‌

Will be populated in the mesh as:

‌

Even though the object was created with no instances, they were discovered from the mesh and populated. Now any service that needs to talk to catalog, can link to this cluster and address all live instance.‌

Configuration Discovery

Each proxy in the mesh is connected to the control plane through a gRPC stream to the Grey Matter Control server. Though gm-control-api houses all the configuration for the mesh, it's ultimately gm-control that turns these configs into full Envoy configuration objects and sends them to the proxies.‌

The configuration in the Control API is mapped to physical proxies by the name field in the proxy API object. It's very important that this field exactly match the service-cluster identifier that the intended target proxy used when registering with gm-control.

In the example below, the proxy object, and all other objects linked by their appropriate keys, will be turned into a full Envoy configuration and sent to any proxies that announce as a cluster catalog.‌

Services that announce as:

‌Will receive the config form the object below, because XDS_CLUSTER==name, and they're both in the same zone.

Questions?

Grey Matter consists of a control plane, data plane, mesh, telemetry-powered business intelligence, and AI. It can be deployed on multiple cloud-native or legacy infrastructures without placing predetermined downstream requirements on existing investments.

Learn about the inner workings of Grey Matter's three core components: Fabric, Data, and Sense.

Grey Matter enables unified hybrid microservice deployments and hybrid/multi-cloud operations without special requirements for underlying infrastructure like containers or container platforms regardless of cloud vendor, PaaS or infrastructure.

Questions?

The Challenge

Service discovery is important but hard. Microservices constantly communicate with instances of other microservices, intensifying the volume and frequency of communications.

The Solution

Grey Matter's service-level visibility and telemetry improves service inventory and dependency analysis.

How It Works

The server supports several service discovery options and platforms.

• Control performs service discovery in the mesh and acts as the Envoy xDS server to which all proxies connect.

• Service discovery lets services communicate transparently with a member of a cluster without knowing its identity, or even how many members there are.

• When a service joins or leaves the cluster, it joins or leaves the load-balancing list.

Implementations

Grey Matter supports *several service discovery implementations, with more to come. These include a central server or servers that maintain a global view of addresses (Kubernetes keeps this central server), and clients that connect to the central server to update and retrieve addresses.

*Except for flat-file cases; we farm out that functionality to some other service discovery system.

Here is a short explanation of the workings of each of our service discovery implementations.

Kubernetes

Grey Matter's Kubernetes service discovery scheme keeps track of labels placed on pods when they're deployed. Our software uses the Kubernetes API to ask "what pods are in this namespace?" Then we use the labels to match those pods to "clusters", so services can talk to other services just by saying "send this request to some instance of this cluster". When pods die, they get removed from Kubernetes' service discovery list, and therefore from our software's list of running instances for a cluster.

Consul

For existing infrastructures already using Consul, you can easily configure Grey Matter to discover services from the existing catalog without migration to Kubernetes or any other platform. Infrastructures with services running in heterogeneous environments, or looking to leverage features such as health checking, load-balancing, or distributed service configuration management can use Consul as a central service registry from which Grey Matter will discover.

Envoy

Many cloud infrastructure use Envoy's or to keep track of instances and upstream clusters. Since Grey Matter Control uses xDS as the configuration discovery mechanism, Grey Matter inherits all Envoy configs. Moreover, any Grey Matter proxy can act as a vanilla Envoy proxy instance, and any Envoy proxy can integrate with Grey Matter Control.

File

We also support service discovery for flat-file cases.

Features

Grey Matter monitors cloud-native environments in real-time. Grey Matter can detect and address meaningful conditions in seconds.

Prerequisites

1. git installed

The Grey Matter Sidecar is a L7 reverse proxy based off of the popular Open-Source . Grey Matter's proxy enhances the base capabilities with custom filters, logic, and the ability for developers to write full-featured Envoy filters in Go.

Fabric Mesh

The primary use of the Grey Matter Sidecar is to act as the distributed network of proxies in the Grey Matter Fabric service mesh. In this use-case; each proxy starts out with very simple configuration, which is then modified by the control plane to suit the changing needs of the network. The documentation here is focused on the individual proxy itself; low-level configuration, filter specifications, etc.

Service meshes, microservices, server-less, and containers are key elements of Mesh application and service architecture (MASA) implementations. MASA, APIs, and internal traffic patterns represent one of the most effective pathways to enterprise modernization, but this doesn’t come without challenges.

Industry has signaled increased interest in zero-trust infrastructure for service-to-service mTLS connections, scheduled or on-demand key rotations, service cryptographic identifiers, observability (continuous monitoring, granular audit compliance, etc.), service-level management, and policy management throughout the enterprise service fleet.

Security Models

Data

Grey Matter Data is a microservice for the versioned and encrypted storage of media blobs and assets. It is a high-performance time-series object storage. It contains an immutable sequence of events which collectively describes a file system at a given moment in time. If the first event is a creation of an object and the second is its deletion, then the object will not appear in listings as of the current time, but it will appear as of any time after the creation and before the deletion.

Get Started with Grey Matter

Prerequisites

Because the Sidecar admin interface can contain sensitive information and allows users to manipulate the sidecar, access is not routed through the mesh. In a Grey Matter deployment, access is locked down and only accessible by admin users who have direct access to the pod.

This guide is a step by step walkthrough on how to access a Sidecar's admin interface. This interface is used to both inspect and operate on the running server; doing tasks like inspecting stats or performing a hard shut-down. In this walkthrough, we'll perform the following common tasks:

• Examine the live stats

Prerequisites

1. An existing Grey Matter deployment running on Kubernetes ()

2. kubectl or oc setup with access to the cluster

NOTE, the operations on the admin server do not depend on the platform in use. Since this guide is focused on Kubernetes, we'll use kubectl for access to the server. On other platforms, tools like ssh sessions can be used instead.

Steps

For all examples shown here, the Sidecar's admin server has been started on port 8001. This is the default for Grey Matter deployments.

1. Establish a Session

The first step is to establish a session inside the pod we need to examine. For this walkthrough, lets look at the edge service. To do this, we'll need the POD ID for any edge nodes in the deployment:

We can see we have one edge pod with an ID of edge-7d7bf848b9-xjs5l. We'll now use the kubectl tool to start a shell inside that pod. After running the command below, you'll be left off with a shell inside the container.

Now that we're in the pod, we can run commands against the admin interface. Here we'll use the curl to send commands since it's widely available and installed by default on Grey Matter docker images.

2. General Admin Commands

To see the full list of endpoints available, send a GET request to the /help endpoint. Full descriptions of each can be found in the .

In the next 3 sections, we'll show sample usage of the /stats and /config_dump endpoints.

2. Stats

The /stats endpoint is used to output all of the collected statistics. The first ~30 lines are shown below, but the full output is much larger and is dependent on the exact configuration of the sidecar being examined. For each new filter, listener, or cluster that is configured, new stats will be output. The stats will also be different depending on each connection type, e.g. or .

To query just a subset of the stats, use the filter query parameter:

or

3. Configuration Dump

Next we'll dump out the entire configuration of the Sidecar. The output of this command is routinely many thousands of lines long, but is very useful to inspect the exact state of a sidecar at any given moment. This config is mostly useful for verifying the exact behavior of the sidecar against what was intended.

5. Exit

Type exit in the terminal to return to drop the connection to the pod and return to your local shell.

This guide is a step by step walkthrough on deploying a new service into an existing Grey Matter deployment. This guide is for a SPIFFE/SPIRE enabled deployment. For a walkthrough on deploying a service into a non-SPIFFE/SPIRE Grey Matter setup, follow the Quickstart Launch Service to Kubernetes for a non-SPIFFE/SPIRE Deployment.

If you are looking for a quick service deployment and/or configuration guide, check out the Grey Matter templates.

Prerequisites

1. An existing Grey Matter deployment running on Kubernetes ()

2. kubectl or oc setup with access to the cluster

3. greymatter cli with access to the deployment

Overview

1. Launch pod with the service and sidecar

2. Create Fabric configuration for the sidecar to talk to the service

3. Create Fabric configuration for the Edge to talk to the sidecar

Steps

1. Launch pod

The service we'll launch is a simple Fibonacci service. It has one route: /fibonacci that calculates the fibonacci sequence of any integer supplied.

Note the spire-specific configurations to the deployment - the volume and volume mount spire-socket and the environment variable SPIRE_PATH. These are the additions that will need to be made to any deployment for a service you wish to add to the mesh with SPIFFE/SPIRE.

Save the above deployment file as deployment.yaml and launch the pod:

2. Local Routing

The next steps are to create objects in the Fabric API. These objects will create all the configuration for the Sidecar to handle requests on behalf of the deployed service.

This step creates and configures the Grey Matter objects necessary to allow the sidecar container in the deployment to route to the fibonacci container (the service itself). We will refer to this as "local" routing. The will configure the Edge proxy to route to the fibonacci sidecar, thus fully wiring the new service into the mesh for routing.

NOTE: This guide goes over deploying a new service and configuring it for ingress routing. To configure a service for both ingress and egress routing within the mesh, see the .

For each Grey Matter object created, create the local file and send them to the API with: greymatter create <object> < <file_name.json>.

Domain

The first object to create is a Grey Matter Domain the ingress domain for the fibonacci sidecar. This object does virtual host identification, but for this service we'll accept any hosts ("name": "*") that come in on the port 10080 (the port with name proxy-or value of Grey Matter Control environment variable GM_CONTROL_KUBERNETES_PORT_NAME-in the sidecar container). Note that force_https is set to true, requiring any incoming connection be https.

See the for more information.

Save this file as domain.json and apply it with

Listener

The next object is the ingress listener. This is the physical binding of the Sidecar to a host interface and port and is linked in the field domain_keys to a specific domain. The listener and domain configurations determine where the sidecar should listen for incoming connections on and what kind of connections it should accept.

The listener object is also the place to configure . See the for more information.

Note the secret field. This field is required for service to service communication in a SPIFFE/SPIRE setup. This secret tells the sidecar to fetch its SVID (with ID spiffe://quickstart.greymatter.io/fibonacci) from Envoy and present it to incoming connections. It also will set a with match subject alternative names specifies to only accept incoming requests with SAN spiffe://quickstart.greymatter.io/edge. See the for specifics. The listener secret configuration will be important for the .

Save this file as listener.json and apply it with

Proxy

The proxy object links a sidecar deployment to it's Grey Matter objects. The name field must match the label on the deployment (in this case greymatter.io/control) that Grey Matter Control is looking for in it's environment variable GM_CONTROL_KUBERNETES_CLUSTER_LABEL. It takes a list of domain_keys and listener_keys to link to the deployment with cluster label matching name.

See the for more information.

Save this file as proxy.json and apply it with

Local Cluster

The next object to create is a local cluster. The cluster is in charge of the egress connection from a sidecar to whatever service is located at its configured instances, and can set things like circuit breakers, health checks, and load balancing policies.

This "local" cluster will tell the sidecar where to find the fibonacci container to send requests. From the deployment above, we configured the fibonacci container at port 8080. Since the sidecar and fibonacci containers are running in the same pod, they can communicate over localhost.

See the for more information.

Save this file as fibonacci-local-cluster.json and apply it with

Local Shared Rules

The shared rules object is used to match to . They can do some features of routes like setting retry_policies and appending response data, but they also can perform traffic splitting between clusters for operations like blue/green deployments.

This local shared rules object will be used to link the in the next step to the that we just created.

See the for more information.

Save this file as fibonacci-local-rules.json and apply it with

Local Route

match against requests by things like URI path, headers, cookies, or metadata and map to shared_rules. Since this service only needs to forward everything it receives to the local microservice, the setup is fairly simple.

This "local" route will link the (fibonacci-domain) to the (fibonacci-local-rules) that we just created. We know that the fibonacci-local-rules object is used to link routes to the fibonacci-cluster, thus with this route object applied, the fibonacci sidecar will be configured to accept requests and route to the fibonacci service.

See the for more information.

The path indicates that any request coming into the sidecar with path / should be routed to the fibonacci service. We will see in the next step when configuring that all requests from the Edge proxy to the fibonacci service will come in at this path.

Save this file as fibonacci-local-route.json and apply it with

The sidecar will now be configured to properly accept requests and route to the fibonacci service. The next step will configure the Edge proxy to route to the sidecar.

3. Edge Routing

Now that the Sidecar-to-Service routing has been configured, we will set up the Edge-to-Sidecar routing because we want this service to be available to external users.

The process will take similar steps to what was done before, but we only need to create a cluster, a shared_rules object pointing at that cluster, and two routes.

Edge to Fibonacci Cluster

This will handle traffic from the Edge to the Fibonacci Sidecar. The Edge has existing domain (with domain key edge), listener, and proxy much like the ones we just created for the fibonacci service. The first step to configure the Edge to fibonacci service is to create a cluster to tell it where to find the fibonacci sidecar.

NOTE that there are several differences between this cluster and the created above:

1. The instances field is left as an empty string, whereas the fibonacci-local-cluster instances were configured. This is because Grey Matter Control will discover the fibonacci deployment and the instances array will be automatically populated from this service discovery: the instances will go up and down whenever the service scales or changes. To do this, (in the same way as described in creating the object above) the name field must match the cluster label on the deployment.

2. This cluster has a secret set on it, and

Save this file as edge-to-fibonacci-cluster.json and apply it with

Edge to Fibonacci Shared Rules

The edge to fibonacci will be used to link the to the edge-to-fibonacci-cluster .

Save this file as edge-to-fibonacci-rules.json and apply it with

Edge to Fibonacci Routes

There are two that need to be created to send requests through the Edge proxy to the fibonacci service. In the same way that the was connected to the fibonacci-domain, these routes will be connected to the edge domain, and will configure how the edge sidecar sends routes.

They are nearly identical, but one provides for the case that the path on the request to send to the fibonacci service ends in a slash, and one for the case that it doesn't.

This route looks for the path on a request into the edge proxy /services/fibonacci with no trailing slash, replaces that path with /, and sends the request to the cluster (edge-to-fibonacci-cluster) being pointed to by the edge-to-fibonacci-rules.

This route looks for the path on a request into the edge proxy /services/fibonacci/ this time with a trailing slash, replaces that path with /, and sends the request to the same cluster.

Save these files as edge-to-fibonacci-route.json and edge-to-fibonacci-route-slash.json and apply them with

Once these routes are applied, the service is fully configured in the mesh! You should be able to access the service at https://{your-gm-ingress-url}:{your-gm-ingress-port}/services/fibonacci/ with response Alive. To send a request for a specific fibonacci number, https:///{your-gm-ingress-url}:{your-gm-ingress-port}/services/fibonacci/fibonacci/<number>.

If you don't know your gm ingress url and you followed the guide, run

and copy the EXTERNAL-IP and port (by default the port will be 10808).

4. Catalog

The last step in deploying a service is to add the expected service entry to the Grey Matter Catalog service. This will interface with the control plane, and provide information to the Intelligence 360 Application for display.

Save this file as fibonacci-catalog.json and using the quickstart certificates from the helm chart repository at ./certs, make the following post request to the catalog service:

You'll see the following response if the addition was successful.

And the service will display in the Intelligence 360 Application.

This is an example of deploying a new service into the mesh and configuring it for ingress and egress actions using SPIFFE/SPIRE identities.

Prerequisites

1. An existing Grey Matter deployment following the Grey Matter Quickstart Install with Helm/Kubernetes guide.

   1. This guide assumes you've followed the step to configure a load balancer for ingress access to the mesh. If so, you should have a URL to access the Grey Matter Intelligence 360 application (e.g. a2832d300724811eaac960a7ca83e992-749721369.us-east-1.elb.amazonaws.com:10808).

   2. Make note of your load balancer ingress URL and use it wherever this guide references {your-gm-ingress-url}.

2. setup with a running Fabric mesh.

Overview

1. Generate k8s deployment with the service and sidecar

2. Configure the sidecar for ingress actions

3. Configure the sidecar for egress actions

Steps

1. Create Deployment

The service we will be using for this walkthrough is a simple egress/ingress service with two endpoints. The /ingress endpoint will return Ingress request to simple-service successful!. The /egress endpoint takes an environment variable EGRESS_ROUTE, generates a request to that route, and returns its response.

There are a few things to note before generating the deployment file:

1. The value of the pod label that Grey Matter Control is discovering, environment variable GM_CONTROL_KUBERNETES_CLUSTER_LABEL

2. The value of the kubernetes port name that Grey Matter Control is discovering, environment variable GM_CONTROL_KUBERNETES_PORT_NAME

3. The trust_domain of the SPIRE server

Using the Grey Matter Helm Charts, the values of the first and second points are greymatter.io/control and proxy by default. The deployment will then need that pod label greymatter.io/control with a value identifying the service to Grey Matter Control, and its sidecar container will need a port labeled proxy.

The SPIRE trust domain, by default, is quickstart.greymatter.io. This will be important for . The SPIRE registrar service, by default, is configured to register entries with the SPIRE server for every pod that is created with the same label that Grey Matter Control is looking for, greymatter.io/control. It will generate SPIFFE identities for these pods in the form spiffe://<trust_domain>/<greymatter.io/control-value>. For example, for a pod in the default setup with label greymatter.io/control: control-api, the registrar will generate an entry with SPIFFE ID spiffe://quickstart.greymatter.io/control-api.

Note: If the registrar service in your setup is configured without the pod_label specification, it will generate entries with the SPIRE server in a different form, which will determine both the way that the mesh is configured and the deployment. The entries in this setup will be of the form spiffe://<trust_domain>/ns/<namespace>/sa/<service_account>, and thus the secrets in the mesh configurations will need to reflect this format instead.

Based on the defaults described, the deployment for the simple-service will look like the following:

Note: we set the environment for the service container EGRESS_ROUTE to https://localhost:10909/catalog/summary. This will be important when

Save the deployment as deployment.yaml and apply the deployment:

Run kubectl get pods -l=greymatter.io/control=simple-service and make sure that the containers are running 2/2 before moving on the the mesh configuration.

2. Configure the deployment for ingress actions

Now, we will generate the mesh configurations in order to route from edge to the sidecar of the deployment and from the sidecar to the simple-service itself. At the end of this step, the /ingress endpoint of the simple service should be accessible through the edge and properly return Ingress request to simple-service successful!.

Domain

The first object to generate is the ingress domain of the service.

Save this file as ingress-domain.json and apply the domain object using the greymatter cli:

Listener

Next, create the corresponding ingress listener of the service.

The combination of the ingress domain and listener opens 0.0.0.0:10808 to accept connections. The secret set on the listener specifies the SPIFFE identity for the simple-service pod, spiffe://quickstart.greymatter.io/simple-service. The sidecar will fetch the certificate for this identity via . The subject_names field of the secret specifies that only incoming connections with certificate SAN equal to spiffe://quickstart.greymatter.io/edge will be accepted by the listener.

Save this file as ingress-listener.json and apply it:

Proxy

The proxy object will indicate that the above domain and listener are meant to configure this specific sidecar. In order to link these objects to the deployment, the name field of this proxy object must equal the value of the deployment label greymatter.io/control, which in this case is simple-service.

Note: the domain_keys and listener_keys fields take a list of strings. When we add a second domain and listener in , we will add their keys to this proxy.

Save this file as proxy.json and apply it:

Clusters

There are two clusters necessary for ingress connections. The first is the edge-to-simple-service-cluster. This cluster will be used to locate and route to the simple-service sidecar from edge. In opposition to the listener secret above, the secret on the edge-to-simple-service-cluster specifies then that when connecting to the sidecar, use the spiffe certificate with identity spiffe://quickstart.greymatter.io/edge and only connect to certificate with SAN equal to spiffe://quickstart.greymatter.io/simple-service.

The instances field is left empty because Grey Matter Control will discover the sidecar with label greymatter.io/control: simple-service, and add the instances to this cluster because the name field matches. Thus, like the proxy object, it is important to note that the name field must equal the greymatter.io/control label.

The second cluster connects the sidecar to the simple-service, which is running on port 8080 as specified in the deployment. Because the two containers share a pod, this connection is plaintext over localhost and needs no secret. Because we know where the service is running in connection to the sidecar, localhost:8080, we can hardcode the instance.

Save these files as edge-to-simple-service-cluster.json and simple-service-cluster.json, respectively, and apply them:

Shared Rules

The two clusters created above, edge-to-simple-service-cluster and simple-service-cluster need two shared_rules objects to allow routes to connect to them.

When specifying routes from edge to the simple-service sidecar, we'll use edge-to-simple-service-rules, and when specifying routes from the simple-service sidecar to the service, we'll use simple-service-rules.

Save these files as edge-to-simple-service-rules.json and simple-service-rules.json, respectively, and apply them:

Routes

The last step in the ingress configuration is routes. There will be three routes for ingress to the service.

The first two specify when and how edge should connect to the simple-service sidecar. These two routes will be configured on the edge sidecar, and when a request comes in with path prefix equal to /services/simple-service/ or /services/simple-service, it will strip off that value from the path and forward the request to the cluster specified in the shared_rules object that it links to.

The last route configures the simple-service sidecar to route any incoming requests with path prefix "/" to the simple-service.

Save these files as edge-route.json, edge-route-slash.json, and service-route.json and apply them:

Once all of the configurations are applied, you should be able to access the simple-service at https://{your-gm-ingress-url}/services/simple-service/ and the ingress route at https://{your-gm-ingress-url}/services/simple-service/ingress. Note that if you try to hit https://{your-gm-ingress-url}/services/simple-service/egress, the request will fail with upstream connect error or disconnect/reset before headers. reset reason: connection termination. This is because we have not yet configured the service for egress actions.

3. Configure the deployment for egress actions

For any service being deployed in place of simple-service that generates requests to other services in the mesh, there is another set of configurations necessary.

For this walkthrough, we have configured the simple-service with environment variable EGRESS_ROUTE=http://localhost:10909/catalog/summary. The example that this will show is the simple-service /egress endpoint generating a request to the Grey Matter Catalog service, which is also running inside the mesh, and returning the json response from a GET request of its /summary endpoint.

SPIFFE/SPIRE and egress explanation

In order to allow the simple-service to make requests inside the mesh, we need to open up a second domain/listener combo on the sidecar, this time on a different port than the ingress at 10808. We will use 10909. In a non-SPIFFE/SPIRE setup, this second domain/listener is not necessary.

This is necessary for a SPIFFE/SPIRE setup because all of the sidecars are using SPIFFE certificates, fetched by the sidecar using Envoy SDS and not through a physical mount into the service or sidecar containers. Thus, the containers themselves can not have and do not need certificates mounted into them. Because of this, any request generated from within the service (service-A) can only be http. But, all incoming requests to a service (say service-B) within the mesh must go through it's sidecar, and the sidecar is only accepting https requests with specific SPIFFE certificates.

The services service-A and service-B can talk plaintext to their own sidecars. All sidecars have access to their SPIFFE identities via Envoy SDS, so two sidecars can communicate with each other over https. Thus, the flow of a request generated by service-A to service-B looks like the following:

This allows us to break down the need for the second, "egress" / combo on port 10909, and how the EGRESS_ROUTE is formed. With the sidecar for service-A (or in this example the simple-service sidecar), listening for plaintext connections over localhost at 10909, the simple-service itself can direct requests to its own sidecar at http://localhost:10909. The sidecar, then, can use path_matching on a configured to forward specific requests to an . As described below, this egress cluster will be configured with the correct SPIFFE identity for service-A, and the request will be sent to the ingress domain/listener for service-B

For this example, where simple-service wants to make a request to the Grey Matter Catalog service for its /summary endpoint, the flow will look like:

Egress Domain

The egress domain object will have force_https set to false, as we want the sidecar to accept plaintext connections on 10909. It will add the custom header x-forwarded-proto: https to requests coming from the domain.

Save this file as egress-domain.json and apply it:

Egress Listener

Save this file as egress-listener.json and apply it:

Egress Cluster

The egress cluster must have name equal to the greymatter.io/control pod label for the service it is trying to reach in order to Grey Matter Control to give it the instance for the service. In our case, this is the Grey Matter Catalog service, which has greymatter.io/control label "catalog".

The secret on this cluster will use the SPIFFE certificate with id spiffe://quickstart.greymatter.io/simple-service and connect only over connections with SAN spiffe://quickstart.greymatter.io/catalog.

Save this file as simple-service-to-catalog-cluster.json and apply it:

Egress Shared Rules

To configure the egress route to send requests to the simple-service-to-catalog-cluster, we'll generate a shared_rules.

Save this file as simple-service-to-catalog-rules.json and apply it:

Egress Route

For this example, the egress route we want is to the Grey Matter Catalog service. The way we have chosen to specify this is using path prefix /catalog on the request. This route will take requests into the simple-service-domain-egress with path prefix math /catalog, strip the prefix and forward the request through the cluster specified in shared_rules object simple-service-to-catalog-rules.

Save this file as simple-service-to-catalog-route.json and apply it:

Now there are several updates to existing Grey Matter objects that need to be made. First, we know that the simple-service-to-catalog-cluster will send requests to the Catalog service using SPIFFE certificate with id (or SAN) spiffe://quickstart.greymatter.io/simple-service. The Catalog service ingress listener secret will need to have this id added to its subject_names in order to accept this request.

Locate the secret field. The subject_names should contain only the edge identity, spiffe://quickstart.greymatter.io/edge. Change the subject names field to also allow our simple-service identity,

Lastly, we need to tell the object for the simple-service to add the egress domain and listener to the service configuration.

and add simple-service-domain-egress to the domain_keys and simple-service-listener-egress to the listener_keys. It should look like:

Now, the simple-service should be configured to successfully make requests to the Grey Matter Catalog service. Go to https://{your-gm-ingress-url}/services/simple-service/egress to test this out. The egress request should return a string of json, and if you hit the catalog /summary endpoint directly through edge, https://{your-gm-ingress-url}/services/catalog/latest/summary, you should see the same result.

For any internal services that your service needs to be able to connect to in the above way will need their own service-a-to-service-b cluster, route, and shared_rules like the egress ones above. They will connect to the egress domain.

Prerequisites

1. An existing Grey Matter deployment running on Kubernetes _(tutorial)

2. kubectl or oc setup with access to the cluster

3. greymatter cli setup with access to the deployment

Overview

1. Delete service from Intelligence 360 Application

2. Delete Edge routes

3. Delete pod with the service and sidecar

NOTE: The exact configurations and commands done here assume you've gone through the tutorial to setup this service. For any other service; you'll just need to substitute your object keys and service deployments where appropriate.

Steps

1. Delete Catalog Entry

The first step is to remove the service entry from the Catalog server so that users will not see it and expect to be able to use it.

This will remove the service card from display.

2. Edge Routing

Next we'll remove any Edge routing configuration. This will prevent users from using the service.

First delete the cluster object.

Then delete the route and shared_rules. We can delete these both with a single call because they're explicitly linked by object keys.

Shortly after these steps (generally a few seconds) the Edge will have removed the configuration to route to this service and users can no longer call the service.

3. Delete Deployment

Now that the service is no longer routed to from the edge, we can fully spin down the pod so that it no longer can receive any traffic and stops announcing into the mesh. This can be done in any way supported by k8s:

• Scale the deployment/replicaset to 0

• Delete the resources (deployment, replicaset, pod)

Assuming you've followed out tutorial, the deployment set can be removed the kubectl command:.

4. Delete Sidecar config

NOTE This step is optional for removing a service, and should only be done if this service is not expected to spin back up. These configuration objects are not sent to any Sidecar now that the pod is spun down, and nothing is referencing them for routing.

The last step is to delete the configuration objects in the API for this service.

First we'll delete the cluster for the Sidecar to talk to it's local microservice.

Then we'll delete the domain, and use the --deep option to make sure no other Sidecars are using it.

Third, delete the listener:

Then we can delete the route and shared_rules together

Finally, we'll delete the overarching proxy object.

The service and all configs are now removed from GreyMatter.

Envoy allows configuration of how many requests / second a service can field. This is useful in preventing DDOS attacks and otherwise making sure your server's resources aren't easily overrun.

Unlike circuit breakers which are configured on each cluster, rate limiting is configured across multiple listeners or clusters. Overall, circuit breakers are good for avoiding cascading failures of a bad downstream host. However, when:

[A] large number of hosts are forwarding to a small number of hosts and the average request latency is low (e.g., connections/requests to a database server). If the target hosts become backed up, the downstream hosts will overwhelm the upstream cluster. In this scenario it is extremely difficult to configure a tight enough circuit breaking limit on each downstream host such that the system will operate normally during typical request patterns but still prevent cascading failure when the system starts to fail. Global rate limiting is a good solution for this case.

This is because rate limiting sets a global number of requests / second across a service, which is independent of the number of instances configured for a cluster. This guide shows how to configure rate limiting on the edge node in a Grey Matter deployment to an edge namespace.

Prerequisites

1. setup with a running Fabric mesh.

2. An existing Grey Matter deployment running on Kubernetes _()

3. kubectl or oc setup with access to the cluster

Steps

1. Deploy Ratelimit Service

Rate limiting relies on an external service to regulate and calculate the current number of requests / second. For this example we use which is based on . is a good example of this architecture.

Begin by creating a deployment for the rate limit service by applying these configs to kubernetes. Note for all these examples I use edge as the namespace – this may differ for your deployment. Additionally, be sure to update the Redis password with your own:

When applied with kubectl, this service will serve as a central hub to limit the number of requests coming from the domain "edge" to 100 requests / second.

2. Configure Grey Matter Sidecar To Use Ratelimit Filter

In order to configure rate limiting on a sidecar, we need to configure the sidecar with an additional cluster that points to the rate limit service. As a convenience, Grey Matter Sidecar allows for defining a cluster using environment variables. The following sample environment variables define a cluster ratelimit that points to our deployed ratelimit service:

Make sure that you can see this cluster on whichever sidecar you have configured. It's available on localhost:8001/clusters under the ratelimit cluster.

Now let's update the listener config for the sidecar we've configured. Edit the listener in the Grey Matter CLI and add the following attributes:

You should see that requests to the ratelimit cluster succeed on the localhost:8001/clusters endpoint on the sidecar. You should also see logs in the ratelimit pod on every request since the logs are set to debug. If there are ever more than 1 request / second, all requests are blocked until the 1 request / second threshold is released.

3. Trust but Verify

The ratelimit we set can be tested by changing the number of requests / second to 1 and spamming the sidecar. Make a series of curl requests to the edge. The response code should be 429, although if TLS is enabled this often is just an SSL error. You should see also something like the following logs in the ratelimit pod:

This shows that the ratelimit service is registering calls from edge and is checking the requests / second against the limit of a maximum of 1 request / second. If you wish to raise the limit in a production environment, 100 requests / second is a good starting point.

References

The Grey Matter Command Line Interface (CLI) is a configuration tool for Grey Matter. Leveraging the Grey Matter Control API, the CLI mainly performs dynamic configuration of the Fabric mesh.

Prerequisites

To complete this tutorial, you’ll need an understanding of, and local access to the following environments and tools:

• Unix Shell (or equivalent)

• A

Step 1: Download and Install the greymatter CLI

The greymatter CLI is a binary written in Go. There are two ways to install it:

1. use , a manager for installing and setting different greymatter versions

2. download the binary manually from our release pages

We recommend using gmenv because it allows for easier versioning against different Grey Matter environments.

Option A (recommended): Use gmenv

• using homebrew or manually following the README

• (the email and password used to access ). For ease of use, it is recommended to .

• Run gmenv install 1.4.2 to install the latest version of the CLI. You should see:

Option B: Install the greymatter binary manually

Retrieve the or visit Grey Matter's to browse all released versions of the CLI. When prompted, enter the username and password associated with your Grey Matter account.

Alternatively, any artifact in nexus can also be downloaded with a terminal. The below command demonstrates how to do this with curl. Before executing, replace -u user.name@organization.com with your username, and make sure the desired artifact is specified.

greymatter is distributed as a precompiled binary. Once you have downloaded the desired release, install it with the following steps:

1. Unpack the tarball

   You will see the binaries:

2. Move the greymatter binary for your operator onto your system's PATH with name greymatter:

Step 2: Test Installation

Run a quick test of the binary with the following command to verify a successful installation.

If you don't see this response, verify that the greymatter binary is on your PATH.

Step 3: Configure Your Environment for Grey Matter

Once the CLI has been downloaded and installed, it will need to be configured for your Grey Matter installation to communicate with the Grey Matter Control API. If you don't yet have an existing Grey Matter installation, get started with the , and you will be able to configure the Grey Matter CLI and complete once it is up.

If you have an existing deployment running, configure the CLI by setting the following environment variables in your terminal. Note that as each deployment is different, the specific endpoint and security context will be different, so make sure to verify your settings against the deployed environment.

It is recommended to use an absolute path for your GREYMATTER_API_SSLCERT and GREYMATTER_API_SSLKEY variables.

NOTE: If you've used our - see the section for the specific configurations.

The full configuration and usage of the CLI can be found in the pages or by running greymatter --help, but some quick examples are shown here.

Configuration options can also be set by CLI flags with each use. These flags will take precedence over the set environment variables.

Step 4: Test connection to the Grey Matter Control API

List Zone

Use greymatter list zone with the following keys and certs to connect to the Grey Matter Control API.

The returned Zone indicates that the connection was successful, and you've been able to inspect the Fabric Mesh. You can now interact with your Grey Matter installation using the CLI!

If you see a connection refused error, your configuration is likely wrong. Check the values you have configured by running greymatter with no command, the options configured from the environment will be listed at the bottom of the output.

Questions?

The auditing capability of the Grey Matter Sidecar enables observability for all events within Grey Matter Fabric. This tutorial will guide you through a few easy steps to add an audit trail for your Fabric service.

Prerequisites

To complete this tutorial, you’ll need an understanding of, and local access to the following environments and tools:

Learn how to configure the Grey Matter Sidecar in Fabric to achieve the following:

• Dynamic configuration for adaptive proxying

• Service discovery through the Grey Matter Control API

Prerequisites

1. An existing Grey Matter deployment.

These examples show how you can apply mesh-wide policy changes using the cli and .

Prerequisites

1. command-line JSON processor

The Grey Matter Route object offers the ability to configure on specific routes of a domain. Read the reference documentation on per route filter configuration .

This guide will walk through the steps of setting two different configurations of the Grey Matter on different routes. To demonstrate this, provides the files to deploy a fibonacci service and configure it into the mesh for testing. For a step by step guide on service deployment, see the or the . To test the per filter route configuration for observables on a service already existing in the mesh, skip to .

There are four main filters that can be used to construct an Open ID Connect (OIDC) filter chain:

• • This filter checks the existence of a request attribute (header, cookie, etc.) and will copy/move it to the next filter in the chain.