Grey Matter Control performs service discovery and distribution of configuration in the Grey Matter service mesh. Control provides policy and configuration for all running sidecars within the Grey Matter platform.

The Control server works in conjunction with the Grey Matter Control API to manage, maintain, operate, and govern the Grey Matter hybrid mesh platform.

xDS

The Control server performs service discovery in the mesh and acts as an xDS server to which all proxies connect.

xDS is the generic name for the following:

• Endpoints (EDS)

• Clusters (CDS)

• Routes (RDS)

• Listeners (LDS)

Service Discovery

Service Discovery is the way Grey Matter dynamically adds and removes instances of each microservice. Discovery adds the initial instances that come online, and modifies the mesh to react to any scaling actions that happen. To keep flexibility in the Grey Matter platform, the Control server supports a number of different service discovery options and platforms.

The ability to automatically populate instances of a particular microservice comes from the clusterobject. In particular, the name field in the cluster object determines which nodes will be pulled out of the mesh and populated in the instances array. In the example below, the name is catalog. This means that all services that announce as catalog in service discovery, will be found and populated into the instances array after creation.‌

Create the following object:

‌

Will be populated in the mesh as:

‌

Even though the object was created with no instances, they were discovered from the mesh and populated. Now any service that needs to talk to catalog, can link to this cluster and address all live instance.‌

Configuration Discovery

Each proxy in the mesh is connected to the control plane through a gRPC stream to the Grey Matter Control server. Though gm-control-api houses all the configuration for the mesh, it's ultimately gm-control that turns these configs into full Envoy configuration objects and sends them to the proxies.‌

The configuration in the Control API is mapped to physical proxies by the name field in the proxy API object. It's very important that this field exactly match the service-cluster identifier that the intended target proxy used when registering with gm-control.

In the example below, the proxy object, and all other objects linked by their appropriate keys, will be turned into a full Envoy configuration and sent to any proxies that announce as a cluster catalog.‌

Services that announce as:

‌Will receive the config form the object below, because XDS_CLUSTER==name, and they're both in the same zone.

Questions?

Grey Matter consists of a control plane, data plane, mesh, telemetry-powered business intelligence, and AI. It can be deployed on multiple cloud-native or legacy infrastructures without placing predetermined downstream requirements on existing investments.

Learn about the inner workings of Grey Matter's three core components: Fabric, Data, and Sense.

Grey Matter enables unified hybrid microservice deployments and hybrid/multi-cloud operations without special requirements for underlying infrastructure like containers or container platforms regardless of cloud vendor, PaaS or infrastructure.

Overview of Grey Matter's Functionality

Grey Matter is composed of Fabric, Data, and Sense. Internal to each component is a series of microservices that offers several core features. Each feature simplifies technical challenges associated with service management, such as:

• Announcement

• Discovery

• Instrumentation

• Logging

Workload Distribution

The following diagram shows the workload distribution between Grey Matter's core components.

Grey Matter Fabric

Fabric powers the zero-trust hybrid service mesh, which consists of the , , , and . You can use Fabric to connect services regardless of language, framework, or runtime environment.

Secure network fabrics provide bridge points, observability, routing, policy assertion, and more between on-premise, multi-cloud, and multi-PaaS capabilities. Fabric offers workload distribution and management within a hybrid environment.

Support for Multiple Runtime Environments

Grey Matter supports multiple runtime environments with multi-mesh bridges as shown below. These environments include:

• Multiple cloud providers (i.e. AWS and Azure)

• Container management solutions (i.e., K8s, OpenShift and ECS)

• On-premise infrastructure

OSI Model Layers

Fabric operates at layers 3 (network), 4 (transport), and 7 (application) simultaneously. Providing a powerful, performant, and unified platform to run, manage, connect, and perform distributed workloads across a hybrid architecture.

Layer 3 operates at the TCP level. Responsible for transferring data “packets” from one host to another using IP addresses, TCP ports, etc., determining which route is the most suitable from source to its destination. At this level, network-segmentation is able to be performed using ABAC, RBAC, and NGAC policies set within each sidecar. More details can be found in the section.

Layer 4 coordinates data transfer between clients and hosts. Adding load balancing, rate limiting, discovery, health checks, observability, and more built on top of TCP/IP. Layer 3 and 4 alone live within the TCP/IP space and are unable to make routing decisions based on different URLs to backend systems or services. This is where layer 7 comes into the architecture.

Layer 7 sits at the top of the OSI model, interacting directly with services and applications responsible for presenting data to users. HTTP requests and responses accessing services, webpages, images, data, etc. are layer 7 actions.

The following graphic shows Fabric's basic capabilities--access, routing decisions, rate limits, health checks, discoverability, observability, proxying, network and micro-segmentation--and how they leverage all features found within each of the OSI layers described above.

Edge

Grey Matter Edge handles flowing through the mesh. Multiple edge nodes can be configured depending on throughput or regulatory requirements requiring segmented routing or security policy rules.

• Traffic flow management in and out of the hybrid mesh.

• Hybrid cloud jump points.

• Load balancing and protocol control.

• Edge OAuth security.

Control

• Automatic discovery throughout your hybrid mesh.

• Templated static or dynamic sidecar configuration.

• Telemetry and observable collection and aggregation.

• Neural net brain.

Security

Grey Matter Fabric offers the following security features:

• Verifies that tokens presented by the invoking service are trusted for such operations.

• Performs operations on behalf of a trusted third party within the Hybrid Mesh.

Sidecar

Add Grey Matter to services by deploying a sidecar proxy throughout your environment. This sidecar intercepts all network communication between microservices.

The Grey Matter Sidecar offers the following capabilities:

• Multiple protocol support.

• Observable events for all traffic and content streams.

• Filter SDK.

• Certified, Tested, Production-Ready Sidecars.

Once you've deployed the Grey Matter Sidecar, you can configure and manage Grey Matter with its control plane functionality.

Grey Matter Control Plane Functionality

• Automatic load balancing for HTTP, gRPC, WebSocket, and TCP traffic

• Fine-grained control of traffic behavior with rich routing rules, retries, failover, and fault injection

• A policy layer and configuration API supporting access controls, rate limits and quotas

• Automatic metrics, logs, and traces for all traffic within a cluster, including cluster ingress and egress

Example

The following diagram shows how the Grey Matter Sidecar would operate in a North/South traffic pattern.

Grey Matter Data

Grey Matter Data is an API that enables secure and flexible access control for your microservices. Data consists of Grey Matter Data and JWT server, and includes an API Explorer to help you manage the API.

Grey Matter Sense

Grey Matter Sense consists of four primary components: , , and .

Intelligence 360

Intelligence 360 is our user dashboard that paints a high-level picture of the service mesh. Intelligence 360 includes the following features:

• Mesh Overview

  • Running state of all services

  • Search, sort and filter options

• Historical metrics per service

SLO

Grey Matter Service Level Objectives (SLOs) allows users to manage objectives towards service-level agreements. These objectives can be internal to business operations or made between a company and its customers. They are generic and are valuable in more than one use case.

SLOs combine with Intelligence 360 time-series charts to visualize warning and violation thresholds for targeted performance analysis. These objectives are used even further to train Sense AI for service scaling recommendations.

Business Impact

Business Impact allows users to set metadata on services with the goal of associating how critical a service is towards the operations of a company, mission, or customer. Business Impact provides a list of values (Critical, High, Medium, Low) that correlates each service's business impact. Sense lets users of Intelligence 360 configure these values themselves, which can be used to filter and search via the mesh overview.

Catalog

acts as an interface between the data plane (network of sidecars) of the service mesh and Intelligence 360. Catalog provides a user-focused representation of the mesh.

Learn how to here.

Questions?

Need Help?

Simple Deployment Model

In the simple deployment model, there is a single point of ingress between the Client and the Grey Matter Edge. The Edge routes traffic to appropriate Grey Matter Sidecars.

Multi-Mesh Deployment Model

The multi-mesh deployment model extends the simple deployment model by allowing each mesh egress to communicate with another mesh ingress using mTLS, as shown below.

Questions?

The Grey Matter Sidecar is a L7 reverse proxy based off of the popular Open-Source Envoy Proxy. Grey Matter's proxy enhances the base capabilities with custom filters, logic, and the ability for developers to write full-featured Envoy filters in Go.

Fabric Mesh

The primary use of the Grey Matter Sidecar is to act as the distributed network of proxies in the Grey Matter Fabric service mesh. In this use-case; each proxy starts out with very simple configuration, which is then modified by the control plane to suit the changing needs of the network. The documentation here is focused on the individual proxy itself; low-level configuration, filter specifications, etc.

How does event auditing work?

Individual Service

At the level of the individual service, event auditing works as follows:

1. One proxy collects all metrics that happen on the individual service.

2. At the Edge, they extract the PKI/cert.

3. The user that has accessed the service from outside Fabric is then decomposed based on one of the observable fields emitted by the Sidecar proxy.

4. This information, coupled with IP address information from the originating request, is added to the stack of the

Service-to-Service

At the service-to-service level, the sidecar tracks service-to-service calls within Fabric. This enables architecture inference and service dependency observation.

Observable Indexer

Grey Matter also has an observable indexer which can capture geolocation info and move it into Elasticsearch. Customizable event mappings are also available. These can be tailored per individual route so that a POST request may result in an EventAccess event in one route, while resulting in EventCreate on another.

Questions?

Grey Matter is a zero-trust hybrid mesh platform built using open architecture and mesh app and service architecture (MASA) principles.

Each microservice within Grey Matter runs and scales independently to improve secure interoperations, resiliency, continuity of operations, and insight for your business. Our omnichannel support provides rich, fluid, and dynamic connections between people, content, devices, processes, and services.

Core Architectural Principles

Our core architecture principles support the following business needs.

Security

Designed to operate using a zero-trust threat model to ensure each service running within a Grey Matter enabled hybrid mesh is appropriately secured, observed, and managed.

Portability

Enable on-premise, multi-cloud, and multi-platform as a service (PaaS) runtime environments.

Cloud-native

Built with elasticity, high availability, and cloud computing models in mind - provides a unified mesh platform to build applications as microservices, utilize container management solutions, and dynamically orchestrate workloads across hybrid enterprise.

Web-scale

Provides a solid foundation to scale with the growth of your business. Enabling modern architectural patterns supporting rapid increase or decrease in traffic volume, maintaining business insight for effectiveness and efficiency, and aiding in the reduction of bottlenecks when the time matters.

Modular

Modular service delivery - enabling loosely coupled systems and services developed independent of each other, taking advantage of continuous delivery to achieve reliability and faster time to market.

Interoperability

Creates a secure unified zero-trust network fabric, allowing systems to interchangeably serve or receive services from other systems providing enterprises the ability to perform multi-environment segmentation and observe traffic flows between environments. Managed through a runtime environment agnostic Grey Matter Control API.

Adaptive

Able to react to digital business changes, providing a pathway enabling business insight, security, and connectivity across multiple environments reducing complexities while increasing and facilitating a business's digital transformation journey.

Intelligent

Use of artificial intelligence (AI) techniques simplifying and assisting a user experience while providing business insight and fleet wide management across Grey Matter connected resources.

Automation

Integration into any ecosystems and end-to-end automation through the lifecycle of the mesh app and service architecture.

Questions?

Data

Grey Matter Data is a microservice for the versioned and encrypted storage of media blobs and assets. It is a high-performance time-series object storage. It contains an immutable sequence of events which collectively describes a file system at a given moment in time. If the first event is a creation of an object and the second is its deletion, then the object will not appear in listings as of the current time, but it will appear as of any time after the creation and before the deletion.

Questions?

Prerequisites

1. git installed

2. helm v3

3. envsubst (a dependency of our helm charts)

4. eksctl or an already running Kubernetes cluster.

Steps

1. Install a Kubernetes Cluster

NOTE: if you already have a Kubernetes cluster up and running, move to step 2. Just verify you can connect to the cluster with a command like kubectl get nodes

For this deployment, we'll use to automatically provision a Kubernetes cluster for us. The eksctl will use our preconfigured AWS credentials to create master nodes and worker nodes to our specifications, and will leave us off with kubectl setup to manipulate the cluster.

The regions, node type/size, etc can all be tuned to your use case, the values given are simply examples.

Cluster provisioning usually takes between 10 and 15 minutes. When it is complete, you will see the follwing output:

When your cluster is ready, run the following to test that your kubectl configuration is correct:

2. Clone the Grey Matter Helm Charts Repo

Though Helm is not the only way to install Grey Matter into Kubernetes, it does make some things very easy and reduces a large number of individual configurations to a few charts. For this step, we'll clone the public git repository that holds Grey Matter and cd into the resulting directory.

NOTE: this tutorial is using a release candidate, so only a specific branch is being pulled. The entire repository can be cloned if desired.

3. Setup Credentials

Before running this step, determine whether or not you wish to install . If so, determine whether or not you will use S3 for backing. If you do want to configure Grey Matter Data with S3, follow the guide. You will need the AWS credentials from here.

To set up credentials, we need to create a credentials.yaml file that holds some secret information like usernames and passwords. The helm-charts repository contains some convenience scripts to make this easier.

Run:

and follow the prompts. The email and password you are prompted for should match your credentials to access the Decipher Nexus at . If you have decided to install Grey Matter Data persisting to S3, indicate that when prompted, and provide the access credentials, region, and bucket name.

Note that if your credentials are not valid, you will see the following response:

4. Configurations

To see the default configurations, check the global.yaml file from the root directory of your cloned repo. In general for this tutorial, you should use the default options, but there are a couple of things to note.

• If you would like to install a Grey Matter Data that is external and reachable from the dashboard, set global.data.external.enabled to true.

  • If you are installing data and set up your , set global.data.external.uses3 to true.

• If you plan to update ingress certificates or modify RBAC configurations in the mesh, set global.rbac.edge

You can set global.environment to eks instead of kubernetes for reference, but we will also override this value with a flag during the installation steps in .

5. Install Grey Matter component Charts

Grey Matter is made up of a handful of components, each handling different pieces of the overall platform. Please follow each installation step in order.

1. Add the charts to your local Helm repository, install the credentials file, and install the Spire server.

2. Watch the Spire server pod.

   Watch it until the READY status is 2/2, then proceed to the next step.

3. Install the Spire agent, and remaining Grey Matter charts.

6. Accessing the dashboard

NOTE: for easy setup, access to this deployment was provisioned with quickstart SSL certificates. They can be found in the helm chart repository at ./certs. For access to the dashboard via the public access point, import the ./certs/quickstart.p12 file into your browser of choice - the password is password.

An will be created automatically when we specified the flag --set=global.environment=eks during installation. The ELB is accessible through the randomly created URL attached to the edge service:

You will need to use this value for EXTERNAL-IP in the .

Visit the url (e.g. https://a2832d300724811eaac960a7ca83e992-749721369.us-east-1.elb.amazonaws.com:10808/) in the browser to access the Intelligence 360 Application

7. Configure the CLI

If you intend to move onto into your installation, or otherwise modify/explore the Grey Matter configurations, you will need to .

For this installation, the configurations will be as follows. Fill in the value of the edge service's external IP from the for <EDGE-EXTERNAL-IP>, and the path to your helm-charts directory in <path/to/helm-charts>:

Run these in your terminal, and you should be able to use the CLI, greymatter list cluster.

You have now successfully installed Grey Matter!

Cleanup

If you're ready to shut down your cluster:

Delete the Grey Matter Installation

Delete The EKS Cluster

NOTE: this deletion actually takes longer than the output would indicate to terminate all resources. Attempting to create a new cluster with the same name will fail for some time until all resources are purged from AWS.

Service meshes, microservices, server-less, and containers are key elements of Mesh application and service architecture (MASA) implementations. MASA, APIs, and internal traffic patterns represent one of the most effective pathways to enterprise modernization, but this doesn’t come without challenges.

Industry has signaled increased interest in zero-trust infrastructure for service-to-service mTLS connections, scheduled or on-demand key rotations, service cryptographic identifiers, observability (continuous monitoring, granular audit compliance, etc.), service-level management, and policy management throughout the enterprise service fleet.

Security Models

Understanding how the roles of Authentication, Authorization, Claims, and Principals will play within your MASA is important (figure 1). Authentication and authorization are both significant in any security model, but follow different concepts and implementation patterns. Authentication establishes and confirms an identity. Authorization takes action based on the confirmed identity authenticated. Principals are asserted claims that provide entitlements granting access to systems, services, or data based on Role-based Access Control (RBAC), Attribute-based Access Control (ABAC) and Next Generation Access Control (NGAC) controls.

Authentication

Grey Matter's authentication scheme establishes identities for every transaction within the platform. There are two types of identities: users and services.

User Authentication methods:

• OpenID Connect (OIDC)

• mTLS x.509 certificates (Distinguished names represent who the user is)

Service-to-Service Authentication methods:

• mTLS x.509 certificates (SPIFFE identities are incorporated into the x.509 certificate)

While distinct, these identities are not mutually exclusive. One of the most common access patterns within Grey Matter is a service making a request to another service on behalf of a user. In this case, there are three identities (two services and a user), each of which must be verified in order for the transaction to succeed. As users or services authenticate with Grey Matter, principals are asserted and flow to upstream services. This ensures that upstream services are aware of the entity (user or service) making a request. Grey Matter supports user authentication and service-to-service authentication methodologies identified below.

User Authentication with an OIDC Provider

Grey Matter integrates with existing public OIDC providers (Google, Github, etc.) or private OIDC providers (e.g., Ory Hydra) to support user authentication. OIDC is an authentication protocol built on top of OAuth 2.0 that allows delegation of authentication responsibility to a trusted external identity provider. Many implementations of OIDC providers are available and support on premise, cloud or as a service via a host of underlying technologies (e.g., LDAP). This sequence diagram (figure 3) shows the OIDC flow within Grey Matter.

1. The client initiates a request to Grey Matter Edge.

2. Grey Matter Edge responds with a 302 HTTP status code used to perform URL redirection, along with a callback URL.

3. Based on the redirect URL, the client initiates a request to the specified OIDC provider.

4. Once the client is authenticated, the OIDC provider responds with a 302 HTTP code (based on the callback URL) and provides an OIDC code.

x.509 Certificates

Grey Matter supports x.509 for both users and for service to service transactions.

1. A client (user or service) initiates a request to a server.

2. The server responds with its server certificate.

   2.1. The client verifies that the server’s certificate is valid based on its certificate information.

3. The server requests the client's certificate.

User Authentication with x.509 Certs

Enterprise IT organizations that have existing public key infrastructure (PKI) in place for user authentication can pass their certifications with requests made to the Grey Matter Edge.

Service Authentication with x.509 Certs

Service authentication, (service-to-service communication), is based solely upon x.509 certificates and mTLS. Grey Matter Fabric is installed with a certificate authority that issues and reissues short- lived x.509 certificates to each sidecar proxy for intermesh communication. Each certificate contains a SPIFFE identity that uniquely identifies the sidecar to which it is issued. No sidecar will accept a connection from any service that does not present a certificate issued by the certificate authority. Like user authentication, these service identities enable authorization.

Note: In cases where requests already contain a signed cookie the edge simply verifies the signature and expiry. If valid, the edge forwards the request. If not valid, the request is treated as unauthenticated.

Authorization

Authorization is the process by which identities (users or services) are granted permission to access resources within the mesh. For example, we may wish to restrict access to a specific resource to a limited set of users, services or data. As an added complication, it is often more desirable to grant or deny access for a resource to entire classes of identities (i.e., administrative users or trusted services). Grey Matter uses the authenticated identities and their attributes to support fine-grained access controls using the following methods:

• Authorization Filters

• Data Authorization via the Grey Matter Data Platform Service

It's important to note that Sidecar-to-Sidecar (service-to-service) authorization follows similar patterns of a user with the exception that sidecar identities typically do not include additional attributes; however, there is nothing precluding the addition of attributes for a sidecar identity.

Authorization filters

Upon choosing an authorization pattern, access control becomes a deployment concern, not a development concern. Allowing microservice developers to focus on business value since their services will not receive any unauthorized request. Authenticated identity and attributes are available to the service should they be required.

The Grey Matter Sidecar uses authorization filters to manage who is allowed to access which resources and how. Since all requests to the mesh are authenticated, filters can be dynamically configured at runtime with no additional requirements. Attribute based authorization is also implemented via Grey Matter Sidecar filters but requires that requests contain a signed JSON Web Token (JWT) containing the identity claims. The creation and population of these tokens is left to the enterprise.

List Authorization Filter

The Grey Matter Sidecar supports list-based authorization decisions within the ListAuth filter. This filter allows whitelisting and blacklisting of individual identities based upon the identities distinguished name (i.e, “cn=user, dc=example, dc=com” or “cn=web server, dc=example, dc=com”) or relative distinguished name (i.e., “dc=example, dc=com”). This filter applies to all requests for the proxied service or services.

Role-Based Access Control Filter

The Grey Matter Sidecar supports fine grained authorization decisions to authorize actions by identified clients using Role-Based Access Control (RBAC). This filter allows complex whitelisting and blacklisting of individual identities based upon the identities distinguished name. Matching of regular expressions is supported to add additional flexibility. Further, whereas the ListAuth filter applies to all requests, the Role Based Access Control Filter can be defined for any combination of service, route, or verb. This is useful to explicitly manage callers to a service running within the Grey Matter mesh platform and protect the mesh from unexpected or forbidden agents.

Supported HTTP verbs include:

• GET The GET method requests a representation of the specified resource. Requests using GET should only retrieve data.*

• POST The POST method is used to submit an entity to the specified resource, often causing a change in state or side effects on the server.*

• PUT The PUT method replaces all current representations of the target resource with the request payload.*

Definitions as described by the Mozilla Developer Network (MDN).

In situations where the identity is not sufficient to make all authorization decisions, the Grey Matter Sidecar can enforce finer-grained control based upon identity attributes, provided the request contains a sign JWT.

Using the RBAC filter, rules can be created to authorize specific claims found within a JWT to perform specific actions. This requires an external service to generate a signed JWT for each request. Since the JWT is included as a header, if a JWT is passed, it will propagate to all sidecars in the request chain. With that said, if the request is completed—meaning the destination service has received it, processed it, and invokes another service in the Mesh—this is a new request and the calling service would be required to pass the JWT for further authorization purposes.

Custom Access Control Filter

The Grey Matter sidecar offers a custom filter interface, so customers have the ability to create business-specific logic around their security and regulation concerns if required. This makes the mesh fully adaptable to an enterprise’s needs, and provides a way to take advantage of existing IT investments.

Data Authorization

One of the unique facets of Grey Matter is that data security and sharing is addressed by Grey Matter Data. As enterprises shift from monoliths to microservices, data tends to be duplicated across the architecture. Grey Matter Data provides a service to address the secure sharing of this data without marshalling it into and out of processes. This feature is described in greater detail in the Data Segmentation portion of this document, but the pattern employed by Grey Matter Data can be used by any service to enforce complex security policies for any resource via the Grey Matter JWT Security Service or customer JWT service adhering to the Grey Matter Data interface.

Traffic Patterns

One key feature of the Grey Matter hybrid mesh is its ability to secure, manage, and govern the traffic patterns of running services.

East/West Traffic

East/West traffic within the Fabric Mesh should be done via mTLS. Grey Matter uses two methods enabling this: direct integration with existing CAs and, automatic setup via SPIFFE/SPIRE. For integration with existing CAs, each sidecar in the mesh is configured to use provided x. certificates. In the automatic setup, each sidecar uses a unique SPIFFE ID to authenticate with SPIRE servers. Unique short-lived x.509 certificates are then automatically created and rotated for each connection between sidecars.

North/South Traffic

North/South traffic patterns use the Grey Matter Edge to establish principals and pass them to called services. The Edge supports both OIDC and mTLS x.509 certificate authentication modes, however, the Fabric Mesh is not limited to a single Edge. Multiple nodes can be configured to expose both authentication modes wherever an access point is needed. Note that the Edge node does not have to be exposed to a publicly addressable URL. In many cases, an API mediation layer may be put in front of the Edge node. In all cases, the Edge node is responsible for verifying and ensuring that the proper principals are available for downstream services within the Fabric Mesh to consume.

Principals such as user identities are moved by the Edge node into a user_dn header which flows through the entire service-to-service request chain. Each following link in the request chain is performed via mTLS, with each unique service using automatically rotating x.509 certificates established via SPIFFE Identities and the SPIRE framework.

In some cases traffic needs to flow outside of the mesh. Common scenarios include mesh-to- mesh communications, proxying to serverless functions, and supporting legacy systems that can’t be moved directly into the mesh. In all these cases, proxies are setup within the mesh with the sole purpose of communicating outside. Principals are established at the Edge. Inter-mesh communication is still handled by mTLS, and requests are authenticated via the outside system by whatever method it accepts: RPC, HTTP, mTLS, or OIDC.

Traffic Splitting

Traffic splitting is another important pattern in stable environments. Traffic splitting allows configurable service requests to siphon off percentages of requests to another source. This allows services, apps, or entire meshes to experience small amounts of live traffic while keeping most users on the original source. The percentage of users on the original service is then decreased until the service is fully migrated.

Circuit Breaking

Circuit Breaking is a way for each sidecar to protect the thing it is proxying to, but it is not a way to have that proxy harden itself. Grey Matter provides circuit breakers at every point in the mesh.

The most common place for this to occur is at the edge, where a DDOS could overwhelm the edge nodes themselves. To solve this, we employ Rate Limiting, which can protect the edge node from accepting too many requests and opening too many file handles and crashing. With proper configuration, each sidecar ceases queueing new requests before they’re overwhelmed, allowing the service time to heal. This ensures capabilities can withstand malicious attacks and accidental recursive network calls without going down.

Network Segmentation

Enterprises prefer hybrid environments capable of leveraging unified on-premise and cloud resources. Traditional networking patterns use features such as VLANs to create perimeter-based firewalls, but this concept breaks down with modern mesh application service architecture (MASA) patterns. In MASA, services are designed to be ephemeral, dynamically generating different IP:PORT pairs each time a new instance spins up.

Securing this type of architecture requires network segmentation. Grey Matter isolates services and network fabric communications to specific runtime environments or infrastructure resources. Grey Matter Fabric supports segmentation to a very fine level of granularity. Each service launched onto Fabric comes online with no knowledge of or connections to any other point on the mesh. The desired mesh is then built up through configuration with the required network topology. Segmentation is enforced through routing rules, service discovery, and mTLS. Dynamic configuration can facilitate any permutation of intra-mesh communication required. In addition to segmentation of individual meshes, Grey Matter can also support multi-mesh operations. This allows the bridging of environments already physically or logically isolated from each other.

Micro-segmentation

Micro-segmentation is a method of creating secure isolation zones either on-premise or in the cloud in order to separate different workloads. Authentication plays a key role in micro-segmentation. Authentication is responsible for establishing network communications and flow through the mesh. Strong authentication models enable Grey Matter to perform micro-segmentation for users, services, and data throughout the mesh.

User-to-Service segmentation is controlled through user authorization signatures. These can be coupled with claim-based assertions. User identities and claims flow through mTLS-encrypted communication channels established by service-to-service micro-segmentation patterns. Complex security policies within each sidecar allow ABAC/RBAC down to the service, route, and HTTP verb permit. This enables a very high degree of isolation. ABAC/RBAC policies cannot be achieved without strong authentication methodologies establishing identities for both users and services.

Service-to-Service segmentation is controlled through mTLS certificates and SPIFFE identities. These can be coupled with claims-based assertions and ABAC/RBAC policies. Images in the following section illustrate how this is achieved.

Data Segmentation

Grey Matter’s data segmentation capability is a key differentiator. Data segmentation is the process of dividing data and grouping it with similar data based on set parameters. Grey Matter Data adds complex policy assertions to stored objects. These object policies govern which users or services may access the objects. Objects stored within Grey Matter Data are encrypted at rest and in transit. A JSON Web Token (JWT) is provided to gain access to an object stored in Data.

The token’s claims are dynamically mapped to the policies stored with the object. JWTs for both users and services can be created enabling end-to-end security using authentication principals.

The example above shows how data-segmentation is achieved through simple policy. However, the Grey Matter Data policy engine is designed to deal with complex rules designed to suit any scenario. The following scenario presents a more complex use case.

1. Sidecar A saves an object into Data and provides access privileges to Sidecar B’s SPIFFE identity. Sidecar A dynamically discovers Data via the Data Sidecar routing information.

2. Data Sidecar receives Sidecar A’s request and streams the object (with policy) into the Grey Matter Data node.

3. Sidecar B (through a means of event-based architecture patterns) is notified that Sidecar A just saved an object of interest into Data. Sidecar B calls into Data (through the Data Sidecar) to retrieve the object. Sidecar B’s SPIFFE identity is passed along with the request.

   3.1. Data Sidecar receives the request from Sidecar B and passes it to Data. Data uses the Sidecar B principal (i.e. SPIFFE identity) to receive Sidecar B’s JWT claims and authorize access to decrypt and retrieve the object.

Since Grey Matter uses a unified principal model, data segmentation can be achieved for users as well. Grey Matter Data policies can be set to identify different access privileges for services and users on a single stored object, and can be customized around business needs. This paradigm provides a new model that combines network, information assurance, and protection concepts around zero-trust.

Grey Matter Data supports the ability to host multiple Data nodes available through different routing rules. When coupled with other segmentation features, enterprises are able to further isolate how information is stored, accessed, and controlled based on customer regulations and requirements.

For example, logs and observable traffic can be isolated based on zones. Data nodes with specific routing rules and policies are set to enforce the topology. Customer application data can be stored and accessed via different Data nodes (on-premise or in the cloud) and tightly controlled at the micro-segmentation layer or via data policies. These types of flows are depicted in the diagram below.

Conclusion

Grey Matter’s zero-trust threat model ensures security across every service in the hybrid mesh. Each transaction is authenticated and authorized through a combination of mTLS and SPIFFE authentication and SPIRE authorization providing multiple layers of zero-trust security. Grey Matter also supports fine-grained access control by combining authenticated identities with policy-enforced object authorization and enables East-West and North-South traffic pattern splitting and shadowing for in-depth monitoring and configuration. Finally, Grey Matter uses network and data segmentation to decompose operations to their most basic elements, to mitigate cyber intrusion impacts, and to optimize operations.

Questions?

This guide will help you set up a secure, zero-trust environment in Grey Matter to achieve the following:

• Establish trust in user identities

• Enforce adaptive and risk-based policies

• Enable secure access to all apps

• Enforce transaction and data security

Grey Matter uses to enable zero-trust security. For more information about how Grey Matter uses SPIRE, see the .

Learn more about Grey Matter's approach to zero-trust security here.

Prerequisites

• Unix shell and Decipher account

• helm and kubectl installed

• A Kubernetes or OpenShift deployment of Grey Matter with SPIRE enabled

Step 1: Install

To install Grey Matter using SPIRE, verify that global.spire.enabled is true (the default) for your helm charts setup and .

Step 2: Deploy a new service

For a full walkthrough of an example service deployment in a SPIRE enabled environment, see .

To adapt an existing service deployment to enable SPIRE, add this environment variable to the sidecar container:

Then add the following to the deployment volumes:

and mount it into the sidecar container as:

This creates the Unix socket over which the sidecar will communicate with the SPIRE agent.

Step 3: Mesh configurations

There are several updates to make to the mesh configurations for a new service to enable SPIRE. The following describe updates necessary to configure ingress to the service using SPIRE, if your service also has egress actions, check out the .

Domain

If you have existing mesh configurations for this service in a non-SPIRE installation, remove any from the ingress domain object, but keep force_https to true. The domain should look like .

Listener

The of the listener object is used to configure ingress mTLS using SPIRE.

If you installed Grey Matter using the helm charts, each deployment should have a label with key greymatter.io/control and value the name of the service (see ). This value will be used to indicate the SPIFFE ID for a sidecar.

Let {service-name} be the value of the label greymatter.io/control in your service deployment. Add the following secret to your listener object:

Once this is configured, the sidecar will use its SPIFFE certificate for ingress traffic on this listener.

Edge to new service routing

The cluster created for edge to connect to the service will need a similar update for egress traffic to the new service. Remove any ssl_config on the edge-to-{service-name}-cluster and set the secret instead:

and should be configured as usual.

Step 4: Test

When you setup services to participate in the mesh, SPIFFE identities are setup for them. This means that the service will get a certificate that is made for that service. As an example of probing into data, you can use openssl to verify that it is setup to use SPIFFE.

In a kubernetes setup, you can find the ip of your deployment with kubectl describe pod {pod-id} | grep IP. Copy this ip and use openssl to check the certificate. You can use openssl from within the data container -

and then to check your service:

or

You should see from the info that the certificate chain and SAN that the certificate your service is presenting is from SPIRE.

You can also verify that SDS is working for your service by execing into its sidecar pod kubectl exec -it {pod-id} -c sidecar -- /bin/sh and running curl localhost:8001/certs. If the sidecar is configured properly, it's SPIFFE certificate will be listed there.

Questions?

HashiCorp Consul

Description

gm-control consul is used to discover service instances from the Consul service registry.

Usage

gm-control [GLOBAL OPTIONS] consul [OPTIONS]

Help

To list available commands run with the global help flag, gm-control get --help:

gm-control can be configured to log a leaderboard of non-2xx requests on a time interval to stdout. This is useful as a quick way to see which endpoints are performing poorly throughout the mesh, without getting into advanced debugging.

Configuration

Leader board logging is configured with two following parameters:

Leaderboarding can be disabled by setting GM_CONTROL_XDS_GRPC_LOG_TOP to 0.

Schema

Leaderboards are logged to standard out in the format:

Example:

When working with service meshes on various platforms, there is a benefit in supporting multiple methods for accessing secrets. Some platforms (like Kubernetes and OpenShift) provide means of securely storing secrets and mounting them in to running containers.

Others, like AWS ECS and the AWS Secrets manager, don't support such easy operations. To support operations on these other platforms, the Grey Matter Proxy contains functionality to parse a limited selection of Base64 encoded SSL certificates and write them directly to disk.

Questions

Envoy Admin Interface

Envoy has a built-in admin server that has a large amount of useful tools for debugging. The full abilities of this interface can be found in the docs, but some highlights are given here.

Log Levels

Envoy has the ability to set different log levels for different components of the running system. To see how they're all currently set:

NOTE all examples below assume the admin interface is started on port 8001 (the default option). Adjust according to your configuration.

You can set log levels of "trace, debug, info, warning, error, critical, off" on the global state.

Alternatively, you can set the level of just a specific logger with a format similar to the below. This one changes just the logger for the filters.

You can deploy the Grey Matter JSON Web Token (JWT) Service many ways, including the following:

• The preferred approach is to deploy via a Docker container running inside of an OpenShift or Kubernetes Pod.

• The service is also packaged as a TAR file. The TAR contains an executable binary file you can deploy to a server.

Follow the configuration requirements below to set up the Grey Matter JWT Security service.

You can configure the gm-jwt-security service to search an LDAP server for user payloads. To use LDAP as a backend service, refer to the following configuration options.

Enable LDAP Configuration

To enable, USE_LDAP must be set to true.

Grey Matter Intelligence 360 is configured with the following environment variables set on the host machine. The term host machine can apply to an AWS EC2 server, Docker container, Kubernetes Pod, etc.

Configuration Variables

Questions

To enable TLS support for the service, perform the following steps:

1. Set ENABLE_TLS to true

2. Specify cert, trust, and key either through a volume mount (recommended) or the following environment variables

Summary

Each domain controls requests for a specific host:port. This permits different handling of requests to domains like localhost:8080 or catalog:8080 if desired. If uniform handling is required, wildcards are understood to apply to all domains. A domain set to match *:8080 will match both of the above domains.

NOTEThe domain port should, in most cases, match the port of the exposed on the proxy. If they do not match, users will need to supply HOST: header keywords to all requests to match the virtual domain.

Features

• Virtual host:port matching and redirecting

• GZIP of requests

• CORS

• Setting custom headers for downstream requests

Example Object

TLS Configuration

NOTE Do not set an ssl_config on any domain object whose service you want to use SPIFFE/SPIRE. If a domain ssl_config is set, it will override the secret set on the corresponding listener and the mesh configuration will be wrong.

The domain object has an optional ssl_config field, which can be used to set up TLS and specify it's configuration. The Domain SSL Config Object appears as follows:

The Domain SSL Configuration is used to populate a for the Envoy Listener.

The sni field for a domain accepts a list of strings and configures the Envoy Listener to detect the requested Server Name Indication.

To specify a minimum and maximum TLS protocol version, set the protocols field to one of the following: "TLSv1_0", "TLSv1_1", "TLSv1_2", "TLSv1_3". If one protocol is specified, it will be set as both the minimum and maximum protocol versions in Envoy. If more than one protocol version is specified in the list, the lowest will set the minimum TLS protocol version and the highest will set the maximum TLS protocol version. If this field is left empty, Envoy will choose the default TLS version.

The cipher_filter field takes a colon : delimited string to populate the cipher_suites cipher list in envoy for TLS.

redirects

This field can be used to configure redirect routes for the domain. See for details.

Fields:

• name

  • the name of the redirect

• from

Envoy Reference

Fields

domain_key

A unique key used to identify this particular domain configuration. This key is used in proxy, listener, and route objects.

zone_key

The zone in which this object will live. It will only be able to be referenced by objects or sent to Sidecars that live in the same zone.

name

The name of this virtual domain, e.g. localhost, www.greymatter.io, or catalog.svc.local. Only requests coming in to the named host will be matched and handled by attached . Is used in conjunction with the field.

This field can be set to a wildcard (*) which will match against all hostnames.

port

Set the specific port of the virtual host to match. Is used in conjunction with the field.

E.g. port: 8080 and name: * will setup a virtual domain matching any request made to port 8080 regardless of the host.

ssl_config

for this cluster. Setting the SSL Config at the domain level set this same config on all that are directly linked to this domain.

redirects

Array of URL

gzip_enabled

DEPRECATION: This field has been deprecated and will be removed in the next major version release.

This field has no effect.

cors_config

A to attach to this domain.

aliases

An array of additional hostnames that should be matched in this domain. E.g. name: "www.greymatter.io" with aliases: ["greymatter.io", "localhost"]

force_https

If true, listeners attached to this domain will only accept HTTPS connections. In this case, one of the or fields should be set. If false, attached listeners will only accept plaintext HTTP connections.

custom_headers

An array of header key, value pairs to set on all requests that pass through this domain.

E.g.

checksum

An API calculated checksum. Can be used to verify that the API contains the expected object before performing a write.

Summary

Example object

Fields

cipher_filter

. If specified, only the listed ciphers will be accepted. Only valid with TLSv1-TLSv1.2, but has no affect with TLSv1.3.

Examples include the values below, but full options should be found in the link above.

• [ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305]

• [ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305]

• ECDHE-ECDSA-AES128-SHA

• ECDHE-RSA-AES128-SHA

protocols

Array of SSL protocols to accept: "TLSv1, TLSv1.1, TLSv1.2, TLSv1.3"

cert_key_pairs

Array of (cert, key) pairs to use when receiving requests on this . Each cert or key must point to files on disk.

require_client_certs

If true, client cert verification will be performed. false will disable this check and not require client certificates to be presented when connecting to this listener.

trust_file

String representing the path on disk to the SSL trust file to use when receiving requests on this . If omitted, then no trust verification will be performed.

sni

String representing how this listener will identify itself during SSL SNI.

Summary

Example object

Fields

name

Header key to match on. Supports regex expressions.

value

Header value to match on. Supports regex expressions.

case_sensitive

If true, then the regex matching will be case sensitive. Defaults to false.

invert

If true, invert the regex match. This allows easier "not" expressions.

For example, to match only X-Forwarded-Proto: "http":

But to match anything NOT "https":

Summary

Example object

Summary

Redirects specify how URLs may need to be rewritten. Each Redirect has a name, a regex that matches the requested URL, a to indicating how the url should be rewritten, and a flag to indicate how the redirect will be handled by the proxying layer.

Example object

Fields

name

Common name for this redirect, e.g. "force-https".

from

Regex pattern to match against incoming request URLs. Capture groups set here can be used in the field.

to

New URL of the redirect. Can be direct string (") or reference capture groups from the field ("").

redirect_type

One of "permanent" or "temporary". Selection "permanent" will set the response code to 301, "temporary" will be 302.

header_constraints

Array of that must match for the redirect to take effect.

Summary

Grey Matter supports the configuration of cross-origin resource sharing on a sidecar. CORS can be configured to allow an application to access resources at a different origin (domain, protocol, or port) than its own.

For more information on CORS, see this CORS reference.

Simple Requests

Simple requests are classified as requests that don't require a . This distinction is made between requests that might be dangerous (i.e. modifies server resources) and those that are most likely benign. A request is considered simple when all of the following criteria is true:

• The method is GET, HEAD, or POST

• Only are present

• The Content Type header is set to one of application/x-www-form-urlencoded

A more comprehensive list and explanation can be found .

As an example, say an app running at http://localhost:8080 is trying to call a backend service with a Grey Matter sidecar at http://localhost:10808. Without a CORS configuration, this request would fail, because the app localhost:8080 is trying to access resources from a server at a different origin, localhost:10808. To solve this, the following CORS config is set on its domain:

With this configuration, if a simple request comes in to the sidecar from the app, it will have an Origin header value of http://localhost:8080, and this request will succeed. The server will attach a header Access-Control-Allow-Origin: http://localhost:8080 to the response, which signals to the browser that this request is allowed.

Preflight Requests

are initiated by the browser using the OPTIONS HTTP method before sending a request in order to determine if the real request is safe to send. The response to this kind of request contains information about what is allowed from a request, and the server determines whether or not to send the actual request based on this information.

This response information is in the form of three HTTP headers, access-control-request-method, access-control-request-headers, and the origin header. These correspond to of the cors_config - thus these configurations can be specified to determine how the Grey Matter sidecar will respond to preflight requests. If a preflight request comes in to the Grey Matter Sidecar that does not meet the specification for one of these configured fields, the sidecar will send back a response to not initiate the request.

Based on the same , say the app running at http://localhost:8080 wants to send requests to the backend service at http://localhost:10808 with a content-type of application/json;charset=UTF-8. This particular content-type is outside of those allowed by CORS for simple requests, and thus would result in sending a preflight request to determine if the request can be sent. In order for the CORS configuration to indicate that the request can be sent, it would need to allow the content-type header by configuring the field:

In the above configuration, CORS will allow requests with Origin header value http://localhost:8080 only and indicate that the content-type header can be set according to the request.

Configuration

To set up CORS, set the cors_config field on a domain object with the desired configuration, see the below.

For an existing domain, run

and add the desired cors_config object.

Example object

Fields

allowed_origins

This field specifies an array of string patterns that match allowed origins. The proxy will use these matchers to set the header. This header will be set on any cross-origin response that matches one of the allowed_origins.

Available matchers include:

• exact

• prefix

• suffix

Example:

A wildcard value * is allowed except when using the regex matcher.

allow_credentials

Specifies the content for the header that the proxy will set on any cross-origin request that matches one of the allowed_origins. This header specifies whether or not the upstream service allows credentials.

exposed_headers

Specifies the content for the header that the proxy will set on any cross-origin request that matches one of the allowed_origins. This header specifies an array of headers that are allowed on the response.

max_age

Specifies the content for the header that the proxy will set on the preflight response. This header is an integer value specifying how long a preflight request can be cached by the browser.

allowed_methods

Specifies the content for the header that the proxy will set on the preflight response. This header specifies an array of methods allowed by the upstream service.

allowed_headers

Specifies the content for the header that the proxy will set on the preflight response. This header specifies an array of headers allowed by the upstream service.

Notes

• By default, the proxy will use the upstream service's CORS policy on the gateway and on the upstream service. The gateway policy is ignored.

• Because CORS is a browser construct, curl can always make a request to the server, with or without CORS. However, it can be used to mimic a browser and verify how the proxy will react to CORS requests:

Summary

Rules dictate logic on traffic is routed. Attributes from incoming requests are matched against preconfigured rule objects to determine where the outgoing upstream request should be routed. Rules are specified programmatically in routes and shared_rules as an array in the Rules attribute.

Example object

Fields

rule_key

A unique key for each rule. When a request is routed by a rule, it appends the header "X-Gm-Rule" with the rule key.

methods

The supported request methods for this rule. Setting to an empty array will allow all methods.

matches

for this rule.

constraints

The constraints field defines arrays that map requests to clusters. Currently, the only implemented field is the light field which is used to determine the Instance to which the live request will be sent and from which the response will be sent to the caller.

Currently, the constraints field must set a light field that contains an array of .

NOTE: constraints also contains a dark and a light field which currently have no effect. dark array will be used in future versions to support traffic shadowing to Instances. Similarly the tap array will determine an Instance to send a copy of the request to, comparing the response to the light response.

cohort_seed

This field has no effect.

Summary

A retry policy is a way for the Grey Matter Sidecar or Edge to automatically retry a failed request on behalf of the client. This is mostly transparent to the client; they will only get the status and return of the final request attempted (failed or succeeded). The only effects they should see from a successful retry is a longer average request time and fewer failures.

Example object

Fields

num_retries

This is the max number of retries attempted. Setting this field to N will cause up to N retries to be attempted before returning a result to the user.

Setting to 0 means only the original request will be sent and no retries are attempted. A value of 1 means the original request plus up to 1 retry will be sent, resulting in potentially 2 total requests to the server. A value of N will result in up to N+1 total requests going to the service.

per_try_timeout_msec

This is the timeout for each retry. The retry attempts can have longer or shorter timeouts than the original request. However, if the per_try_timeout_msec is too long, it is possible that not all retries will be attempted as it would violate the field.

timeout_msec

This is the total timeout for the entire chain: initial request + all timeouts. This should typically be set large enough to accommodate the request and all retries desired.

Welcome to Grey Matter documentation. This documentation covers all available features and options for Grey Matter, the intelligent hybrid mesh platform.