Deploy Service for Ingress/Egress Actions to Grey Matter on Kubernetes

This is an example of deploying a new service into the mesh and configuring it for ingress and egress actions using SPIFFE/SPIRE identities.

Prerequisites

1. An existing Grey Matter deployment following the Grey Matter Quickstart Install with Helm/Kubernetes guide.

   1. This guide assumes you've followed the step to configure a load balancer for ingress access to the mesh. If so, you should have a URL to access the Grey Matter Intelligence 360 application (e.g. a2832d300724811eaac960a7ca83e992-749721369.us-east-1.elb.amazonaws.com:10808).

   2. Make note of your load balancer ingress URL and use it wherever this guide references {your-gm-ingress-url}.

2. greymatter setup with a running Fabric mesh.

Overview

1. Generate k8s deployment with the service and sidecar

2. Configure the sidecar for ingress actions

3. Configure the sidecar for egress actions

Steps

1. Create Deployment

The service we will be using for this walkthrough is a simple egress/ingress service with two endpoints. The /ingress endpoint will return Ingress request to simple-service successful!. The /egress endpoint takes an environment variable EGRESS_ROUTE, generates a request to that route, and returns its response.

There are a few things to note before generating the deployment file:

1. The value of the pod label that Grey Matter Control is discovering, environment variable GM_CONTROL_KUBERNETES_CLUSTER_LABEL

2. The value of the kubernetes port name that Grey Matter Control is discovering, environment variable GM_CONTROL_KUBERNETES_PORT_NAME

3. The trust_domain of the SPIRE server

4. The SPIRE server registrar service configuration

Using the Grey Matter Helm Charts, the values of the first and second points are greymatter.io/control and proxy by default. The deployment will then need that pod label greymatter.io/control with a value identifying the service to Grey Matter Control, and its sidecar container will need a port labeled proxy.

The SPIRE trust domain, by default, is quickstart.greymatter.io. This will be important for generating the mesh configurations. The SPIRE registrar service, by default, is configured to register entries with the SPIRE server for every pod that is created with the same label that Grey Matter Control is looking for, greymatter.io/control. It will generate SPIFFE identities for these pods in the form spiffe://<trust_domain>/<greymatter.io/control-value>. For example, for a pod in the default setup with label greymatter.io/control: control-api, the registrar will generate an entry with SPIFFE ID spiffe://quickstart.greymatter.io/control-api.

Note: If the registrar service in your setup is configured without the pod_label specification, it will generate entries with the SPIRE server in a different form, which will determine both the way that the mesh is configured and the deployment. The entries in this setup will be of the form spiffe://<trust_domain>/ns/<namespace>/sa/<service_account>, and thus the secrets in the mesh configurations will need to reflect this format instead.

Based on the defaults described, the deployment for the simple-service will look like the following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: simple-service
    greymatter.io/control: simple-service
  name: simple-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: simple-service
      greymatter.io/control: simple-service
  template:
    metadata:
      labels:
        app: simple-service
        greymatter.io/control: simple-service
    spec:
      containers:
        - name: service
          image: "zoemccormick/simple-service:latest"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: EGRESS_ROUTE
              value: http://localhost:10909/catalog/summary
        - name: sidecar
          image: "docker.greymatter.io/release/gm-proxy:1.4.5"
          imagePullPolicy: IfNotPresent
          ports:
            - name: metrics
              containerPort: 8081
            - name: proxy
              containerPort: 10808
          env:
            - name: ENVOY_ADMIN_LOG_PATH
              value: "/dev/stdout"
            - name: PROXY_DYNAMIC
              value: "true"
            - name: SPIRE_PATH
              value: "/run/spire/socket/agent.sock"
            - name: XDS_CLUSTER
              value: "simple-service"
            - name: XDS_HOST
              value: "control.default.svc"
            - name: XDS_NODE_ID
              value: "default"
            - name: XDS_PORT
              value: "50000"
            - name: XDS_ZONE
              value: "zone-default-zone"
          volumeMounts:
            - name: spire-socket
              mountPath: /run/spire/socket
              readOnly: false
      imagePullSecrets:
        - name: docker.greymatter.io
      volumes:
        - name: spire-socket
          hostPath:
            path: /run/spire/socket
            type: DirectoryOrCreate

Note: we set the environment for the service container EGRESS_ROUTE to https://localhost:10909/catalog/summary. This will be important when configuring the egress actions

Save the deployment as deployment.yaml and apply the deployment:

kubectl apply -f deployment.yaml

Run kubectl get pods -l=greymatter.io/control=simple-service and make sure that the containers are running 2/2 before moving on the the mesh configuration.

2. Configure the deployment for ingress actions

Now, we will generate the mesh configurations in order to route from edge to the sidecar of the deployment and from the sidecar to the simple-service itself. At the end of this step, the /ingress endpoint of the simple service should be accessible through the edge and properly return Ingress request to simple-service successful!.

Domain

The first object to generate is the ingress domain of the service.

{
  "zone_key": "zone-default-zone",
  "domain_key": "simple-service-domain",
  "name": "*",
  "port": 10808,
  "force_https": true
}

Save this file as ingress-domain.json and apply the domain object using the greymatter cli:

greymatter create domain < ingress-domain.json

Listener

Next, create the corresponding ingress listener of the service.

{
  "zone_key": "zone-default-zone",
  "listener_key": "simple-service-listener",
  "domain_keys": [
    "simple-service-domain"
  ],
  "name": "ingress",
  "ip": "0.0.0.0",
  "port": 10808,
  "protocol": "http_auto",
  "secret": {
    "secret_key": "simple-service.identity",
    "secret_name": "spiffe://quickstart.greymatter.io/simple-service",
    "secret_validation_name": "spiffe://quickstart.greymatter.io",
    "subject_names": [
      "spiffe://quickstart.greymatter.io/edge"
    ],
    "ecdh_curves": [
      "X25519:P-256:P-521:P-384"
    ]
  }
}

The combination of the ingress domain and listener opens 0.0.0.0:10808 to accept connections. The secret set on the listener specifies the SPIFFE identity for the simple-service pod, spiffe://quickstart.greymatter.io/simple-service. The sidecar will fetch the certificate for this identity via Envoy SDS. The subject_names field of the secret specifies that only incoming connections with certificate SAN equal to spiffe://quickstart.greymatter.io/edge will be accepted by the listener.

Save this file as ingress-listener.json and apply it:

greymatter create listener < ingress-listener.json

Proxy

The proxy object will indicate that the above domain and listener are meant to configure this specific sidecar. In order to link these objects to the deployment, the name field of this proxy object must equal the value of the deployment label greymatter.io/control, which in this case is simple-service.

{
  "zone_key": "zone-default-zone",
  "proxy_key": "simple-service-proxy",
  "domain_keys": [
    "simple-service-domain"
  ],
  "listener_keys": [
    "simple-service-listener"
  ],
  "name": "simple-service",
  "listeners": null
}

Note: the domain_keys and listener_keys fields take a list of strings. When we add a second domain and listener in egress configurations, we will add their keys to this proxy.

Save this file as proxy.json and apply it:

greymatter create proxy < proxy.json

Clusters

There are two clusters necessary for ingress connections. The first is the edge-to-simple-service-cluster. This cluster will be used to locate and route to the simple-service sidecar from edge. In opposition to the listener secret above, the secret on the edge-to-simple-service-cluster specifies then that when connecting to the sidecar, use the spiffe certificate with identity spiffe://quickstart.greymatter.io/edge and only connect to certificate with SAN equal to spiffe://quickstart.greymatter.io/simple-service.

The instances field is left empty because Grey Matter Control will discover the sidecar with label greymatter.io/control: simple-service, and add the instances to this cluster because the name field matches. Thus, like the proxy object, it is important to note that the name field must equal the greymatter.io/control label.

{
  "zone_key": "zone-default-zone",
  "cluster_key": "edge-to-simple-service-cluster",
  "name": "simple-service",
  "instances": [],
  "require_tls": true,
  "secret": {
    "secret_key": "edge.identity",
    "secret_name": "spiffe://quickstart.greymatter.io/edge",
    "secret_validation_name": "spiffe://quickstart.greymatter.io",
    "subject_names": [
      "spiffe://quickstart.greymatter.io/simple-service"
    ],
    "ecdh_curves": [
      "X25519:P-256:P-521:P-384"
    ]
  }
}

The second cluster connects the sidecar to the simple-service, which is running on port 8080 as specified in the deployment. Because the two containers share a pod, this connection is plaintext over localhost and needs no secret. Because we know where the service is running in connection to the sidecar, localhost:8080, we can hardcode the instance.

{
  "zone_key": "zone-default-zone",
  "cluster_key": "simple-service-cluster",
  "name": "service",
  "instances": [
    {
      "host": "localhost",
      "port": 8080
    }
  ],
  "require_tls": false
}

Save these files as edge-to-simple-service-cluster.json and simple-service-cluster.json, respectively, and apply them:

greymatter create cluster < edge-to-simple-service-cluster.json
greymatter create cluster < simple-service-cluster.json

Shared Rules

The two clusters created above, edge-to-simple-service-cluster and simple-service-cluster need two shared_rules objects to allow routes to connect to them.

{
  "zone_key": "zone-default-zone",
  "shared_rules_key": "edge-to-simple-service-rules",
  "name": "edge-to-simple-service",
  "default": {
    "light": [
      {
        "constraint_key": "",
        "cluster_key": "edge-to-simple-service-cluster",
        "metadata": null,
        "properties": null,
        "response_data": {},
        "weight": 1
      }
    ],
    "dark": null,
    "tap": null
  }
}

{
  "zone_key": "zone-default-zone",
  "shared_rules_key": "simple-service-rules",
  "name": "service",
  "default": {
    "light": [
      {
        "constraint_key": "",
        "cluster_key": "simple-service-cluster",
        "metadata": null,
        "properties": null,
        "response_data": {},
        "weight": 1
      }
    ],
    "dark": null,
    "tap": null
  }
}

When specifying routes from edge to the simple-service sidecar, we'll use edge-to-simple-service-rules, and when specifying routes from the simple-service sidecar to the service, we'll use simple-service-rules.

Save these files as edge-to-simple-service-rules.json and simple-service-rules.json, respectively, and apply them:

greymatter create cluster < edge-to-simple-service-rules.json
greymatter create cluster < simple-service-rules.json

Routes

The last step in the ingress configuration is routes. There will be three routes for ingress to the service.

The first two specify when and how edge should connect to the simple-service sidecar. These two routes will be configured on the edge sidecar, and when a request comes in with path prefix equal to /services/simple-service/ or /services/simple-service, it will strip off that value from the path and forward the request to the cluster specified in the shared_rules object that it links to.

{
  "zone_key": "zone-default-zone",
  "domain_key": "edge",
  "route_key": "edge-to-simple-service-route",
  "path": "/services/simple-service/",
  "prefix_rewrite": "/",
  "shared_rules_key": "edge-to-simple-service-rules"
}

{
  "zone_key": "zone-default-zone",
  "domain_key": "edge",
  "route_key": "edge-to-simple-service-route-slash",
  "path": "/services/simple-service",
  "prefix_rewrite": "/services/simple-service/",
  "shared_rules_key": "edge-to-simple-service-rules"
}

The last route configures the simple-service sidecar to route any incoming requests with path prefix "/" to the simple-service.

{
  "zone_key": "zone-default-zone",
  "domain_key": "simple-service-domain",
  "route_key": "simple-service-route",
  "path": "/",
  "prefix_rewrite": "",
  "shared_rules_key": "simple-service-rules"
}

Save these files as edge-route.json, edge-route-slash.json, and service-route.json and apply them:

greymatter create route < edge-route.json
greymatter create route < edge-route-slash.json
greymatter create route < service-route.json

Once all of the configurations are applied, you should be able to access the simple-service at https://{your-gm-ingress-url}/services/simple-service/ and the ingress route at https://{your-gm-ingress-url}/services/simple-service/ingress. Note that if you try to hit https://{your-gm-ingress-url}/services/simple-service/egress, the request will fail with upstream connect error or disconnect/reset before headers. reset reason: connection termination. This is because we have not yet configured the service for egress actions.

3. Configure the deployment for egress actions

For any service being deployed in place of simple-service that generates requests to other services in the mesh, there is another set of configurations necessary.

For this walkthrough, we have configured the simple-service with environment variable EGRESS_ROUTE=http://localhost:10909/catalog/summary. The example that this will show is the simple-service /egress endpoint generating a request to the Grey Matter Catalog service, which is also running inside the mesh, and returning the json response from a GET request of its /summary endpoint.

SPIFFE/SPIRE and egress explanation

In order to allow the simple-service to make requests inside the mesh, we need to open up a second domain/listener combo on the sidecar, this time on a different port than the ingress at 10808. We will use 10909. In a non-SPIFFE/SPIRE setup, this second domain/listener is not necessary.

This is necessary for a SPIFFE/SPIRE setup because all of the sidecars are using SPIFFE certificates, fetched by the sidecar using Envoy SDS and not through a physical mount into the service or sidecar containers. Thus, the containers themselves can not have and do not need certificates mounted into them. Because of this, any request generated from within the service (service-A) can only be http. But, all incoming requests to a service (say service-B) within the mesh must go through it's sidecar, and the sidecar is only accepting https requests with specific SPIFFE certificates.

The services service-A and service-B can talk plaintext to their own sidecars. All sidecars have access to their SPIFFE identities via Envoy SDS, so two sidecars can communicate with each other over https. Thus, the flow of a request generated by service-A to service-B looks like the following:

Simple Egress Flow

This allows us to break down the need for the second, "egress" domain/listener combo on port 10909, and how the EGRESS_ROUTE is formed. With the sidecar for service-A (or in this example the simple-service sidecar), listening for plaintext connections over localhost at 10909, the simple-service itself can direct requests to its own sidecar at http://localhost:10909. The sidecar, then, can use path_matching on a configured egress route to forward specific requests to an internal, or egress, cluster. As described below, this egress cluster will be configured with the correct SPIFFE identity for service-A, and the request will be sent to the ingress domain/listener for service-B at 10808.

For this example, where simple-service wants to make a request to the Grey Matter Catalog service for its /summary endpoint, the flow will look like:

Example Egress Flow

Egress Domain

The egress domain object will have force_https set to false, as we want the sidecar to accept plaintext connections on 10909. It will add the custom header x-forwarded-proto: https to requests coming from the domain.

{
  "zone_key": "zone-default-zone",
  "domain_key": "simple-service-domain-egress",
  "name": "*",
  "port": 10909,
  "force_https": false,
  "custom_headers": [
    {
      "key": "x-forwarded-proto",
      "value": "https"
    }
  ]
}

Save this file as egress-domain.json and apply it:

greymatter create domain < egress-domain.json

Egress Listener

{
  "zone_key": "zone-default-zone",
  "listener_key": "simple-service-listener-egress",
  "domain_keys": [
    "simple-service-domain-egress"
  ],
  "name": "egress",
  "ip": "0.0.0.0",
  "port": 10909,
  "protocol": "http_auto"
}

Save this file as egress-listener.json and apply it:

greymatter create listener < egress-listener.json

Egress Cluster

The egress cluster must have name equal to the greymatter.io/control pod label for the service it is trying to reach in order to Grey Matter Control to give it the instance for the service. In our case, this is the Grey Matter Catalog service, which has greymatter.io/control label "catalog".

The secret on this cluster will use the SPIFFE certificate with id spiffe://quickstart.greymatter.io/simple-service and connect only over connections with SAN spiffe://quickstart.greymatter.io/catalog.

{
  "zone_key": "zone-default-zone",
  "cluster_key": "simple-service-to-catalog-cluster",
  "name": "catalog",
  "instances": [],
  "require_tls": true,
  "secret": {
    "secret_key": "simple-service.identity",
    "secret_name": "spiffe://quickstart.greymatter.io/simple-service",
    "secret_validation_name": "spiffe://quickstart.greymatter.io",
    "subject_names": [
      "spiffe://quickstart.greymatter.io/catalog"
    ],
    "ecdh_curves": [
      "X25519:P-256:P-521:P-384"
    ]
  }
}

Save this file as simple-service-to-catalog-cluster.json and apply it:

greymatter create cluster < simple-service-to-catalog-cluster.json

Egress Shared Rules

To configure the egress route to send requests to the simple-service-to-catalog-cluster, we'll generate a shared_rules.

{
  "zone_key": "zone-default-zone",
  "shared_rules_key": "simple-service-to-catalog-rules",
  "name": "simple-service-to-catalog",
  "default": {
    "light": [
      {
        "constraint_key": "",
        "cluster_key": "simple-service-to-catalog-cluster",
        "metadata": null,
        "properties": null,
        "response_data": {},
        "weight": 1
      }
    ],
    "dark": null,
    "tap": null
  }
}

Save this file as simple-service-to-catalog-rules.json and apply it:

greymatter create shared_rules < simple-service-to-catalog-rules.json

Egress Route

For this example, the egress route we want is to the Grey Matter Catalog service. The way we have chosen to specify this is using path prefix /catalog on the request. This route will take requests into the simple-service-domain-egress with path prefix math /catalog, strip the prefix and forward the request through the cluster specified in shared_rules object simple-service-to-catalog-rules.

{
  "zone_key": "zone-default-zone",
  "domain_key": "simple-service-domain-egress",
  "route_key": "simple-service-to-catalog-egress-route",
  "path": "/catalog/",
  "prefix_rewrite": "/",
  "shared_rules_key": "simple-service-to-catalog-rules"
}

Save this file as simple-service-to-catalog-route.json and apply it:

greymatter create route < simple-service-to-catalog-route.json

Now there are several updates to existing Grey Matter objects that need to be made. First, we know that the simple-service-to-catalog-cluster will send requests to the Catalog service using SPIFFE certificate with id (or SAN) spiffe://quickstart.greymatter.io/simple-service. The Catalog service ingress listener secret will need to have this id added to its subject_names in order to accept this request.

greymatter edit listener listener-catalog

Locate the secret field. The subject_names should contain only the edge identity, spiffe://quickstart.greymatter.io/edge. Change the subject names field to also allow our simple-service identity,

  "subject_names": [
    "spiffe://quickstart.greymatter.io/edge",
    "spiffe://quickstart.greymatter.io/simple-service"
  ],

Lastly, we need to tell the proxy object for the simple-service to add the egress domain and listener to the service configuration.

greymatter edit proxy simple-service-proxy

and add simple-service-domain-egress to the domain_keys and simple-service-listener-egress to the listener_keys. It should look like:

{
  "zone_key": "zone-default-zone",
  "proxy_key": "simple-service-proxy",
  "domain_keys": [
    "simple-service-domain",
    "simple-service-domain-egress"
  ],
  "listener_keys": [
    "simple-service-listener",
    "simple-service-listener-egress"
  ],
  "name": "simple-service",
  "listeners": null
}

Now, the simple-service should be configured to successfully make requests to the Grey Matter Catalog service. Go to https://{your-gm-ingress-url}/services/simple-service/egress to test this out. The egress request should return a string of json, and if you hit the catalog /summary endpoint directly through edge, https://{your-gm-ingress-url}/services/catalog/latest/summary, you should see the same result.

For any internal services that your service needs to be able to connect to in the above way will need their own service-a-to-service-b cluster, route, and shared_rules like the egress ones above. They will connect to the egress domain.