# Install on Kubernetes

This guide covers the necessary steps to install and configure the Grey Matter service mesh on a cloud based Kubernetes cluster. While these instructions should support all versions of Kubernetes, they have been tested and confirmed against versions `1.15`, `1.16`, `1.17`, `1.18`.

**Warning:** The Grey Matter mesh installed at the end of this guide is not intended for production use. Contact [Grey Matter Customer Support](mailto:info@greymatter.io) for more information on a Production deployment.

## Prerequisites

1. [helm v3](https://helm.sh/docs/intro/install/)
2. [eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
3. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
4. Grey Matter credentials requested via [Grey Matter Support](https://support.greymatter.io/support/home)

## Steps

### 1. Create Kubernetes Cluster

> NOTE: If you already have a Kubernetes cluster up and running, move to Step 2. Just verify you can connect to the cluster with a command like `kubectl get nodes`.

{% tabs %}
{% tab title="Amazon EKS" %}
For this deployment, we'll use [Amazon EKS](https://aws.amazon.com/eks/) to automatically provision a Kubernetes cluster for us. `eksctl` will use your preconfigured `default` AWS credentials to create master and worker nodes to our specifications, and configure `kubectl` so we can manipulate the cluster.

The regions, node type/size and other settings used below may need to be tuned to your use case. The minimum Kubernetes version supported by each platform slowly moves over time, so this may need to be updated periodically. Run, the following commands:

```bash
eksctl create cluster \
    --name production \
    --version 1.17 \
    --nodegroup-name workers \
    --node-type m5.2xlarge \
    --nodes=2 \
    --node-ami auto \
    --region us-east-1 \
    --zones us-east-1a,us-east-1b \
    --profile default
```

After 10 - 15 minutes, your cluster should be ready. You can test that your configuration is correct by running:

```bash
eksctl get cluster --region us-east-1 --profile default
eksctl get nodegroup --region us-east-1 --profile default --cluster production
```

{% endtab %}

{% tab title="Google GKE" %}
For this deployment, we'll use [Google GKE](https://cloud.google.com/kubernetes-engine) to automatically provision a Kubernetes cluster for us. `gcloud` will create master and worker nodes to our specifications, and configure `kubectl` so we can manipulate the cluster.

The regions, node type/size and other settings used below may need to be tuned to your use case. The minimum Kubernetes version supported by each platform slowly moves over time, so this may need to be updated periodically. Run, the following commands:

```bash
gcloud container clusters create production \
    --machine-type e2-standard-8 \
    --num-nodes 2 \
    --cluster-version 1.17 \
    --zone us-central1-a \
    --node-locations us-central1-a
```

After 3-5 minutes, your cluster should be ready. You can test that your configuration is correct by running:

```bash
gcloud container clusters list
```

{% endtab %}

{% tab title="Microsoft AKS" %}
For this deployment, we'll use [Microsoft AKS](https://docs.microsoft.com/en-us/azure/aks/) to automatically provision a Kubernetes cluster for us. `az` will create master and worker nodes to our specifications, and configure `kubectl` so we can manipulate the cluster.

The regions, node type/size and other settings used below may need to be tuned to your use case. The minimum Kubernetes version supported by each platform slowly moves over time, so this may need to be updated periodically. Run, the following commands:

```bash
az group create --name production --location eastus
az aks create --resource-group production \
  --name production \
  --node-vm-size standard_b8ms \
  --node-count 2 \
  --kubernetes-version 1.18.14 \
  --enable-addons monitoring \
  --generate-ssh-keys
```

Now we'll use `az` to retrieve our credentials and setup `kubectl` for access to the cluster.

```bash
az aks get-credentials --resource-group production --name production
```

After 3-5 minutes, your cluster should be ready. You can test that your configuration is correct by running:

```bash
az aks list
```

{% endtab %}
{% endtabs %}

### 2. Set up Credentials

The credentials identified in the prerequisite steps are used to create a Kubernetes Image Pull Secret that grants access to the Grey Matter Docker images in the [Grey Matter Nexus Repository](https://nexus.greymatter.io/).

If you do not have credentials yet, please [contact Grey Matter support](https://support.greymatter.io/support/home).

### 3. Get the Grey Matter Helm Charts

The Grey Matter Helm Charts are available from our [GitHub helm-charts repository](https://github.com/greymatter-io/helm-charts). Run the following command to add our helm charts repository.

```bash
helm repo add greymatter https://greymatter-io.github.io/helm-charts
helm repo update
```

### 4. Set up Secrets

Using your [credentials](#2-set-up-credentials), create the following file and save it as `credentials.yaml`.

```yaml
dockerCredentials:
  - registry: docker.greymatter.io
    email: <username>
    username: <username>
    password: <password>
```

### 5. Generate Configurations

Run the following to download the base configuration file:

```bash
wget https://raw.githubusercontent.com/greymatter-io/helm-charts/release-2.3/global.yaml
```

This file is where you can specify any custom configurations for the installation. The file downloaded as is will install Grey Matter with all default values. If you wish to modify the defaults, make changes to any existing values in `global.yaml`.

{% hint style="info" %}
Some configuration changes will change the installation process.

If you set `global.spire.enabled` to false, skip the `server` and `agent` release installations in [step 6](#6-install).

If you set `global.data.external.enabled` to true, you will need to add the data installation to part 3 of [step 6](#6-install), `helm install data greymatter/data -f global.yaml --set=global.environment=eks --set=global.waiter.service_account.create=false`.
{% endhint %}

To set more complex configurations like image versions, service environment variables, etc., check out the values files in each of the [Grey Matter helm charts](https://github.com/greymatter-io/helm-charts). Any additional configurations you wish to set can be added to the `global.yaml` file with **the same directory structure** as found in the `<chart>/values.yaml` file.

For example, to set the version of the Grey Matter proxy for the edge proxy, just simply replace the version as indicated below:

```yaml
edge:
  version: '<desired-version>'
```

### 6. Install

Grey Matter is made up of a handful of components, each handling different pieces of the overall platform. Once you have set up your `credentials.yaml` and `global.yaml` files, run the following steps in order:

1. Install the necessary secrets

   Grey Matter requires several Kubernetes secrets. The necessary secrets have been extracted into a single helm-chart for ease of installation.

   ```bash
   helm install secrets greymatter/secrets --version 2.4.2 -f credentials.yaml -f global.yaml
   ```
2. Install SPIRE server and agent

   This guide will use SPIRE to issue certificates to enable mTLS through the mesh. These commands will install the SPIRE server and agents into the `spire` namespace.

   ```bash
   helm install spire-server greymatter/server --version 2.3.2 -f global.yaml
   ```

   Before installing the agent, watch the server pod come up:

   ```bash
   kubectl get pods -n spire -w
   ```

   Wait until the server pod is 2/2:

   ```bash
      NAME          READY   STATUS    RESTARTS   AGE
      server-0      2/2     Running   1          3h54m
   ```

   Then, install the spire agent:

   ```bash
   helm install spire-agent greymatter/agent --version 2.3.2 -f global.yaml
   ```

   Verify SPIRE installation

   ```bash
   kubectl get pods -n spire -w
   ```

   > The SPIRE agent runs as a DaemonSet, so the number of Agent pods is directly related to how many nodes are in your cluster.

   ```bash
      NAME          READY   STATUS    RESTARTS   AGE
      agent-5d7q4   1/1     Running   0          3h54m
      agent-8c9lq   1/1     Running   0          3h54m
      agent-s2svz   1/1     Running   0          3h54m
      server-0      2/2     Running   1          3h54m
   ```
3. Install Grey Matter Charts

   At this point, we're ready to install Grey Matter. Grey Matter is installed through a series of Helm Charts that install the different components of Grey Matter. You'll notice we're also setting a few values on the command line. These could be updated in the `global.yaml` file, but we wanted to call them to your attention here.

   * `global.environment`: This is set to `eks` to drive EKS specific configurations
   * `edge.ingress.type`: This is set to `LoadBalancer` to update the Edge service to modify it to a type of LoadBalancer.
   * `global.waiter.service_account.create`: This is set to false to prevent the Sense helm-chart from attempting to create the `waiter` service account.

   ```bash
   helm install fabric greymatter/fabric --version 3.0.16 -f global.yaml --set=global.environment=eks
   helm install edge greymatter/edge --version 3.0.8 -f global.yaml --set=global.environment=eks --set=edge.ingress.type=LoadBalancer
   helm install sense greymatter/sense --version 3.0.11 -f global.yaml --set=global.environment=eks --set=global.waiter.service_account.create=false
   ```

   Notice in the edge installation we are setting `--set=edge.ingress.type=LoadBalancer`, this value sets the service type for edge. The default is `ClusterIP`. In this example we want an AWS ELB to be created automatically for edge ingress (see [below](#5-accessing-the-dashboard)), thus we are setting it to `LoadBalancer`. See the Kubernetes [publishing services docs](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types) for guidance on what this value should be in your specific installation.

   If you receive `Error: could not find tiller` after running the above helm commands, then you're running an older version of Helm and must install Helm v3. If you need to manage multiple versions of Helm, we highly recommend using [helmenv](https://github.com/yuya-takeyama/helmenv) to easily switch between versions.

   While these are being installed, you can use the `kubectl` command to check if everything is running. When all pods are `Running` or `Completed`, the install is finished and Grey Matter is ready to go.

   ```bash
   kubectl get pods
   ```

   The running output will look like the following:

   ```bash
   NAME                           READY   STATUS      RESTARTS   AGE
   catalog-75d4c66477-h9nnr       2/2     Running     0          49s
   catalog-init-v778j             0/1     Completed   0          49s
   control-78f8ccf5f6-krlbd       1/1     Running     0          111s
   control-api-0                  2/2     Running     0          111s
   control-api-init-pblsb         0/1     Completed   0          111s
   dashboard-dd95bddbc-hrzm5      2/2     Running     0          49s
   edge-d56fd6795-qdctl           1/1     Running     0          63s
   jwt-redis-6b84846ffc-fkpjg     1/1     Running     0          111s
   jwt-security-7bc8bfb9f-jbz6l   2/2     Running     0          111s
   mesh-redis-0                   1/1     Running     0          111s
   postgres-slo-0                 1/1     Running     0          49s
   prometheus-0                   2/2     Running     0          49s
   slo-d98698db9-p7ksz            2/2     Running     0          49s
   ```

### 7. Accessing the Application

1. Get the User Certificate

   By default, Grey Matter leverages mutual TLS (mTLS) communications for all traffic, including inbound traffic to the mesh. This means that all `https` requests must include TLS certificates whether that be via a web browser or RESTful client. The Grey Matter helm charts have the ability to generate random Ingress and User certificates to ensure unique certificates every time a cluster is launched. For web based authentication, these certificates can then be imported into a web browser, to access resources in the mesh.

   Following the instructions in this guide, Grey Matter will automatically provision the required mTLS certificates for the server and the user.

   To get the user certificate, run these commands:

   ```
   kubectl get secret greymatter-user-cert -o jsonpath="{.data['tls\.crt']}" | base64 -d > tls.crt
   kubectl get secret greymatter-user-cert -o jsonpath="{.data['tls\.key']}" | base64 -d > tls.key
   kubectl get secret greymatter-user-cert -o jsonpath="{.data['ca\.crt']}" | base64 -d > ca.crt
   ```

   Then create a new p12 certificate and load it into your browser:

   ```
   openssl pkcs12 -export -out greymatter.p12 -inkey tls.key -in tls.crt -certfile ca.crt -passout pass:password
   ```

   > If you want to provide your own valid certificates for ingress, set `.Values.global.auto_generate_edge_certs` to `false` and provide the cert information in the secrets chart, at `.Values.edge.certificate.ingress`. You will need to ensure you have a valid User certificate from the same Certificate Authority for Grey Matter to authenticate the user.
2. Access Grey Matter

   An [Amazon ELB](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html) will be created automatically when we specified the flags `--set=global.environment=eks` and `--set=edge.ingress.type=LoadBalancer` during installation. The ELB is accessible through the randomly created URL attached to the edge service:

   ```bash
   kubectl get svc edge
   ```

   The output will look like the following:

   ```bash
   NAME   TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                          AGE
   edge   LoadBalancer   10.100.252.235   a255c23a43350427a93a860856d52155-1106205970.us-east-1.elb.amazonaws.com   10808:31098/TCP,8081:31216/TCP   4m
   ```

   Visit the url listed under `EXTERNAL-IP` *(e.g. `https://a255c23a43350427a93a860856d52155-1106205970.us-east-1.elb.amazonaws.com:10808/`)* in the browser to access the Grey Matter application:

   ![Grey Matter application](https://1676458320-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LsNFVozLgvw3NDMzxBg%2Fsync%2Fc4274928954ca6d13900eccb146e37991d88a5fe.png?generation=1627486481724580\&alt=media)

#### Configure the Grey Matter CLI

In order to add or modify service configurations, make sure you have the [greymatter CLI](https://greymatter.gitbook.io/grey-matter-documentation/1.3/installation/commands-cli) installed.

To configure it for this installation, replace `{edge-amazon-elb-dns}` with the edge service external IP from [step 5](#5-accessing-the-dashboard), and replace \<path/to/greymatter/helm-charts> with the path on your machine to the Grey Matter Helm Charts directory in the block below:

```bash
export EDITOR=vim
export GREYMATTER_API_HOST={edge-amazon-elb-dns}:10808
export GREYMATTER_API_INSECURE=true
export GREYMATTER_API_PREFIX=/services/control-api/latest
export GREYMATTER_API_SSL=true
export GREYMATTER_API_SSLCERT=<path/to/greymatter/helm-charts>/certs/quickstart.crt
export GREYMATTER_API_SSLKEY=<path/to/greymatter/helm-charts>/certs/quickstart.key
export GREYMATTER_CATALOG_HOST={edge-amazon-elb-dns}:10808
export GREYMATTER_CATALOG_PREFIX=/services/catalog/latest
export GREYMATTER_CATALOG_INSECURE=true
export GREYMATTER_CATALOG_SSL=true
export GREYMATTER_CATALOG_SSLCERT=<path/to/greymatter/helm-charts>/certs/quickstart.crt
export GREYMATTER_CATALOG_SSLKEY=<path/to/greymatter/helm-charts>/certs/quickstart.key
export GREYMATTER_CATALOG_ZONENAME=zone-default-zone
export GREYMATTER_CONSOLE_LEVEL=debug
```

Now if you can run `greymatter list cluster` and `greymatter list catalog_cluster` and there are no errors, the CLI is properly configured.

### Cleanup

#### Delete the Grey Matter Installation

```bash
helm uninstall sense edge fabric spire-agent spire-server secrets
```

#### Delete The Kubernetes Cluster

> NOTE: this deletion actually takes longer than the output would indicate to terminate all resources. Attempting to create a new cluster with the same name will fail for some time until all resources are purged.

{% tabs %}
{% tab title="Amazon EKS" %}

```bash
eksctl delete cluster --name production
```

```bash
[ℹ]  using region us-east-1
[ℹ]  deleting EKS cluster "production"
[✔]  kubeconfig has been updated
[ℹ]  cleaning up LoadBalancer services
[ℹ]  2 sequential tasks: { delete nodegroup "workers", delete cluster control plane "prod" [async] }
[ℹ]  will delete stack "eksctl-production-nodegroup-workers"
[ℹ]  waiting for stack "eksctl-production-nodegroup-workers" to get deleted
[ℹ]  will delete stack "eksctl-production-cluster"
[✔]  all cluster resources were deleted
```

{% endtab %}

{% tab title="Google GKE" %}

```bash
gcloud container clusters delete production --quiet
```

```bash
Deleting cluster production...done.
Deleted [https://container.googleapis.com/v1/projects/psychic-era-307017/zones/us-central1-a/clusters/production].
```

{% endtab %}

{% tab title="Microsoft AKS" %}

```bash
az aks delete --resource-group production --name production -y
```

{% endtab %}
{% endtabs %}