# 1.3 Migration Guide This guide covers the necessary steps to upgrade an existing Grey Matter 1.2 installation running on Kubernetes to Grey Matter 1.3. The procedure detailed below was run against an AWS EKS cluster but should work for other Kubernetes-based systems. {% hint style="info" %} The Grey Matter mesh shown in this guide is for demonstration/proof of concept purposes only and is not intended for production use. Contact [Grey Matter Customer Support](mailto:info@greymatter.io) for more information on a production deployment. While these instructions should work against all versions of AWS EKS, they have been specifically tested and confirmed against versions: `1.15`, `1.16`, `1.17`, `1.18`. {% endhint %} ## Prerequisites 1. [helm v3](https://helm.sh/docs/intro/install/) 2. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 3. Grey Matter credentials requested via [Grey Matter Support](https://support.greymatter.io/support/home) ## Steps ### 1. Confirm Grey Matter mesh is running ```bash kubectl get pods ``` The output should look similar to below. Check to make sure that all the pods are in `Running` or `Completed` state. You may see more pods than listed below depending on if there are other deployed services. ```bash NAME READY STATUS RESTARTS AGE catalog-6c768b578f-t2vmv 2/2 Running 0 4m47s catalog-init-xgxgd 0/1 Completed 3 4m14s control-7849f77964-gszdt 1/1 Running 0 6m30s control-api-0 2/2 Running 0 6m30s control-api-init-cjxrx 0/1 Completed 0 6m29s dashboard-b8bff9c69-rzp2z 2/2 Running 0 4m47s data-0 2/2 Running 0 5m11s data-internal-0 2/2 Running 0 5m11s data-mongo-0 1/1 Running 0 5m11s edge-fd9fc77c5-6zjr2 1/1 Running 0 5m35s internal-data-mongo-0 1/1 Running 0 5m11s internal-jwt-security-7f4bbdd995-f2czx 2/2 Running 0 6m30s internal-redis-cc7f64cd-p2gl7 1/1 Running 0 6m30s jwt-security-757d67f477-5tqxb 2/2 Running 2 6m30s postgres-slo-0 1/1 Running 0 4m47s prometheus-0 2/2 Running 0 4m47s redis-7957ffffd4-qshxh 1/1 Running 0 6m30s slo-5fbd879488-87jcl 2/2 Running 0 4m47s ``` ### 2. Download resources and set deployment type * Download the migration scripts from the [Grey Matter Nexus Repository](https://nexus.greymatter.io/repository/raw/release/upgrades/gm-1.3-upgrade.tar.gz). * Clone the [Grey Matter helm-charts](https://github.com/greymatter-io/helm-charts) GitHub repo and set the branch to `release-2.3`. * Download and install the [Grey Matter CLI](https://greymatter.gitbook.io/grey-matter-documentation/1.3/installation/commands-cli). ```bash git clone git@github.com:greymatter-io/helm-charts.git cd helm-charts && git switch release-2.3 ``` * Set the environment type. In the current helm-charts directory, edit `global.yaml` and change the following setting according to the installed environment type (the options are `eks`, `kubernetes`, or `openshift`). In this example, we will be using `eks`. ```yaml global: environment: eks ``` ### 3. Set up environment variables Untar the migration scripts that you downloaded in the previous step. ```bash tar -xvf gm-1.3-upgrade.tar.gz cd gm-1.3-upgrade ``` Source-in the environment script to set the appropriate environment variables. The script will attempt to determine the host:port of the mesh load balancer. If it does not find it, you'll be prompted to enter the value manually. In that scenario, use an existing hostname and port or external IP and port. ```bash source ./environment ``` {% hint style="info" %} This script is targeted for POSIX shells like Bash. If running in Zsh, you'll need to emulate a POSIX shell. If subsequent commands timeout or you receive errors, double check the values of the environment variables by running `env | grep GREYMATTER_`. You can override any of them manually to suit your needs. {% endhint %} ### 4. Get Grey Matter mesh configs Switch to the mesh directory and run the `getmesh.sh` script. ```bash cd mesh && ./getmesh.sh ``` The `getmesh.sh` script should have created a bunch of directories containing the mesh configs. Run the following commands to check the generated files for content. You're just checking to make sure the configs are actually captured and the files don't contain errors or some other output. ```bash cat cluster/cluster.json cat domain/domain.json cat listener/listener.json cat proxy/proxy.json cat route/route.json cat shared_rules/shared_rules.json cat zone/zone.json ``` ### 5. Get Grey Matter Catalog configs Switch to the `catalog` directory and run the `getcatalog.sh` script. ```bash cd ../catalog && ./getcatalog.sh ``` The command should display output similar to the below indicating a successful connection to the Catalog service. Note that some paths/URLs will be different. ```bash --> GREYMATTER_CONSOLE_LEVEL: debug --> GREYMATTER_API_HOST: a6a0ea8ea86e343e2825e59383797d12-1005777820.us-east-1.elb.amazonaws.com:10808 --> GREYMATTER_API_SSLCERT: /Users/mike/Code/gm-upgrade-1.3/certs/quickstart.crt --> GREYMATTER_API_SSLKEY: /Users/mike/Code/gm-upgrade-1.3/certs/quickstart.key --> GREYMATTER_API_SSL: true --> GREYMATTER_API_INSECURE: true * * * Connection was made to Grey Matter Control API * * * --> GREYMATTER_CONSOLE_LEVEL: debug --> GREYMATTER_API_HOST: a6a0ea8ea86e343e2825e59383797d12-1005777820.us-east-1.elb.amazonaws.com:10808 --> GREYMATTER_API_SSLCERT: /Users/mike/Code/gm-upgrade-1.3/certs/quickstart.crt --> GREYMATTER_API_SSLKEY: /Users/mike/Code/gm-upgrade-1.3/certs/quickstart.key --> GREYMATTER_API_SSL: true --> GREYMATTER_API_INSECURE: true * * * Connection was made to Catalog Api API * * * ``` The `getcatalog.sh` script should have created 2 directories containing the Catalog configs. Run the following commands to check the generated files for content. You're just checking to make sure the configs are actually captured and the files don't contain errors or some other output. ```bash cat clusters/clusters.json cat zones/zones.json ``` ### 6. Upgrade Grey Matter mesh configs Switch to the `fabric` directory in the `helm-charts` repo and run the Grey Matter fabric upgrade. ```bash cd ../../helm-charts/fabric && make upgrade-fabric ``` You should get output similar to the below indicating that the Helm chart was applied successfully. ```bash Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "greymatter" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 4 charts Deleting outdated charts [mike@orion:~/Code/helm-charts/fabric]$ make upgrade-fabric rm -f ./charts/* echo "target hit package-fabric" target hit package-fabric helm dep up . Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "greymatter" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 4 charts Deleting outdated charts helm upgrade fabric . --set=global.waiter.service_account.create=false -f ../global.yaml --no-hooks coalesce.go:199: warning: destination for resources is a table. Ignoring non-table value coalesce.go:199: warning: destination for resources is a table. Ignoring non-table value coalesce.go:199: warning: destination for resources is a table. Ignoring non-table value Release "fabric" has been upgraded. Happy Helming! NAME: fabric LAST DEPLOYED: Tue Nov 24 15:10:57 2020 NAMESPACE: default STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: Grey Matter 3.0.0 has been installed. fabric deployed to namespace "default" at 03:10:57 on 11/24/03 NOTE: It may take a few minutes for the installation to become stable. You can watch the status of the pods by running 'kubectl get pods -w -n default' ``` Run this command to check the status of the Kubernetes pods. ```bash kubectl get pod ``` Confirm all of the pods are in running/completed state before continuing, similar to what is shown below. You may need to rerun the previous command a few times or add the `-w` flag to the previous command until everything stabilizes. ```bash NAME READY STATUS RESTARTS AGE catalog-6c768b578f-t2vmv 2/2 Running 0 22h catalog-init-xgxgd 0/1 Completed 3 22h control-6db59845fd-d4tx9 1/1 Running 0 2m5s control-api-0 2/2 Running 0 92s control-api-init-cjxrx 0/1 Completed 0 22h dashboard-b8bff9c69-rzp2z 2/2 Running 0 22h data-0 2/2 Running 0 22h data-internal-0 2/2 Running 0 22h data-mongo-0 1/1 Running 0 22h edge-fd9fc77c5-6zjr2 1/1 Running 0 22h internal-data-mongo-0 1/1 Running 0 22h jwt-redis-db87bcc4c-v6dtw 1/1 Running 0 2m5s jwt-security-5c96d95749-2htgs 2/2 Running 0 2m5s mesh-redis-0 1/1 Running 0 2m5s postgres-slo-0 1/1 Running 0 22h prometheus-0 2/2 Running 0 22h slo-5fbd879488-87jcl 2/2 Running 0 22h ``` ### 7. Re-apply the mesh configs After the Grey Matter fabric portion of the upgrade, the mesh configs that were capture in step 4 need to be reapplied. ```bash cd ../../gm-1.3-upgrade cd mesh && ./populate.sh ``` You should see a bunch of configs getting applied and their config outputted in series (it will take a minute or two to complete). There should be no errors if everything is working and connected properly. ### 8. Upgrade Grey Matter Edge At this point, you will not be able to access the dashboard until the Grey Matter `edge` is updated, so we will do that now. ```bash cd ../../helm-charts/edge && helm upgrade edge . -f ../global.yaml --set edge.ingress.type=LoadBalancer ``` You should receive the below output indicating that the Helm chart was successfully applied. ```bash Release "edge" has been upgraded. Happy Helming! NAME: edge LAST DEPLOYED: Tue Nov 24 15:31:22 2020 NAMESPACE: default STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: Grey Matter Edge 1.5.1 has been installed. NOTE: It may take a few minutes for the installation to become stable. You can watch the status of the pods by running 'kubectl get pods -w -n default' ``` Run this command to check the status of the Kubernetes pods. ```bash kubectl get pod ``` Confirm all of the pods are in running/completed state before continuing, similar to what is shown below. You may need to rerun the previous command a few times or add the `-w` flag to the previous command until everything stabilizes. ```bash NAME READY STATUS RESTARTS AGE catalog-6c768b578f-t2vmv 2/2 Running 0 23h catalog-init-xgxgd 0/1 Completed 3 23h control-6db59845fd-d4tx9 1/1 Running 0 48m control-api-0 2/2 Running 0 48m control-api-init-cjxrx 0/1 Completed 0 23h dashboard-b8bff9c69-rzp2z 2/2 Running 0 23h data-0 2/2 Running 0 23h data-internal-0 2/2 Running 0 23h data-mongo-0 1/1 Running 0 23h edge-ff5578dbf-5xkfw 1/1 Running 0 68s internal-data-mongo-0 1/1 Running 0 23h jwt-redis-db87bcc4c-v6dtw 1/1 Running 0 48m jwt-security-5c96d95749-2htgs 2/2 Running 0 48m mesh-redis-0 1/1 Running 0 48m postgres-slo-0 1/1 Running 0 23h prometheus-0 2/2 Running 0 23h slo-5fbd879488-87jcl 2/2 Running 0 23h ``` Once all the pods are in running/completed state, confirm that you can access the Grey Matter dashboard again. ### 9. Upgrade Grey Matter Sense We will now upgrade Grey Matter sense. Once we do this, the Catalog service will lose its knowledge of the services due to changing the persistence to Redis. The step after this one will remedy this. Switch to the sense directory and upgrade Grey Matter sense. ```bash cd ../sense && make upgrade-sense ``` You should get the below output if the Helm charts were applied successfully. ```bash rm -f ./charts/* helm dep up . Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "greymatter" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 3 charts Deleting outdated charts --disable-openapi-validation kubectl delete rolebinding prometheus-sa-rolebinding rolebinding.rbac.authorization.k8s.io "prometheus-sa-rolebinding" deleted helm upgrade sense . --set=global.waiter.service_account.create=false -f ../global.yaml --no-hooks --install --disable-openapi-validation Release "sense" has been upgraded. Happy Helming! NAME: sense LAST DEPLOYED: Mon Dec 7 14:13:09 2020 NAMESPACE: default STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: Grey Matter 3.0.1 has been installed. sense deployed to namespace "default" at 02:13:09 on 12/07/02 NOTE: It may take a few minutes for the installation to become stable. You can watch the status of the pods by running 'kubectl get pods -w -n default' ``` Run this command to check the status of the Kubernetes pods. ```bash kubectl get pod ``` Confirm all of the pods are in running/completed state before continuing, similar to what is shown below. You may need to rerun the previous command a few times or add the `-w` flag to the previous command until everything stabilizes. ```bash NAME READY STATUS RESTARTS AGE catalog-6c768b578f-t2vmv 2/2 Running 0 23h catalog-init-xgxgd 0/1 Completed 3 23h control-6db59845fd-d4tx9 1/1 Running 0 48m control-api-0 2/2 Running 0 48m control-api-init-cjxrx 0/1 Completed 0 23h dashboard-b8bff9c69-rzp2z 2/2 Running 0 23h data-0 2/2 Running 0 23h data-internal-0 2/2 Running 0 23h data-mongo-0 1/1 Running 0 23h edge-ff5578dbf-5xkfw 1/1 Running 0 68s internal-data-mongo-0 1/1 Running 0 23h jwt-redis-db87bcc4c-v6dtw 1/1 Running 0 48m jwt-security-5c96d95749-2htgs 2/2 Running 0 48m mesh-redis-0 1/1 Running 0 48m postgres-slo-0 1/1 Running 0 23h prometheus-0 2/2 Running 0 23h slo-5fbd879488-87jcl 2/2 Running 0 23h ``` ### 10. Re-populate Catalog Service Due to the change in persistence mechanisms, the Catalog service will need to be re-populated with the entries that were exported in a prior step. ```bash cd ../../gm-1.3-upgrade/catalog && ./populate.sh ``` You should see a bunch of configs getting applied and their config outputted in series. There should be no errors if everything is working and connected properly. Once the command completes, confirm that you have access to the Grey Matter dashboard and that the cards for the various services are present and green. ### 11. Download and run the Business Impacts migration tool Since the persistence mechanism for business impacts was migrated to Redis, you need to run a script in order to copy over the existing business impacts. First, navigate to the Nexus repository containing the different versions of the script using the below URL. Once logged in, you will see three (3) versions of the script available with different extensions: `.darwin` (for MacOS), `.linux` (for Linux), and `.windows` (for Windows). Click on the version appropriate for your computer. Then click on the link next to **Path** in the panel on the right to download. Before running the script, we will need to set some environment variables. ```bash export SLO_ROUTE=/services/slo/latest export CATALOG_ROUTE=/services/catalog/latest ``` Then, navigate to the location where the script was downloaded and run the script. In the example below, we are running the `.darwin` version of the script from the Mac default download location (`~/Downloads`). You may need to change the commands accordingly if working with another OS. ```bash cd ~/Downloads && ./migrate-business-impacts.darwin ``` You should see output similar to the below. The number of clusters updated may differ based on what's installed in the Grey Matter mesh and the business impact levels. ```bash Updated cluster Grey Matter Catalog 1.0.7 with business impact: 'high' Updated cluster Grey Matter Control API 1.4.3 with business impact: 'critical' Updated cluster Grey Matter Edge 1.4.5 with business impact: 'critical' Updated cluster Grey Matter JWT Security 1.1.1 with business impact: 'critical' Finished. 4 clusters updated. ``` At this point, the upgrade is complete. Check the dashboard and services to make sure everything looks like it should.