sync

greymatter sync and its child commands allow the CLI users to setup "GitOps" i.e. all the mesh object definitions are stored in a version controlled storage such as GitHub. By running greymatter sync apply, CLI applies the stored configurations to destination if:

1. Resources' definitions in the cache storage differ from the last time greymatter sync apply was executed

2. Destination objects differ from the last time greymatter sync apply was executed

You can review what actions will be taken if you run apply by running greymatter sync plan.

Configuring sync functionality is complex and not needed for regular CLI use (unless you choose to do so). If sync.source.command is empty in ~/.greymatter.yaml, CLI will assume that you are not intending to use this functionality, hence no further check for the validity of the configuration for sync command would be done. For more information about configuring sync functionality, please see the Configuration section below.

Child Commands

Command	Description
plan	Plans the synchronization of the source and destination without applying
apply	Applies the source to the destination
utils	Utility commands for sync

plan

plan allows us to review what actions will be taken when we synchronize the source and the destination. The output of greymatter sync plan looks like the following:

Action: NOOP, Key: domain-prometheus-egress, Kind: DOMAIN
Action: NOOP, Key: domain-dashboard-egress, Kind: DOMAIN
Action: CREATE, Key: listener-control-api-egress, Kind: LISTENER
Action: UPDATE, Key: cluster-jwt-security, Kind: CLUSTER
Action: UPDATE, Key: edge-to-catalog-cluster, Kind: CLUSTER
Action: DELETE, Key: edge-prometheus-route, Kind: ROUTE

• NOOP No operation

• CREATE Source repository contains an object that does not exist in the destination

• UPDATE Source object and destination object differs

• DELETE Destination contains an object that does not exist in the source

apply

apply will actually synchronize the source and the destination mesh objects.

Utils

cipher

cipher is a utility tool to encrypt/decrypt a value which will be used like this in the mesh object configurations:

"{{ decrypt "AgA+jVHymttkItPqpm1ukHecZ9lnPdtdHkqPvVdar0qk3HytmB3V8ZLVpK80CQzHVW7SEwPObZXwN+GbR0UJ3AHM/agbMWsaEXPnp76K27PliGJLLV5TU/CLpk0XLAp5CE8/zz4ORmLxwkd96YNCmPDkYCuAejRtKWg0Y1w2WE4iSO2cP46Eyba5HIzHNkf1l1TaOtJoHG+3ieyWFs4QrHCDHBHU4Tke+MZEsDHfDR1Mq7Q+S8Rjq3UVEoTQngSA9kxmGNjtgEp8F1tzIRf3GgB7YxKnCY8v8y+bqHAYFTUvJzQAbEZilWyfpcnT0bdqGpbgX2JLU96LkFo1XC87w3QdiATv/KRQgQsfopj5fagdRrdFe2pBvN0+GjvgWnCxy6EYZDEbPMEgVMHjGcI20/PaEndez28JecidybaZdF1gLg2y4osQQu5wdPoJMEWCs92bw20g8360UtbrmX4UC7q6nV4ddTcmbPcrcWDHCpkPKLJQ9NQohCw6/CB9UdD9IHB3D6JYVel5QBcxJP238dG2yzbLlE04Su7xLi1spg8CwV+H91RN5QpfulIKSSrSXhDL8obWIMhaz7yJ1VO/iJvJUN9c2SX7j10wY91YvTxi2bzbjtf0ZuJzsGDnQdAXQt7ZRWfUgagZHlab+TYV5ToawTAvkLT8cyKBwS0TbrrmCpTQ0FyYMgAxrahI+fLNlTZW7EwtICk=" }}"

The outer double quotes will surround the decrypted value to indicate it is a string

This is the simplest method to avoid storing sensitive information in the resource storage. However, since the configuration file itself has the actual encrypted value, you cannot use the same repository for multiple environments. For more flexible approach, please use secret below.

For example, using template.json files with the following in it:

{
  "hello": "{{ decrypt "AgADd7RVzRL8O8qeFdTfWkBP+RV/PdW4leL6Bj+LzJKJfd8Zm8gJNcD24Lhmqeig6kj4gxjsSaxvIpkhTLL0YHKAxYWKsYAooOBhhX3BGFXsXQmNWKJrUohHZLhZgxeAXCQifzni13DJcYRt7Gz7x/AQ8HSFfXQoWNVJLAbYFEYwJPGPZ6S7jclm54fhcxolVFsOEtN6Rh7IaorXvizMYUHi2SG/kE5KWQzcBHxHTZK4Jbt30DSspoKdXGmWcEA0VB5jVDFE3LexPN0v9leeo8mKEZjxeK8UxT5GMuueS9k3nychabwpWFWVH1hVXHqfjVDjY3Z9dGu/HqhbRJcFPWNa4qtydIZX4jKUaD2jfXT7fOYRgqhq1gOYCne4N/V5lTmuVaZvPRKr3xLXNgoDokft71oBgg8x06Lt0GTA0bAms3+1SOhr8svP/pxZ4VVUNmNYWwlrIeoxbqm3C+tmiOCf/Br22XOAtZHwe4Qr7YND15ZM29U4xlSk0aUFbDgx0PqLbcn1yAHJxy3WZOt1V6wGRoVb453EgIUtfunSvDkbmuC9BazYJh6WvAjGADz9KvFnMg5bRMPK1q7zGAKgfsIY24/AdXIfUc5+yaP1Mm2N71EKeUp2vFe0UgQg+0C0cYpZY8u6B/C6otXr6BQ07x4FEnwOtbRrX5Jb8P7bv8LYOTJ0X93XAFiizqDTBSv5q8GOI0te4w==" }}"
}

The output of greymatter sync utils render will look as follows:

$ greymatter sync utils render < template.json
{
  "hello": "world"
}

encrypt

encrypt will read from stdin and encrypt that value.

decrypt

decrypt will read from stdin and decrypt that value. It is also used during the rendering process.

Example:

$ echo hello-world | greymatter sync utils cipher encrypt > output.txt
$ greymatter sync utils cipher decrypt < output.txt
hello-world

secret

secret manages key-value pairs used to replace {{ secret "key" }} in templates with an actual value. This is similar to variable below, but allows it to come from somewhere else with different access restrictions (i.e., Secret).

It has following child commands to manage the stored secrets:

  clear       Clear the secret store
  delete      Delete a value from the secret store
  get         Get a value from the secret store
  list        List the keys in the secret store
  put         Put a value in the secret store

variable

variable manages key-value pairs used to replace {{ variable "key" }} in templates with an actual value.

Variable is intended to be used for environment specific things (i.e., in Kubernetes I can mount a Configmap to back the variables plugin).

    clear       Clear the variable store
    delete      Delete a value from the variable store
    get         Get a value from the variable store
    list        List the keys in the variable store
    put         Put a value in the variable store

render

render allows us to review what the rendered templates look like by reading the template from stdin:

greymatter sync utils render < template.json

Configuration

The common configuration in .greymatter.yaml looks like below:

sync:
  source:
    command:
      - greymatter
      - sync
      - plugin
      - remote
      - git
    configuration:
      url: [e-mail]:user/repo.git
      reference: branch:master
      credentials:
        privateKey: rsa_private.pem
      folders:
        zone: zones
        proxy: proxies
        listener: listeners
        domain: domains
        route: routes
        shared_rules: rules
        cluster: clusters
        catalog_cluster: catalog_clusters
        catalog_zone: catalog_zones

  destination:
    command:
      - greymatter
      - sync
      - plugin
      - remote
      - api

  cache:
    command:
      - greymatter
      - sync
      - plugin
      - store
      - badger
    configuration:
      path: .cache/

  variable:
    command:
      - greymatter
      - sync
      - plugin
      - store
      - badger
    configuration:
      path: .variable/

  secret:
    command:
      - greymatter
      - sync
      - plugin
      - store
      - badger
    configuration:
      path: .secret/

  cipher:
    command:
      - greymatter
      - sync
      - plugin
      - cipher
      - rsa
    configuration:
      privateKey: rsa_private.pem
      publicKey: rsa_public.pem

Grey Matter CLI uses hashicorp/go-plugin behind the scene, and command section in above example indicates which plugin to use. For example, if you were to simply run:

greymatter sync plugin store badger

You will get an output like this:

This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

That is why plugin commands are hidden from help, but they exist.

The configuration: section under a plugin's command in the configuration file (e.g. .greymatter.yaml) will contain necessary information for the particular plugin.

Plugins

cipher plugin

cipher plugin is used by greymatter sync utils cipher

• rsa: this is currently the only cipher plugin Configuration:

  • privateKey: the PKCS1 private key for the cipher which can be created like so:

    openssl genrsa -out rsa_private.pem 4096

  • publicKey: the public key for the cipher which can be created like so:

    openssl rsa -in rsa_private.pem -pubout -out rsa_public.pem

remote

remote plugin is used by greymatter sync plan and greymatter sync apply

• api: gm-control-api as a remote storage Configuration:

  • delete: whether resources that exist in the destination but not in the source should be deleted

• git: GitHub as a remote storage Configuration:

  • url: the URL of the repository. The URL must be a valid URL, and not ssh shorthand (ssh://[e-mail]/user/repo instead of [e-mail]:user/repo)

  • reference: the branch, tag, or commit to clone, in the format branch:<branch_name>, tag:<tag>, or commit:<commit>. Leave empty for HEAD.

  • credentials: the credentials for authenticating with the repository

    • username: the username for authentication (HTTP or SSH). Leave empty to parse the username from the url.

    • password: the password for basic authentication (HTTP) or the private key (SSH).

    • privateKey: the path to the private key file (SSH). If no Private Key is passed, the ssh-agent will be used.

  • folders: a collection of folder names for folders that contain object definition files to compare with resources in the destination.

renderer

renderer plugin is used by greymatter sync utils render

• contextual: the typical renderer Configuration: None

• noop: a renderer that does not do any variable resolutions Configuration: None

store

store plugin is used as a key-value storage and used by greymatter sync utils variable, greymatter sync utils secret, as well as keeping track of what was applied and observed in order to determine what action to take when greymatter sync plan is executed.

• badger: dgraph-io/badger as a key-value storage Configuration:

  • path: the path to the database directory

• file: a file as a key-value storage Configuration:

  • path: the path to the store file

• memory: this is used only for testing purposes. There is no persistence once the command exists. Configuration: None

Example setup for a Git source and Grey Matter Control API destination

Source

The Git repository should store JSON object definition files of a given type (YAML support will come in future) in folders that match the names specified in sync.source.configuration.folders:

folders:
  zone: zones
  proxy: proxies
  listener: listeners
  domain: domains
  route: routes
  shared_rules: rules
  cluster: clusters
  catalog_cluster: catalog_clusters
  catalog_zone: catalog_zones

Given a folder name for an object definition type, all folders in the Git repository that match that folder name will be searched for JSON object definition files. For example, if the domain object definition type has the folder name domain_configs, then JSON object definition files can be found in the following folders:

• /domain_configs/

• /zone-a/domain_configs/

• /zone-b/domain_configs/

• /other-configs/zone-c/service-1/domain_configs/

• etc.

If an object definition type has no folder name configured, the default folder name specified in this example is used.

Source and destination configuration should look similar to below:

  source:
    command:
      - greymatter
      - sync
      - plugin
      - remote
      - git
    configuration:
      url: [e-mail]:user/repo.git
      reference: branch:master
      credentials:
        privateKey: rsa_private.pem
      folders:
        zone: zones
        proxy: proxies
        listener: listeners
        domain: domains
        route: routes
        shared_rules: rules
        cluster: clusters
        catalog_cluster: catalog_clusters
        catalog_zone: catalog_zones

  destination:
    command:
      - greymatter
      - sync
      - plugin
      - remote
      - api

For more detail, please refer to remote section.

Overlap between greymatter generate and greymatter sync

greymatter generate only renders templates. greymatter sync is a GitOps tool that applies mesh configurations from a Git repository. They solve two different problems, generate is to quickly generate configurations on disk based upon templates controlled in a repository. sync is to synchronize configuration in a repository with the api as well as rendering templates. It allows the configurations to be templates and it will substitute values every time it tries to apply them. This is to:

a) prevent secrets from being checked into git un-encrypted and

b) allow for the same configurations to be applied to multiple meshes (by separate sync processes).