# sync `greymatter sync` and its child commands allow the CLI users to setup "GitOps" i.e. all the mesh object definitions are stored in a version controlled storage such as GitHub. By running `greymatter sync apply`, CLI applies the stored configurations to destination if: 1. Resources' definitions in the `cache` storage differ from the last time `greymatter sync apply` was executed 2. Destination objects differ from the last time `greymatter sync apply` was executed You can review what actions will be taken if you run `apply` by running `greymatter sync plan`. Configuring sync functionality is complex and not needed for regular CLI use (unless you choose to do so). If `sync.source.command` is empty in `~/.greymatter.yaml`, CLI will assume that you are not intending to use this functionality, hence no further check for the validity of the configuration for sync command would be done. For more information about configuring sync functionality, please see the [Configuration](#configuration) section below. ## Child Commands | Command | Description | | ------- | ------------------------------------------------------------------------ | | `plan` | Plans the synchronization of the source and destination without applying | | `apply` | Applies the source to the destination | | `utils` | Utility commands for sync | ### plan `plan` allows us to review what actions will be taken when we synchronize the source and the destination. The output of `greymatter sync plan` looks like the following: ```bash Action: NOOP, Key: domain-prometheus-egress, Kind: DOMAIN Action: NOOP, Key: domain-dashboard-egress, Kind: DOMAIN Action: CREATE, Key: listener-control-api-egress, Kind: LISTENER Action: UPDATE, Key: cluster-jwt-security, Kind: CLUSTER Action: UPDATE, Key: edge-to-catalog-cluster, Kind: CLUSTER Action: DELETE, Key: edge-prometheus-route, Kind: ROUTE ``` * `NOOP` No operation * `CREATE` Source repository contains an object that does not exist in the destination * `UPDATE` Source object and destination object differs * `DELETE` Destination contains an object that does not exist in the source ### apply `apply` will actually synchronize the source and the destination mesh objects. ### Utils #### cipher `cipher` is a utility tool to encrypt/decrypt a value which will be used like this in the mesh object configurations: ``` "{{ decrypt "AgA+jVHymttkItPqpm1ukHecZ9lnPdtdHkqPvVdar0qk3HytmB3V8ZLVpK80CQzHVW7SEwPObZXwN+GbR0UJ3AHM/agbMWsaEXPnp76K27PliGJLLV5TU/CLpk0XLAp5CE8/zz4ORmLxwkd96YNCmPDkYCuAejRtKWg0Y1w2WE4iSO2cP46Eyba5HIzHNkf1l1TaOtJoHG+3ieyWFs4QrHCDHBHU4Tke+MZEsDHfDR1Mq7Q+S8Rjq3UVEoTQngSA9kxmGNjtgEp8F1tzIRf3GgB7YxKnCY8v8y+bqHAYFTUvJzQAbEZilWyfpcnT0bdqGpbgX2JLU96LkFo1XC87w3QdiATv/KRQgQsfopj5fagdRrdFe2pBvN0+GjvgWnCxy6EYZDEbPMEgVMHjGcI20/PaEndez28JecidybaZdF1gLg2y4osQQu5wdPoJMEWCs92bw20g8360UtbrmX4UC7q6nV4ddTcmbPcrcWDHCpkPKLJQ9NQohCw6/CB9UdD9IHB3D6JYVel5QBcxJP238dG2yzbLlE04Su7xLi1spg8CwV+H91RN5QpfulIKSSrSXhDL8obWIMhaz7yJ1VO/iJvJUN9c2SX7j10wY91YvTxi2bzbjtf0ZuJzsGDnQdAXQt7ZRWfUgagZHlab+TYV5ToawTAvkLT8cyKBwS0TbrrmCpTQ0FyYMgAxrahI+fLNlTZW7EwtICk=" }}" ``` > *The outer double quotes will surround the decrypted value to indicate it is a string* This is the simplest method to avoid storing sensitive information in the resource storage. However, since the configuration file itself has the actual encrypted value, you cannot use the same repository for multiple environments. For more flexible approach, please use `secret` below. For example, using `template.json` files with the following in it: ```bash { "hello": "{{ decrypt "AgADd7RVzRL8O8qeFdTfWkBP+RV/PdW4leL6Bj+LzJKJfd8Zm8gJNcD24Lhmqeig6kj4gxjsSaxvIpkhTLL0YHKAxYWKsYAooOBhhX3BGFXsXQmNWKJrUohHZLhZgxeAXCQifzni13DJcYRt7Gz7x/AQ8HSFfXQoWNVJLAbYFEYwJPGPZ6S7jclm54fhcxolVFsOEtN6Rh7IaorXvizMYUHi2SG/kE5KWQzcBHxHTZK4Jbt30DSspoKdXGmWcEA0VB5jVDFE3LexPN0v9leeo8mKEZjxeK8UxT5GMuueS9k3nychabwpWFWVH1hVXHqfjVDjY3Z9dGu/HqhbRJcFPWNa4qtydIZX4jKUaD2jfXT7fOYRgqhq1gOYCne4N/V5lTmuVaZvPRKr3xLXNgoDokft71oBgg8x06Lt0GTA0bAms3+1SOhr8svP/pxZ4VVUNmNYWwlrIeoxbqm3C+tmiOCf/Br22XOAtZHwe4Qr7YND15ZM29U4xlSk0aUFbDgx0PqLbcn1yAHJxy3WZOt1V6wGRoVb453EgIUtfunSvDkbmuC9BazYJh6WvAjGADz9KvFnMg5bRMPK1q7zGAKgfsIY24/AdXIfUc5+yaP1Mm2N71EKeUp2vFe0UgQg+0C0cYpZY8u6B/C6otXr6BQ07x4FEnwOtbRrX5Jb8P7bv8LYOTJ0X93XAFiizqDTBSv5q8GOI0te4w==" }}" } ``` The output of `greymatter sync utils render` will look as follows: ```bash $ greymatter sync utils render < template.json { "hello": "world" } ``` **encrypt** `encrypt` will read from stdin and encrypt that value. **decrypt** `decrypt` will read from stdin and decrypt that value. It is also used during the rendering process. Example: ```bash $ echo hello-world | greymatter sync utils cipher encrypt > output.txt $ greymatter sync utils cipher decrypt < output.txt hello-world ``` #### secret `secret` manages key-value pairs used to replace `{{ secret "key" }}` in templates with an actual value. This is similar to [`variable`](#variable) below, but allows it to come from somewhere else with different access restrictions (i.e., Secret). It has following child commands to manage the stored secrets: ```bash clear Clear the secret store delete Delete a value from the secret store get Get a value from the secret store list List the keys in the secret store put Put a value in the secret store ``` #### variable `variable` manages key-value pairs used to replace `{{ variable "key" }}` in templates with an actual value. Variable is intended to be used for environment specific things (i.e., in Kubernetes I can mount a Configmap to back the variables plugin). ```bash clear Clear the variable store delete Delete a value from the variable store get Get a value from the variable store list List the keys in the variable store put Put a value in the variable store ``` #### render `render` allows us to review what the rendered templates look like by reading the template from stdin: ```bash greymatter sync utils render < template.json ``` ## Configuration The common configuration in `.greymatter.yaml` looks like below: ```yaml sync: source: command: - greymatter - sync - plugin - remote - git configuration: url: git@github.com:user/repo.git reference: branch:master credentials: privateKey: rsa_private.pem folders: zone: zones proxy: proxies listener: listeners domain: domains route: routes shared_rules: rules cluster: clusters catalog_cluster: catalog_clusters catalog_zone: catalog_zones destination: command: - greymatter - sync - plugin - remote - api cache: command: - greymatter - sync - plugin - store - badger configuration: path: .cache/ variable: command: - greymatter - sync - plugin - store - badger configuration: path: .variable/ secret: command: - greymatter - sync - plugin - store - badger configuration: path: .secret/ cipher: command: - greymatter - sync - plugin - cipher - rsa configuration: privateKey: rsa_private.pem publicKey: rsa_public.pem ``` Grey Matter CLI uses [hashicorp/go-plugin](https://github.com/hashicorp/go-plugin) behind the scene, and `command` section in above example indicates which plugin to use. For example, if you were to simply run: ```bash greymatter sync plugin store badger ``` You will get an output like this: ``` This binary is a plugin. These are not meant to be executed directly. Please execute the program that consumes these plugins, which will load any plugins automatically ``` That is why plugin commands are hidden from `help`, but they exist. The `configuration:` section under a plugin's `command` in the configuration file (e.g. `.greymatter.yaml`) will contain necessary information for the particular plugin. ### Plugins #### cipher plugin cipher plugin is used by `greymatter sync utils cipher` * rsa: this is currently the only cipher plugin Configuration: * `privateKey`: the PKCS1 private key for the cipher which can be created like so: ```bash openssl genrsa -out rsa_private.pem 4096 ``` * `publicKey`: the public key for the cipher which can be created like so: ```bash openssl rsa -in rsa_private.pem -pubout -out rsa_public.pem ``` #### remote remote plugin is used by `greymatter sync plan` and `greymatter sync apply` * api: gm-control-api as a remote storage Configuration: * `delete`: whether resources that exist in the destination but not in the source should be deleted * git: GitHub as a remote storage Configuration: * `url`: the URL of the repository. The URL must be a valid URL, and not ssh shorthand (ssh://git\@github.com/user/repo instead of :user/repo) * `reference`: the branch, tag, or commit to clone, in the format `branch:`, `tag:`, or `commit:`. Leave empty for HEAD. * `credentials`: the credentials for authenticating with the repository * `username`: the username for authentication (HTTP or SSH). Leave empty to parse the username from the url. * `password`: the password for basic authentication (HTTP) or the private key (SSH). * `privateKey`: the path to the private key file (SSH). If no Private Key is passed, the ssh-agent will be used. * `folders`: a collection of folder names for folders that contain object definition files to compare with resources in the destination. #### renderer renderer plugin is used by `greymatter sync utils render` * contextual: the typical renderer Configuration: None * noop: a renderer that does not do any variable resolutions Configuration: None #### store store plugin is used as a key-value storage and used by `greymatter sync utils variable`, `greymatter sync utils secret`, as well as keeping track of what was applied and observed in order to determine what action to take when `greymatter sync plan` is executed. * badger: [dgraph-io/badger](https://github.com/dgraph-io/badger) as a key-value storage Configuration: * `path`: the path to the database directory * file: a file as a key-value storage Configuration: * `path`: the path to the store file * memory: this is used only for testing purposes. There is no persistence once the command exists. Configuration: None ## Example setup for a Git source and Grey Matter Control API destination ### Source The Git repository should store JSON object definition files of a given type (YAML support will come in future) in folders that match the names specified in `sync.source.configuration.folders`: ```yaml folders: zone: zones proxy: proxies listener: listeners domain: domains route: routes shared_rules: rules cluster: clusters catalog_cluster: catalog_clusters catalog_zone: catalog_zones ``` Given a folder name for an object definition type, **all** folders in the Git repository that match that folder name will be searched for JSON object definition files. For example, if the `domain` object definition type has the folder name `domain_configs`, then JSON object definition files can be found in the following folders: * `/domain_configs/` * `/zone-a/domain_configs/` * `/zone-b/domain_configs/` * `/other-configs/zone-c/service-1/domain_configs/` * etc. If an object definition type has no folder name configured, the default folder name specified in this example is used. Source and destination configuration should look similar to below: ```yaml source: command: - greymatter - sync - plugin - remote - git configuration: url: git@github.com:user/repo.git reference: branch:master credentials: privateKey: rsa_private.pem folders: zone: zones proxy: proxies listener: listeners domain: domains route: routes shared_rules: rules cluster: clusters catalog_cluster: catalog_clusters catalog_zone: catalog_zones destination: command: - greymatter - sync - plugin - remote - api ``` For more detail, please refer to [remote](####remote) section. ## Overlap between `greymatter generate` and `greymatter sync` `greymatter generate` only renders templates. `greymatter sync` is a GitOps tool that applies mesh configurations from a Git repository. They solve two different problems, `generate` is to quickly generate configurations on disk based upon templates controlled in a repository. `sync` is to synchronize configuration in a repository with the api as well as rendering templates. It allows the configurations to be templates and it will substitute values every time it tries to apply them. This is to: a) prevent secrets from being checked into git un-encrypted and b) allow for the same configurations to be applied to multiple meshes (by separate `sync` processes).