Kubernetes Discovery

Kubernetes

Description

Grey Matter Control discovers service instances from kubernetes pods using k8s namespace and pod specifications.

Usage

To use kubernetes discovery with Grey Matter Control running within the kubernetes cluster, set the environment variables specified here.

To use kubernetes discovery with Grey Matter Control running outside of the kubernetes cluster, set the environment variables specified here.

Environment Variable	Description	Type	Default	Required
GM_CONTROL_KUBERNETES_NAMESPACES	Comma-delimited list of kubernetes namespaces to discover pods in	string	default	false
GM_CONTROL_KUBERNETES_PORT_NAME	Named container port assigned to cluster instances	string	http	false
GM_CONTROL_KUBERNETES_CLUSTER_LABEL	Kubernetes label that specifies the cluster a pod belongs to	string	gm_cluster	false
GM_CONTROL_KUBERNETES_SELECTOR	Kubernetes label selector that selects which pods are polled	string		false
GM_CONTROL_KUBERNETES_TIMEOUT	Timeout in seconds used for Kubernetes API requests	string	120	false
GM_CONTROL_KUBERNETES_LOG_LEVEL	Log level used for this discovery plugin	string	info	false
GM_CONTROL_KUBERNETES_KUBERNETES_HOST	Hostname to connect to kubernetes API server. Required if the Control server is to run outside of the kubernetes cluster	string		false
GM_CONTROL_KUBERNETES_CA_CERT	Path to trusted root cert for kubernetes API server.	string		false
GM_CONTROL_KUBERNETES_CLIENT_KEY	Path to key file for Control server to authenticate itself with the kubernetes API server.	string		false
GM_CONTROL_KUBERNETES_CLIENT_CERT	Path to certificate file for Control server to authenticate itself with the kubernetes API server.	string		false

Server Running Inside of Kubernetes Cluster

As described above in the required environment variables, for a Control server running within the Kubernetes cluster it will be discovering from, there are no required variables outside of GM_CONTROL_CMD. However, the full set of optional variables is:

# required
GM_CONTROL_CMD=kubernetes
# optional
GM_CONTROL_KUBERNETES_NAMESPACES={}
GM_CONTROL_KUBERNETES_PORT_NAME={}
GM_CONTROL_KUBERNETES_CLUSTER_LABEL={}
GM_CONTROL_KUBERNETES_SELECTOR={}
GM_CONTROL_KUBERNETES_TIMEOUT={}
GM_CONTROL_KUBERNETES_LOG_LEVEL={}

Server Running Outside of Kubernetes Cluster

For a Control server running outside of the Kubernetes cluster it will be discovering from, there are several required variables:

# required
GM_CONTROL_CMD=kubernetes
GM_CONTROL_KUBERNETES_KUBERNETES_HOST={}
GM_CONTROL_KUBERNETES_CA_CERT={}
GM_CONTROL_KUBERNETES_CLIENT_KEY={}
GM_CONTROL_KUBERNETES_CLIENT_CERT={}
# optional
GM_CONTROL_KUBERNETES_NAMESPACES={}
GM_CONTROL_KUBERNETES_PORT_NAME={}
GM_CONTROL_KUBERNETES_CLUSTER_LABEL={}
GM_CONTROL_KUBERNETES_SELECTOR={}
GM_CONTROL_KUBERNETES_TIMEOUT={}
GM_CONTROL_KUBERNETES_LOG_LEVEL={}

Kubernetes Specifications

Grey Matter Control will discover kubernetes pod instances based on the criteria specified in the environment variables. It will watch any kubernetes namespaces specified by name in the list of GM_CONTROL_KUBERNETES_NAMESPACES. If this value is not set, it will watch the default namespace only.

Only pods within the specified namespaces that also have the pod label in GM_CONTROL_KUBERNETES_SELECTOR will be polled for discovery. By default this value is nil, so all pods within the namespaces will polled. In each pod, all containers must be running before the pod is considered live and ready for inclusion in the API cluster's instance list.

Pods that are being polled will then be checked for another pod label, specified with GM_CONTROL_KUBERNETES_CLUSTER_LABEL - or gm_cluster by default. In order to link a pod to an api cluster, the pod must have this label with value equal to the api cluster name that it corresponds to. For example, if using the default value and a pod has label: gm_cluster: test-cluster, the pod will be discovered for cluster with name test-cluster.

Instances will be created for the cluster with name corresponding to the value of the pod label with key gm_cluster. The host will be the podIP of this pod, and the port will be taken from the container port on the pod with name GM_CONTROL_KUBERNETES_PORT_NAME. If GM_CONTROL_KUBERNETES_PORT_NAME is empty, the first TCP port found is used as the API instance's port. Pods with no container port are ignored. All pod labels (except for the cluster label) are attached as instance metadata.

For example, for a Control server running in a Kubernetes cluster using all of the default values, configuration:

GM_CONTROL_CMD=kubernetes

Pods in the default namespace, with pod label gm_cluster and value of the cluster-name and containing a container with port named http, will be discovered.

Kubernetes Permissions

If the Grey Matter Control server is running within the kubernetes cluster in its own pod, it will need certain permissions to poll the namespaces specified in GM_CONTROL_KUBERNETES_NAMESPACES. See service accounts and single namespace permissions if only one namespace is specified (or if the default default is being used), or multiple namespace permissions if you have specified a list of namespaces.

Service Accounts

The pod running the Control server must have an authorized service account granted by an admin. If access to any or all of the namespaces are revoked, Grey Matter Control will use its last known state of discovered pods. Meanwhile, it will continue attempting to connect to the internal Kubernetes APIs until access is restored.

To grant access, have a cluster admin apply either the Single Namespace or Multiple Namespaces resources (replacing the greymatter namespace with the namespace of the running control server). Then they can add the service account to the pod running Grey Matter Control as follows:

    spec:
      serviceAccountName: control

Single Namespace

This is used when Grey Matter Control is only discovering services in the same namespace it's running in.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: control
  namespace: greymatter

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: control-manager
  namespace: greymatter
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

---

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: control-binding
  namespace: greymatter
subjects:
- kind: ServiceAccount
  name: control
  namespace: greymatter
roleRef:
  kind: Role
  name: control-manager
  apiGroup: rbac.authorization.k8s.io

Multiple Namespaces

This is used when Grey Matter Control will be discovering services from multiple namespaces across the cluster.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: control
  namespace: greymatter

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: control-manager
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: control-binding
subjects:
- kind: ServiceAccount
  name: control
  namespace: greymatter
roleRef:
  kind: ClusterRole
  name: control-manager
  apiGroup: rbac.authorization.k8s.io

Command Line

To use the command line, run gm-control with:

gm-control kubernetes <global-flags>

Help

For help or to list available options for kubernetes discovery using the cli, run gm-control kubernetes --help.