k8s_operator

Posted on Edited on In ,

Overview

When
Consider adding a Custom Resource to Kubernetes if you want to define new controllers, application configuration objects or other declarative API. it’s mostly used for complex stateful application.

How
Custom resources can appear and disappear in a running cluster through dynamic registration, and cluster admins can update custom resources independently of the cluster itself. Once a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like Pods.

Operator pattern
The combination of a custom resource API and a control loop is called the Operator pattern, The Operator pattern is used to manage specific, usually stateful, applications.

Kubernetes provides two ways to add custom resources to your cluster:

  • CRDs are simple and can be created without any programming.
  • API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.

CRDs are easier to use. Aggregated APIs are more flexible. Choose the method that best meets your needs.

Typically, CRDs are a good fit if:

  • You have a handful of fields
  • You are using the resource within your company, or as part of a small open-source project (as opposed to a commercial product)

Operator pattern

CRD

The CustomResourceDefinition(CRD) API resource allows you to define custom resources. Defining a CRD object creates a new custom resource with a name and schema that you specify.

Kubernetes client libraries can be used to access custom resources. Not all client libraries support custom resources. The Go and Python client libraries do.

When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server creates a new RESTful resource path for each version you specify. The CRD can be either namespaced or cluster-scoped, as specified in the CRD’s scope field. As with existing built-in objects, deleting a namespace deletes all custom objects in that namespace. CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.

resourcedefinition.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: crontabs.stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: stable.example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object

              # we can add more to each filed or property, like
              # - default value
              # - validation
              # - limit
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: crontabs
    # singular name to be used as an alias on the CLI and for display
    singular: crontab
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: CronTab
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - ct
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
$ kubectl apply -f resourcedefinition.yaml
# Then a new namespaced RESTful API endpoint is created at:
/apis/stable.example.com/v1/namespaces/*/crontabs/...

# then you can create custom object with these API.

# my-crontab.yaml

apiVersion: "stable.example.com/v1"
kind: CronTab
metadata:
  name: my-new-cron-object
spec:
  cronSpec: "* * * * */5"
  image: my-awesome-cron-image

$ kubectl apply -f my-crontab.yaml
$ kubectl get crontab

Delete CRD
When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint and delete all custom objects stored in it.

More detail to define CRD, refer to CRD guide.

controller

custom controller is a controller who is control loop that watches the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state.

custom controller is also client of the Kubernetes API that act on a Custom Resource, custom controller written with Go or Python runs a daemon to watch Custom Resource through API server and takes proper action to make it in desired state.

In order to write custom controller, you have to learn detail about client library, kubernetes provides client library for various programming languages, like Go, Python, Java, JS etc.

Samples:

Tools of writing operator

Operator involves defining CRD and writing custom controller, it’s little complex if starting from zero, as there are lots of common logic for custom controller, hence there are some tools you can use to write your own cloud native Operator.

  • Charmed Operator Framework
  • kubebuilder
  • KUDO (Kubernetes Universal Declarative Operator)
  • Metacontroller along with WebHooks that you implement yourself
  • Operator Framework (most popular one)
  • shell-operator

REF