zt's Wiki

外观

自定义资源定义(CRD)开发

约 1387 字大约 5 分钟

kubernetescrd

2025-06-17

CRD（Custom Resource Definition）是 Kubernetes 的扩展机制，允许用户定义新的资源类型并像原生资源一样使用。本文将全面讲解 CRD 的 Schema 定义、验证、Webhook、代码生成等开发实践。

CRD 基本结构

CRD Schema（OpenAPI v3）

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
    shortNames:
      - wg                        # kubectl get wg
    categories:
      - all                       # kubectl get all 时也会显示
  scope: Namespaced               # Namespaced 或 Cluster
  versions:
    - name: v1
      served: true                # API Server 是否提供此版本
      storage: true               # 是否作为存储版本
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["replicas", "image"]
              properties:
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 100
                  default: 1
                  description: "Number of widget replicas"
                image:
                  type: string
                  pattern: '^[\w.-]+(/[\w.-]+)*(:\w[\w.-]*)?$'
                  description: "Container image for the widget"
                config:
                  type: object
                  x-kubernetes-preserve-unknown-fields: true  # 允许未知字段
                resources:
                  type: object
                  properties:
                    cpu:
                      type: string
                      pattern: '^(\d+m|\d+(\.\d+)?)$'
                    memory:
                      type: string
                      pattern: '^\d+(Ki|Mi|Gi|Ti)$'
                ports:
                  type: array
                  items:
                    type: object
                    required: ["port"]
                    properties:
                      name:
                        type: string
                      port:
                        type: integer
                        minimum: 1
                        maximum: 65535
                      protocol:
                        type: string
                        enum: ["TCP", "UDP"]
                        default: "TCP"
                  x-kubernetes-list-type: map
                  x-kubernetes-list-map-keys: ["port"]
                labels:
                  type: object
                  additionalProperties:
                    type: string
                  x-kubernetes-map-type: granular
            status:
              type: object
              properties:
                ready:
                  type: boolean
                replicas:
                  type: integer
                conditions:
                  type: array
                  items:
                    type: object
                    properties:
                      type:
                        type: string
                      status:
                        type: string
                        enum: ["True", "False", "Unknown"]
                      lastTransitionTime:
                        type: string
                        format: date-time
                      reason:
                        type: string
                      message:
                        type: string
      subresources:
        status: {}                  # 启用 status 子资源
        scale:                      # 启用 scale 子资源
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Ready
          type: boolean
          jsonPath: .status.ready
        - name: Image
          type: string
          jsonPath: .spec.image
          priority: 1               # priority > 0 在 -o wide 时显示
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp

创建 CR 实例：

apiVersion: example.com/v1
kind: Widget
metadata:
  name: my-widget
  namespace: default
spec:
  replicas: 3
  image: myapp:1.0
  resources:
    cpu: "500m"
    memory: "256Mi"
  ports:
    - name: http
      port: 8080
    - name: metrics
      port: 9090
  labels:
    env: production
    team: backend

Validation（验证）

Schema 验证

OpenAPI v3 Schema 提供了基本的声明式验证：

properties:
  name:
    type: string
    minLength: 1
    maxLength: 63
    pattern: '^[a-z][a-z0-9-]*[a-z0-9]$'
  count:
    type: integer
    minimum: 0
    maximum: 1000
    multipleOf: 2              # 必须是偶数
  mode:
    type: string
    enum: ["active", "passive", "standby"]
  tags:
    type: array
    minItems: 1
    maxItems: 10
    uniqueItems: true          # 元素不能重复
    items:
      type: string

CEL 验证表达式（Kubernetes 1.25+）

schema:
  openAPIV3Schema:
    type: object
    properties:
      spec:
        type: object
        x-kubernetes-validations:
          # 跨字段验证
          - rule: "self.minReplicas <= self.maxReplicas"
            message: "minReplicas must be less than or equal to maxReplicas"
          # 条件验证
          - rule: "self.mode == 'active' ? self.replicas >= 2 : true"
            message: "active mode requires at least 2 replicas"
          # 不可变字段
          - rule: "self.storageClass == oldSelf.storageClass"
            message: "storageClass is immutable"
        properties:
          minReplicas:
            type: integer
          maxReplicas:
            type: integer
          mode:
            type: string
          replicas:
            type: integer
          storageClass:
            type: string

Defaulting（默认值）

Schema 默认值

properties:
  protocol:
    type: string
    default: "TCP"
  timeout:
    type: integer
    default: 30

Mutating Webhook 默认值

更复杂的默认值逻辑需要通过 Webhook 实现：

func (r *Widget) Default() {
    if r.Spec.Replicas == 0 {
        r.Spec.Replicas = 1
    }
    if r.Spec.Resources.CPU == "" {
        r.Spec.Resources.CPU = "100m"
    }
    if r.Spec.Resources.Memory == "" {
        r.Spec.Resources.Memory = "128Mi"
    }
    // 添加标准标签
    if r.Labels == nil {
        r.Labels = make(map[string]string)
    }
    r.Labels["app.kubernetes.io/managed-by"] = "widget-operator"
}

Admission Webhooks

Validating Webhook 配置

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: widget-validator
webhooks:
  - name: validate.widgets.example.com
    clientConfig:
      service:
        name: widget-webhook
        namespace: widget-system
        path: /validate-widgets
      caBundle: <base64-ca-cert>
    rules:
      - apiGroups: ["example.com"]
        apiVersions: ["v1"]
        resources: ["widgets"]
        operations: ["CREATE", "UPDATE"]
    failurePolicy: Fail             # Fail 或 Ignore
    sideEffects: None
    admissionReviewVersions: ["v1"]
    matchPolicy: Equivalent
    timeoutSeconds: 10

Mutating Webhook 配置

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: widget-mutator
webhooks:
  - name: mutate.widgets.example.com
    clientConfig:
      service:
        name: widget-webhook
        namespace: widget-system
        path: /mutate-widgets
      caBundle: <base64-ca-cert>
    rules:
      - apiGroups: ["example.com"]
        apiVersions: ["v1"]
        resources: ["widgets"]
        operations: ["CREATE", "UPDATE"]
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1"]
    reinvocationPolicy: IfNeeded    # 如果其他 webhook 修改了对象，重新调用

Conversion Webhooks（版本转换）

当 CRD 有多个版本时，Conversion Webhook 负责版本间的转换：

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: widgets.example.com
spec:
  group: example.com
  versions:
    - name: v1
      served: true
      storage: true            # 存储版本
      schema:
        openAPIV3Schema:
          # v1 schema...
    - name: v1beta1
      served: true
      storage: false
      schema:
        openAPIV3Schema:
          # v1beta1 schema...
  conversion:
    strategy: Webhook
    webhook:
      clientConfig:
        service:
          name: widget-webhook
          namespace: widget-system
          path: /convert
        caBundle: <base64-ca-cert>
      conversionReviewVersions: ["v1"]

Status 子资源

启用 status 子资源后，spec 和 status 通过不同的 API 端点更新，实现权限分离：

# 更新 spec（需要 update 权限）
kubectl patch widget my-widget --type merge -p '{"spec":{"replicas":5}}'

# 更新 status（需要 update/status 权限）
kubectl patch widget my-widget --type merge --subresource status \
  -p '{"status":{"ready":true,"replicas":5}}'

RBAC 可以分别控制 spec 和 status 的访问权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: widget-status-updater
rules:
  - apiGroups: ["example.com"]
    resources: ["widgets/status"]    # 只能更新 status
    verbs: ["update", "patch"]

Printer Columns

自定义 kubectl get 输出列：

kubectl get widgets
# NAME        REPLICAS   READY   AGE
# my-widget   3          true    2d

kubectl get widgets -o wide
# NAME        REPLICAS   READY   IMAGE       AGE
# my-widget   3          true    myapp:1.0   2d

代码生成

Kubebuilder/controller-gen 基于 Go 类型定义自动生成 CRD YAML、DeepCopy 方法和 RBAC 配置：

# 生成 CRD manifests
controller-gen crd paths="./api/..." output:crd:dir=config/crd/bases

# 生成 DeepCopy 方法
controller-gen object paths="./api/..."

# 生成 RBAC manifests
controller-gen rbac:roleName=manager-role paths="./controllers/..."

client-gen 可生成强类型的 Kubernetes 客户端代码：

# 使用 code-generator
client-gen \
  --input-base "" \
  --input "github.com/example/api/v1alpha1" \
  --output-package "github.com/example/generated/clientset" \
  --clientset-name "versioned"

总结

CRD 开发的关键实践：

  1. Schema 设计要严谨，充分利用 OpenAPI v3 验证避免无效输入
  2. 使用 CEL 表达式 实现复杂的跨字段验证
  3. 启用 status 子资源，分离 spec 和 status 的更新权限
  4. Webhook 设置合理的 timeoutSeconds 和 failurePolicy
  5. 版本演进使用 Conversion Webhook，而非强制用户迁移
  6. 通过 Printer Columns 提升 kubectl 体验
  7. 自动化代码生成，减少手写模板代码

贡献者

withesse

更新日志

2026/3/14 13:09
查看所有更新日志
  • 9f6c2-feat: organize wiki content and refresh site setup于