Module Operations

• 1: Module Online and Offline
• 2: Module Deployment
• 3: Incompatible Deployment of Base and Module
• 4: Module Scaling and Replacement
• 5: Independent Use of Arklet
• 6: Module Information Retrieval
• 7: Module Traffic Service
• 8: Health Check
• 9: All K8S Resource Definitions and Deployment Methods
• 10:

1 - Module Online and Offline

Module Online

To bring a module online in the Kubernetes cluster, create a ModuleDeployment CR resource. For example:

kubectl apply -f koupleless/module-controller/config/samples/module-deployment_v1alpha1_moduledeployment.yaml --namespace yournamespace

Replace deployment_v1alpha1_moduledeployment.yaml with the path to your ModuleDeployment definition YAML file, and yournamespace with your namespace. Here’s the complete content of module-deployment_v1alpha1_moduledeployment.yaml:

apiVersion: koupleless.alipay.com/v1alpha1
kind: ModuleDeployment
metadata:
  labels:
    app.kubernetes.io/name: moduledeployment
    app.kubernetes.io/instance: moduledeployment-sample
    app.kubernetes.io/part-of: module-controller
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: module-controller
  name: moduledeployment-sample
spec:
  baseDeploymentName: dynamic-stock-deployment
  template:
    spec:
      module:
        name: provider
        version: '1.0.2'
        url: http://koupleless.oss-cn-shanghai.aliyuncs.com/module-packages/stable/dynamic-provider-1.0.2-ark-biz.jar
  replicas: 2
  operationStrategy:  # Customize deployment operation strategy here
    upgradePolicy: installThenUninstall
    needConfirm: true
    useBeta: false
    batchCount: 2
  schedulingStrategy: # Customize scheduling strategy here
    schedulingPolicy: Scatter

You can refer to the ModuleDeployment CRD fields explanation for all the fields of ModuleDeployment.
If you want to customize the module deployment operation strategy (such as grouping, beta, pause, etc.), you can configure the operationStrategy and schedulingStrategy. For more details, refer to Module Deployment Operation Strategy.
The example demonstrates using kubectl, but creating a ModuleDeployment CR directly via the K8S API server achieves the same result.

Module Offline To take a module offline in the Kubernetes cluster, delete the ModuleDeployment CR resource. For example:

kubectl delete yourmoduledeployment --namespace yournamespace

Replace yourmoduledeployment with the name of your ModuleDeployment (metadata name of ModuleDeployment), and yournamespace with your namespace. If you want to customize the module deployment operation strategy (such as grouping, beta, pause, etc.), you can configure the operationStrategy and schedulingStrategy. For more details, refer to Module Deployment Operation Strategy.
The example demonstrates using kubectl, but deleting a ModuleDeployment CR directly via the K8S API server achieves the same result.

2 - Module Deployment

Module Deployment

Modify the ModuleDeployment.spec.template.spec.module.version field and ModuleDeployment.spec.template.spec.module.url field (optional) and reapply, you can achieve the group deployment of the new version module, for example:

kubectl apply -f koupleless/module-controller/config/samples/module-deployment_v1alpha1_moduledeployment.yaml --namespace yournamespace

Replace deployment_v1alpha1_moduledeployment.yaml with your ModuleDeployment definition yaml file, and yournamespace with your namespace. The complete content of module-deployment_v1alpha1_moduledeployment.yaml is as follows:

apiVersion: koupleless.alipay.com/v1alpha1
kind: ModuleDeployment
metadata:
  labels:
    app.kubernetes.io/name: moduledeployment
    app.kubernetes.io/instance: moduledeployment-sample
    app.kubernetes.io/part-of: module-controller
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: module-controller
  name: moduledeployment-sample
spec:
  baseDeploymentName: dynamic-stock-deployment
  template:
    spec:
      module:
        name: provider
        version: '2.0.0'  # Note: Modify the version field from 1.0.2 to 2.0.0 here to achieve the group deployment of the new version module
        # Note: The url field can be modified to the new jar package address, or it can be left unchanged
        url: http://koupleless.oss-cn-shanghai.aliyuncs.com/module-packages/stable/dynamic-provider-1.0.2-ark-biz.jar
  replicas: 2
  operationStrategy:
    upgradePolicy: install_then_uninstall
    needConfirm: true
    grayTimeBetweenBatchSeconds: 0
    useBeta: false
    batchCount: 2
  schedulingStrategy:
    schedulingPolicy: scatter

If you want to customize the module deployment operation strategy, you can configure operationStrategy, specifically refer to Module Deployment Operation Strategy.
The example demonstrates using the kubectl method, directly calling the K8S APIServer to modify the ModuleDeployment CR can also achieve group deployment.

Module Rollback

Modify the ModuleDeployment.spec.template.spec.module.version field and ModuleDeployment.spec.template.spec.module.url field (optional) and reapply, you can achieve the group rollback deployment of the module.

3 - Incompatible Deployment of Base and Module

Step 1

Modify the code of both the base and the module, then build the base into a new version of the image and the module into a new version of the code package (in Java, it’s a JAR file).

Step 2

Modify the ModuleDeployment.spec.template.spec.module.url for the module to point to the new module code package.

Step 3

Use K8S Deployment to deploy the base to the new version of the image (which triggers the replacement or restart of the base container). When the base container starts, it will pull the latest module code package address from ModuleDeployment, thereby achieving the incompatible change between the base and the module (i.e., simultaneous deployment).

4 - Module Scaling and Replacement

Module Scaling

To scale a module, modify the replicas field of the ModuleDeployment CR and reapply it. For example:

kubectl apply -f koupleless/module-controller/config/samples/module-deployment_v1alpha1_moduledeployment.yaml --namespace yournamespace

Replace deployment_v1alpha1_moduledeployment.yaml with the path to your ModuleDeployment definition YAML file, and yournamespace with your namespace. Here’s the complete content of module-deployment_v1alpha1_moduledeployment.yaml:

apiVersion: koupleless.alipay.com/v1alpha1
kind: ModuleDeployment
metadata:
  labels:
    app.kubernetes.io/name: moduledeployment
    app.kubernetes.io/instance: moduledeployment-sample
    app.kubernetes.io/part-of: module-controller
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: module-controller
  name: moduledeployment-sample
spec:
  baseDeploymentName: dynamic-stock-deployment
  template:
    spec:
      module:
        name: provider
        version: '1.0.2'
        url: http://koupleless.oss-cn-shanghai.aliyuncs.com/module-packages/stable/dynamic-provider-1.0.2-ark-biz.jar
  replicas: 2  # Note: Modify the number of replicas here to scale the module instances
  operationStrategy:
    upgradePolicy: installThenUninstall
    needConfirm: true
    useBeta: false
    batchCount: 2
  schedulingStrategy: # Customize scheduling strategy here
    schedulingPolicy: Scatter  

If you want to customize the module deployment operation strategy (such as grouping, beta, pause, etc.), you can configure the operationStrategy and schedulingStrategy. For more details, refer to Module Deployment Operation Strategy.
The example demonstrates using kubectl, but modifying the ModuleDeployment CR directly via the K8S API server achieves the same result.

Module Replacement To replace a module in the Kubernetes cluster, delete the Module CR resource. For example:

kubectl delete yourmodule --namespace yournamespace

Replace yourmodule with the name of your Module CR entity (metadata name of Module), and yournamespace with your namespace. The example demonstrates using kubectl, but deleting a Module CR directly via the K8S API server achieves the same result.

5 - Independent Use of Arklet

Arklet, as an agent for Koupleless module deployment and operations (similar in function to K8S’s Kubelet), can be used independently without ModuleController. It exposes a set of HTTP interfaces for installing and uninstalling modules, allowing Koupleless to integrate with your own deployment and operations platform. See the API documentation here.

6 - Module Information Retrieval

View the names and statuses of all installed modules on a base instance

kubectl get module -n <namespace> -l koupleless.alipay.com/base-instance-ip=<pod-ip> -o custom-columns=NAME:.metadata.name,STATUS:.status.status

or

kubectl get module -n <namespace> -l koupleless.alipay.com/base-instance-name=<pod-name> -o custom-columns=NAME:.metadata.name,STATUS:.status.status

View detailed information of all installed modules on a base instance

kubectl describe module -n <namespace> -l koupleless.alipay.com/base-instance-ip=<pod-ip>

or

kubectl describe module -n <namespace> -l koupleless.alipay.com/base-instance-name=<pod-name>

Replace <pod-ip> with the IP of the base instance you want to view, <pod-name> with the name of the base instance you want to view, and <namespace> with the namespace of the resources you want to view.

7 - Module Traffic Service

Introduction to ModuleService

In Kubernetes, Service exposes a network application running on one or a set of Pods as a network service. Modules also support Module-related Services, automatically creating a service to serve the module during module deployment, exposing the module installed on one or a group of Pods as a network service. See: OperationStrategy.ServiceStrategy

apiVersion: koupleless.alipay.com/v1alpha1
kind: ModuleDeployment
metadata:
  labels:
    app.kubernetes.io/name: moduledeployment
    app.kubernetes.io/instance: moduledeployment-sample
    app.kubernetes.io/part-of: module-controller
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: module-controller
  name: moduledeployment-sample-provider
spec:
  baseDeploymentName: dynamic-stock-deployment
  template:
    spec:
      module:
        name: provider
        version: '1.0.2'
        url: http://koupleless.oss-cn-shanghai.aliyuncs.com/module-packages/stable/dynamic-provider-1.0.2-ark-biz.jar
  replicas: 1
  operationStrategy:
    needConfirm: false
    grayTimeBetweenBatchSeconds: 120
    useBeta: false
    batchCount: 1
    upgradePolicy: install_then_uninstall
    serviceStrategy:
      enableModuleService: true
      port: 8080
      targetPort: 8080
  schedulingStrategy:
    schedulingPolicy: scatter

Field Explanation

Explanation of the OperationStrategy.ServiceStrategy field:

Example

kubectl apply -f koupleless/module-controller/config/samples/module-deployment_v1alpha1_moduledeployment_provider.yaml --namespace yournamespace

Automatically created service for the module

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-11-03T09:52:22Z"
  name: dynamic-provider-service
  namespace: default
  resourceVersion: "28170024"
  uid: 1f85e468-65e3-4181-b40e-48959a069df5
spec:
  clusterIP: 10.0.147.22
  clusterIPs:
  - 10.0.147.22
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 32232
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    module.koupleless.alipay.com/dynamic-provider: "true"
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

8 - Health Check

Background

The purpose of health checks is to obtain the status of an application throughout its lifecycle, including the operational and runtime phases, so that users can make decisions based on this status. For instance, if the application status is DOWN, it indicates a malfunction in the application, and the user may choose to restart or replace the machine.

In the case of a single application, health checks are relatively simple:

• Operational phase status:
  • If it’s starting up, the status is UNKNOWN;
  • If startup fails, the status is DOWN;
  • If startup is successful, the status is UP.
• Runtime phase status:
  • If all health checkpoints of the application are healthy, the status is UP;
  • If any health checkpoint of the application is not healthy, the status is DOWN.

In multi-application scenarios, the situation can be much more complex. We need to consider the impact of the multi-application’s status during both the operational phase and the runtime phase on the overall application health. When designing health checks, we need to consider the following two issues:

• During the module operational phase, should the module start-up status affect the overall application health status?

  In different operational scenarios, users have different expectations. koupleless module operations have three scenarios:

  Scenario	Impact of the Module on the Overall Application Health Status
  Module Hot-Deployment	Provide configuration to let users decide whether the hot-deployment result should affect the overall application health status (default configuration is: does not affect the original health status of the application)
  Static Merge Deployment	Module deployment occurs during the base startup, so the module startup status should directly affect the overall health status of the application
  Module Replay	Module replay occurs during the base startup, thus the module startup status should directly affect the overall health status of the application

• During the module runtime phase, should the module running status affect the overall application health status?

  The module runtime phase status should have a direct impact on the overall application health status.

Under this context, we have designed a health check approach for multi-application scenarios.

Usage

Requirements

• Koupleless version >= 1.1.0
• sofa-ark version >= 2.2.9

Obtain the overall health status of the application

There are 3 types of health status for the base:

Since Koupleless supports hot deployment of modules, while obtaining the overall health status of the application, users may wish for the module deployment result to impact the overall application health status or not.

Module launch result does not affect the overall application health status (default)

• Features: For a healthy base, if the module installation fails, it will not affect the overall application health status.
• Usage: Same as the health check configuration for regular Spring Boot, configure in the base’s application.properties:

# or do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
# If you need to display all information, configure the following content
management.endpoint.health.show-components=always
management.endpoint.health.show-details=always

• Access: {baseIp:port}/actuator/health
• Result:

{
    // Overall health status of the application
    "status": "UP",
    "components": {
        // Aggregated health status of the modules
        "arkBizAggregate": {
            "status": "UP",
            "details": {
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP",
                    // Can see the health status of all active HealthIndicators in the modules
                    "details": {
                        "diskSpace": {
                          "status": "UP",
                          "details": {
                            "total": 494384795648,
                            "free": 272435396608,
                            "threshold": 10485760,
                            "exists": true
                            }
                        },
                        "pingHe": {
                          "status": "UP",
                          "details": {}
                        }
                    }
                }
            }
        },
        // Startup health status of base and modules
        "masterBizStartUp": {
            "status": "UP",
            // Including the startup status of each module.
            "details": {
                "base:1.0.0": {
                    "status": "UP"
                },
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP"
                },
                "biz2:0.0.1-SNAPSHOT": {
                    "status": "DOWN"
                }
            }
        }
    }
}

Overall Health Status in Different Scenarios

Scenario 1: Start base

Scenario 2: Start base and install modules with static merge deployment

Scenario 3: After base starts, install modules with hot deployment

Provide configuration to let users decide whether the result of module hot deployment affects the overall health status of the application (The default configuration is: Does not affect the original health status of the application)

Default Configuration: In the scenario of hot deployment, whether or not a module is successfully installed does not affect the overall health status of the application, as follows:

Scenario 4: Base running

Scenario 5: After base started, reinstall module

Reinstall module refers to automatically pulling the module baseline and installing the module after the base is started.

Reinstall module is not supported at the moment

Module launch result affects the overall application health status

• Features: For a healthy base, if a module installation fails, the overall application health status will also fail.
• Usage: In addition to the above configuration, you need to configure koupleless.healthcheck.base.readiness.withAllBizReadiness=true, that is, configure in the base’s application.properties:

# Alternatively, do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
# If you need to display all information, configure the following content
management.endpoint.health.show-components=always
management.endpoint.health.show-details=always
# Do not ignore module startup status
koupleless.healthcheck.base.readiness.withAllBizReadiness=true

• Access: {baseIp:port}/actuator/health
• Result:

{
    // Overall health status of the application
    "status": "UP",
    "components": {
        // Aggregated health status of the modules
        "arkBizAggregate": {
            "status": "UP",
            "details": {
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP",
                    // Can see the health status of all active HealthIndicators in the modules
                    "details": {
                        "diskSpace": {
                          "status": "UP",
                          "details": {
                            "total": 494384795648,
                            "free": 272435396608,
                            "threshold": 10485760,
                            "exists": true
                            }
                        },
                        "pingHe": {
                          "status": "UP",
                          "details": {}
                        }
                    }
                }
            }
        },
        // Startup health status of base and modules
        "masterBizStartUp": {
            "status": "UP",
            // Including the startup status of each module.
            "details": {
                "base:1.0.0": {
                    "status": "UP"
                },
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP"
                }
            }
        }
    }
}

Overall Health Status in Different Scenarios

Scenario 1: Start base

Scenario 2: Start base and install modules with static merge deployment

Scenario 3: After base starts, install modules with hot deployment

Provide configuration to let users decide whether the result of module hot deployment affects the overall health status of the application (The default configuration is: Does not affect the original health status of the application)

When configuring as koupleless.healthcheck.base.readiness.withAllBizReadiness=true:

Scenario 4: Base running

Scenario 5: After base started, reinstall module

Reinstall module refers to automatically pulling the module baseline and installing the module after the base is started.

Reinstall module is not supported at the moment.

Obtaining the Health Status of a Single Module

• Usage: Consistent with the normal springboot health check configuration, enable the health node, i.e. configure in the module’s application.properties:

# or do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health

• Access: {baseIp:port}/{bizWebContextPath}/actuator/info
• Result:

{
    "status": "UP",
    "components": {
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 494384795648,
                "free": 270828220416,
                "threshold": 10485760,
                "exists": true
            }
        },
        "ping": {
            "status": "UP"
        }
    }
}

Get information about base, modules, and plugins

• Usage: Same as the regular springboot health check configuration, enable the info endpoint, i.e., configure in the base’s application.properties:

# Note: If the user configures management.endpoints.web.exposure.include on their own, they need to include the health endpoint, otherwise the health endpoint cannot be accessed
management.endpoints.web.exposure.include=health,info

• Access: {baseIp:port}/actuator/info
• Result:

{
    "arkBizInfo": [
      {
        "bizName": "biz1",
        "bizVersion": "0.0.1-SNAPSHOT",
        "bizState": "ACTIVATED",
        "webContextPath": "biz1"
      },
      {
        "bizName": "base",
        "bizVersion": "1.0.0",
        "bizState": "ACTIVATED",
        "webContextPath": "/"
      }
    ],
    "arkPluginInfo": [
        {
            "pluginName": "koupleless-adapter-log4j2",
            "groupId": "com.alipay.sofa.koupleless",
            "artifactId": "koupleless-adapter-log4j2",
            "pluginVersion": "1.0.1-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/koupleless/koupleless-adapter-log4j2/1.0.1-SNAPSHOT/koupleless-adapter-log4j2-1.0.1-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.koupleless.adapter.Log4j2AdapterActivator"
        },
        {
            "pluginName": "web-ark-plugin",
            "groupId": "com.alipay.sofa",
            "artifactId": "web-ark-plugin",
            "pluginVersion": "2.2.9-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/web-ark-plugin/2.2.9-SNAPSHOT/web-ark-plugin-2.2.9-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.ark.web.embed.WebPluginActivator"
        },
        {
            "pluginName": "koupleless-base-plugin",
            "groupId": "com.alipay.sofa.koupleless",
            "artifactId": "koupleless-base-plugin",
            "pluginVersion": "1.0.1-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/koupleless/koupleless-base-plugin/1.0.1-SNAPSHOT/koupleless-base-plugin-1.0.1-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.koupleless.plugin.ServerlessRuntimeActivator"
        }
    ]
}

9 - All K8S Resource Definitions and Deployment Methods

Resource File Locations

1. ModuleDeployment CRD Definition
2. ModuleReplicaset CRD Definition
3. ModuleTemplate CRD Definition
4. Module CRD Definition
5. Role Definition
6. RBAC Definition
7. ServiceAccount Definition
8. ModuleController Deployment Definition

Deployment Method

Use the kubectl apply command to apply the 8 resource files listed above sequentially to deploy the ModuleController.

10 -

title: Module Deployment Operation Strategy date: 2024-01-25T10:28:32+08:00 description: Koupleless Module Deployment Operation Strategy weight: 600

Operation Strategy

To achieve lossless changes in the production environment, module deployment operations provide secure and reliable change capabilities. Users can configure the change strategy of deployment operations in the operationStrategy field of the ModuleDeployment CR spec. The specific fields in operationStrategy are explained as follows:

Scheduling Strategy

You can configure the Label “koupleless.alipay.com/max-module-count” for the base K8S Pod Deployment to specify how many modules can be installed on each Pod at most. Supports configuration of integers from 0 to N. Modules support scatter scheduling and stacking scheduling.
Scatter Scheduling: Set ModuleDeployment.spec.schedulingStrategy.schedulingPolicy to scatter. Scatter scheduling means that when modules are deployed, scaled, or replaced, they are preferentially scheduled to machines with the fewest number of modules installed.
Stacking Scheduling: Set ModuleDeployment.spec.schedulingStrategy.schedulingPolicy to stacking. Stacking scheduling means that when modules are deployed, scaled, or replaced, they are preferentially scheduled to machines with the most modules installed and not yet reaching the base max-module-count limit.

Protection Mechanism

(Under development, scheduled for release on October 15th) You can configure ModuleDeployment.spec.maxUnavailable to specify how many module replicas can be unavailable simultaneously during deployment operations. Module deployment operations require updating K8S Service and uninstalling modules, which may result in some module replicas being unavailable. Configuring it to 50% means that for a batch of module deployment operations, at least 50% of module replicas must be available, otherwise ModuleDeployment.status will display an error message.

Peer and Non-Peer

You can configure ModuleDeployment.spec.replicas to specify whether the module adopts peer-to-peer or non-peer deployment architecture.
Non-Peer Architecture: Set ModuleDeployment.spec.replicas to 0 - N to indicate a non-peer architecture. In a non-peer architecture, you must set the number of replicas for ModuleDeployment and ModuleReplicaSet, so scaling operations for modules are supported.
Peer Architecture: Set ModuleDeployment.spec.replicas to -1 to indicate a peer architecture. In a peer architecture, as many modules are installed on Pods as the number of replicas for the K8S Pod Deployment, and the number of module replicas is always consistent with the number of replicas for the K8S Pod Deployment. Therefore, scaling operations for modules are not supported in a peer architecture. Peer architecture is under construction and is scheduled for release on October 30th.