This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

4. User Manual

Koupleless User Manual

1 - 4.1 Base Access

1.1 - 4.1.1 SpringBoot or SOFABoot Upgrade to Base

Upgrade SpringBoot or SOFABoot to Koupleless Base

We can create Biz Module in three ways, and this article introduces the second one:

  1. Splitting a large application into multiple modules
  2. Transforming an existing application into a single module
  3. Directly creating a module using a scaffold
  4. Transform ordinary code fragments into a module

This article introduces the operation and verification steps of how to upgrade existing SpringBoot or SOFABoot to modules at a low cost. It only requires adding an ark packaging plugin and configuring module slimming to achieve the one-click upgrade of a regular application to a module application. With the same set of code branches, the application can be independently started like the original SpringBoot, and can also be merged and deployed together with other applications as a module.

Prerequisites

  1. SpringBoot version >= 2.1.9.RELEASE (for SpringBoot users)
  2. SOFABoot version >= 3.9.0 or SOFABoot >= 4.0.0 (for SOFABoot users)
  3. Install maven version >= 3.9.0 locally

Note: SpringBoot version == 2.1.9.RELEASE, see Upgrade SpringBoot 2.1.9 to base

Access Steps

Code and Configuration Modifications

Modify application.properties

# Need to define the application name
spring.application.name = ${Replace with actual base app name}

Modify the main pom.xml

<properties>
    <sofa.ark.version>2.2.15</sofa.ark.version>
    <koupleless.runtime.version>1.3.2</koupleless.runtime.version>
</properties>

<!-- Place this as the first dependency in your build pom -->
<dependency>
    <groupId>com.alipay.koupleless</groupId>
    <artifactId>koupleless-base-starter</artifactId>
    <version>${koupleless.runtime.version}</version>
</dependency>

<!-- If using Spring Boot web, add this dependency. For more details, see https://www.sofastack.tech/projects/sofa-boot/sofa-ark-multi-web-component-deploy/ -->
<dependency>
    <groupId>com.alipay.sofa</groupId>
    <artifactId>web-ark-plugin</artifactId>
</dependency>

<!-- 为了让三方依赖和 koupleless 模式适配，需要引入以下构建插件 -->
<build>
    <plugins>
        <plugin>
            <groupId>com.alipay.sofa.koupleless</groupId>
            <artifactId>koupleless-base-build-plugin</artifactId>
            <version>${koupleless.runtime.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>add-patch</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Integration for Other Versions

Upgrade SpringBoot 2.1.9 to Base

After modifying the above configurations, additional modifications are required:

Modify main pom.xml
<!-- Place this as the first dependency in your pom -->
<dependency>
    <groupId>com.alipay.sofa.koupleless</groupId>
    <artifactId>koupleless-base-starter</artifactId>
    <version>${koupleless.runtime.version}</version>
</dependency>

<!-- If using Spring Boot web, add this dependency. For more details, see https://www.sofastack.tech/projects/sofa-boot/sofa-ark-multi-web-component-deploy/ -->
<dependency>
    <groupId>com.alipay.sofa</groupId>
    <artifactId>web-ark-plugin</artifactId>
</dependency>
<dependency>
    <groupId>com.github.oshi</groupId>
    <artifactId>oshi-core</artifactId>
    <version>3.9.1</version>
</dependency>

        
<!-- 为了让三方依赖和 koupleless 模式适配，需要引入以下构建插件 -->
<build>
    <plugins>
        <plugin>
            <groupId>com.alipay.sofa.koupleless</groupId>
            <artifactId>koupleless-base-build-plugin</artifactId>
            <version>${koupleless.runtime.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>add-patch</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
Modify base startup class

If version of koupleless is equals 1.1.0 or higher than 1.1.0, no need to change。

If version of koupleless is lower than 1.1.0, exclude the HealthAutoConfiguration class in the @SpringBootApplication annotation of the base Springboot startup class, as shown below:

import com.alipay.sofa.koupleless.arklet.springboot.starter.health.HealthAutoConfiguration;
@SpringBootApplication(exclude = { HealthAutoConfiguration.class })
public class BaseApplication {
    public static void main(String[] args) {
        SpringApplication.run(BaseApplication.class, args);
    }
}

Startup Verification

If the foundation application can start normally, the validation is successful!



2 - 4.2 Module Access

Koupleless Module Access

2.1 - 4.2.1 Upgrade to Module from existing SpringBoot or SOFABoot

Upgrade to Module from existing SpringBoot or SOFABoot

We can create Biz Module in three ways, and this article introduces the second one:

  1. Splitting a large application into multiple modules
  2. Transforming an existing application into a single module
  3. Directly creating a module using a scaffold

This article introduces how existing SpringBoot or SOFABoot applications can be cost-effectively upgraded to modules with the operational and validation steps. It requires only the addition of an ark packaging plugin + configuration for module slimming to enable a conventional application to be upgraded to a module application at the push of a button. Moreover, the same set of code branches can be used for independent startup as before, just like a regular SpringBoot application, as well as being capable of being deployed together with other applications as a module.

Prerequisites

  1. SpringBoot version >= 2.3.0 (for SpringBoot users)
  2. SOFABoot >= 3.9.0 or SOFABoot >= 4.0.0 (for SOFABoot users)

Access Steps

Step 1: Modify application.properties

# Need to define the application name
spring.application.name = ${Replace with actual module app name}

Step 2: Add Dependencies and Packaging Plugins for the Module

Note： The order of defining the sofa-ark plugin must be before the springboot packaging plugin;

<!-- Dependencies required for the module, mainly for inter-module communication --> 
<dependencies>
    <dependency>
        <groupId>com.alipay.sofa.koupleless</groupId>
        <artifactId>koupleless-app-starter</artifactId>
        <scope>provided</scope>
    </dependency>
</dependencies>

<plugins>
<!-- Add the ark packaging plugin here -->
    <plugin>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>sofa-ark-maven-plugin</artifactId>
        <version>{sofa.ark.version}</version>
        <executions>
            <execution>
                <id>default-cli</id>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <skipArkExecutable>true</skipArkExecutable>
            <outputDirectory>./target</outputDirectory>
            <bizName>${Replace with module name}</bizName>
            <webContextPath>${Module's custom web context path}</webContextPath>
            <declaredMode>true</declaredMode>
        </configuration>
    </plugin>
<!-- Build a regular SpringBoot fat jar, used for independent deployment, can be removed if not needed -->
    <plugin>
        <!-- Original spring-boot packaging plugin -->
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
    </plugin>
</plugins>

Step 3: Automate Module Slimming

You can leverage the automated slimming capability provided by the ark packaging plugin to slim down the Maven dependencies in your module application. This step is mandatory; otherwise, the resulting module JAR file will be very large, and startup may fail. Extended Reading: If the module does not optimize its dependenciesWhat will happen if SpringBoot framework is imported independently?

Step 4: Build the Module Jar Package

Execute mvn clean package -DskipTest, you can find the packaged ark biz jar in the target directory, or you can find the packaged regular springboot jar in the target/boot directory.

Tip：Full Middleware Compatibility List Supported in the Module。

Experiment: Verifying that the module can be started independently and deployed as a combined module

After adding the module packaging plugin (sofa-ark-maven-plugin) for packaging, only the ark-biz.jar build artifact will be added, which does not conflict with or affect the executable Jar built by the native spring-boot-maven-plugin. When deploying on the server, if you want to start independently, use the executable Jar built by the native spring-boot-maven-plugin as the build artifact; if you want to deploy as an ark module to the base, use the ark-biz.jar built by the sofa-ark-maven-plugin as the build artifact.

Verification of Deployment to the Base

  1. Start the base from the previous step (verification of independent startup).
  2. Initiate module deployment
curl --location --request POST 'localhost:1238/installBiz' \
--header 'Content-Type: application/json' \
--data '{
    "bizName": "${Module Name}",
    "bizVersion": "${Module Version}",
    "bizUrl": "file:///path/to/ark/biz/jar/target/xx-xxxx-ark-biz.jar"
}'

If the following information is returned, it indicates that the module is installed successfully.
image.png

  1. View Current Module Information: Besides the base “base,” there is also a module named “dynamic-provider.”

image.png

  1. Uninstall the module
curl --location --request POST 'localhost:1238/uninstallBiz' \
--header 'Content-Type: application/json' \
--data '{
    "bizName": "dynamic-provider",
    "bizVersion": "0.0.1-SNAPSHOT"
}'

If the following information is returned, it indicates that the uninstallation was successful.

{
    "code": "SUCCESS",
    "data": {
        "code": "SUCCESS",
        "message": "Uninstall biz: dynamic-provider:0.0.1-SNAPSHOT success."
    }
}

Verification of Independent Startup

After transforming a regular application into a module, it can still be started independently to verify some basic startup logic. Simply check the option to automatically add provided scope to the classpath in the startup configuration, and then use the same startup method as for regular applications. Modules transformed through automatic slimming can also be started directly using the SpringBoot jar package located in the target/boot directory. For more details, please refer to this link
image.png

2.2 - 4.2.2 Creating Modules Using Maven Archetype

We can create Biz Module in three ways, and this article introduces the second one:

  1. Splitting a large application into multiple modules
  2. Transforming an existing application into a single module
  3. Directly creating a module using a scaffold
  4. Transform ordinary code fragments into a module

It’s easy to creating a module from maven archetype, all you need to do is input the Maven groupId and artifactId for the archetype in IDEA.

<dependency>
    <groupId>com.alipay.sofa.koupleless</groupId>
    <artifactId>koupleless-common-module-archetype</artifactId>
    <version>{koupleless.runtime.version}</version>
</dependency>

The module created from this archetype has already integrated the module packaging plugin and automatic slimming configuration. It can be directly packaged as a module and installed on the base, or started independently locally.

2.3 -

title: 4.2.3 Java Code Fragment as Module
date: 2024-01-25T10:28:32+08:00
description: Java Code Fragment as Module
weight: 310

Module creation has four methods, and this article introduces the fourth method:

  1. Split multiple modules from a large application
  2. Transform existing applications into a single module
  3. Create a module directly using scaffolding
  4. Transform ordinary code fragments into a module

This article introduces the operation and verification steps of upgrading Java code fragments to modules, and only requires adding an ark packaging plugin and configuring module slimming to achieve the one-click upgrade of Java code fragments into module applications. It enables the same set of code branches to be independently started like the original Java code fragments, and can also be deployed and started with other applications as a module.

Prerequisites

  • JDK 8
    • sofa.ark.version >= 2.2.14-SNAPSHOT
    • koupleless.runtime.version >= 1.3.1-SNAPSHOT
  • JDK 17/JDK 21
    • sofa.ark.version >= 3.1.7-SNAPSHOT
    • koupleless.runtime.version >= 2.1.6-SNAPSHOT

Integration Steps

Step 1: Add dependencies and packaging plugins required for the module

<properties>
    <sofa.ark.version>${see-prerequisites-above}</sofa.ark.version>
    <!-- Use different koupleless versions for different JDK versions, see: https://koupleless.io/docs/tutorials/module-development/runtime-compatibility-list/#%E6%A1%86%E6%9E%B6%E8%87%AA%E8%BA%AB%E5%90%84%E7%89%88%E6%9C%AC%E5%85%BC%E5%AE%B9%E6%80%A7%E5%85%B3%E7%B3%BB -->
    <koupleless.runtime.version>${see-prerequisites-above}</koupleless.runtime.version>
</properties>

<dependencies>
    <dependency>
        <groupId>com.alipay.sofa.koupleless</groupId>
        <artifactId>koupleless-app-starter</artifactId>
        <version>${koupleless.runtime.version}</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
<plugins>
    <!-- Add the ark packaging plugin here -->
    <plugin>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>sofa-ark-maven-plugin</artifactId>
        <version>{sofa.ark.version}</version>
        <executions>
            <execution>
                <id>default-cli</id>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <skipArkExecutable>true</skipArkExecutable>
            <outputDirectory>./target</outputDirectory>
            <bizName>${replace-with-module-name}</bizName>
            <declaredMode>true</declaredMode>
        </configuration>
    </plugin>
    
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
            <execution>
                <goals>
                    <goal>jar</goal>
                </goals>
                <phase>package</phase>
                <configuration>
                    <classifier>lib</classifier>
                    <!-- Ensure other necessary configuration here -->
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

Step 2: Add initialization logic

Add MainApplication.init() in the code snippet to initialize the container.

public static void main(String[] args) {
        // Initialize the module's instance container
        MainApplication.init();
        // ...
    }

In terms of communication between modules and the base, the module registers instances in the container, and the base obtains module instances through SpringServiceFinder. Using biz3 as an example:

  1. biz3 implements two instances that are based on the AppService interface: Biz3AppServiceImpl and Biz3OtherAppServiceImpl:
public class Biz3OtherAppServiceImpl implements AppService {
    // Get the base bean
    private AppService baseAppService = SpringServiceFinder.getBaseService(AppService.class);
    @Override
    public String getAppName() {
        return "biz3OtherAppServiceImpl in the base: " + baseAppService.getAppName();
    }
}
public class Biz3AppServiceImpl implements AppService {
  // Get the base bean
  private AppService baseAppService = SpringServiceFinder.getBaseService(AppService.class);
  public String getAppName() {
    return "biz3AppServiceImpl in the base: " + baseAppService.getAppName();
  }
}

In which, the module obtains the base bean using: SpringServiceFinder.getBaseService(XXX.class), details can be found in: Module and Base Communication under ‘Module calls the base approach two: programming API SpringServiceFinder’.

  1. biz3 registers instances of these two classes in the container:
public static void main(String[] args) {
        // Initialize the module's instance container
        MainApplication.init();
        // Register instances in the module container
        MainApplication.register("biz3AppServiceImpl", new Biz3AppServiceImpl());
        MainApplication.register("biz3OtherAppServiceImpl", new Biz3OtherAppServiceImpl());
        }
  1. The base obtains instances from biz3:
@RestController
public class SampleController {
    // Get specific instances from biz3 through annotation
    @AutowiredFromBiz(bizName = "biz3", bizVersion = "0.0.1-SNAPSHOT", name = "biz3AppServiceImpl")
    private AppService biz3AppServiceImpl;
    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {
        System.out.println(biz3AppServiceImpl.getAppName());
        // Get specific instances from biz3 through an API
        AppService biz3OtherAppServiceImpl = SpringServiceFinder.getModuleService("biz3", "0.0.1-SNAPSHOT",
                "biz3OtherAppServiceImpl", AppService.class);
        System.out.println(biz3OtherAppServiceImpl.getAppName());
        // Get all instances of AppService class from biz3 through an API
        Map<String, AppService> appServiceMap = SpringServiceFinder.listModuleServices("biz3",
                "0.0.1-SNAPSHOT", AppService.class);
        for (AppService appService : appServiceMap.values()) {
            System.out.println(appService.getAppName());
        }
        return "hello to ark master biz";
    }
}

Where SpringBoot / SOFABoot base can obtain module instances through the @AutowiredFromBiz annotation or SpringServiceFinder.getModuleService() programming API, details can be found in: Module and Base Communication under ‘Base calls module’.

Step 3: Automate module slimming

Typically, module dependencies for code fragments are relatively simple. You can set the scope of dependencies in the module that are consistent with the base to “provided”, or use the automated slimming capability of the ark packaging plugin to automatically slim down the maven dependencies in the module. This step is mandatory, otherwise the module jar package will be very large and will result in startup errors.

Step 4: Build the module into a jar package

Execute mvn clean package -DskipTest, and you can find the packaged ark biz jar in the target directory.

Experiment: Verify the module can be deployed and merged

  1. Start the base from the previous step (verify independent start-up steps)
  2. Initiate module deployment Refer to the sample module deployment of biz3: https://github.com/koupleless/samples/blob/main/springboot-samples/service/README-zh_CN.md

3 - 4.3 Module Development

Koupleless Module Development

3.1 - 4.3.1 Coding Standards

Koupleless Coding Standards

Basic Specifications

  1. The list of middleware clients officially verified and compatible in Koupleless modules can be found here. Any middleware client can be used in the base.

  2. If you need to use System.setProperties() and System.getProperties() in module without sharing with the base, please add MultiBizProperties.initSystem() in the main method of the base platform. For details, refer to samples.
  3. If the module hot unload capability is used, you can use the following API to decorate ExecutorService (typical for various thread pools), Timer, and Thread objects declared in the module code. When the module is unloaded, the Koupleless Arklet client will attempt to automatically clean up the decorated ExecutorService, Timer, and Thread:
    • In the module code, decorate the ExecutorService that needs to be automatically cleaned up. The underlying code will call the shutdownNow and awaitTermination interfaces of the ExecutorService object, attempting to gracefully release threads (not guaranteed to release 100%, such as when threads are waiting). The specific usage is:
      ShutdownExecutorServicesOnUninstallEventHandler.manageExecutorService(myExecutorService);
      Where myExecutorService needs to be a subtype of ExecutorService. You can also configure com.alipay.koupleless.executor.cleanup.timeout.seconds in the module’s SpringBoot or SOFABoot properties file to specify the graceful waiting time for thread pool awaitTermination.

    • In the module code, decorate the Timer that needs to be automatically cleaned up. The underlying code will call the cancel method of the Timer object. The specific usage is:
      CancelTimersOnUninstallEventHandler.manageTimer(myTimer);


    • In the module code, decorate the Thread that needs to be automatically cleaned up. The underlying code will forcibly call the stop method of the Thread object. The specific usage is:
      ForceStopThreadsOnUninstallEventHandler.manageThread(myThread);
      Note: JDK does not recommend forcibly stopping threads, as it may cause unexpected problems such as forcibly releasing locks on threads. Unless you are sure that forcibly closing threads will not cause any related issues, use it with caution.

  4. If the module hot unload capability is used and there are other resources or objects that need to be cleaned up, you can listen for the Spring ContextClosedEvent event and clean up the necessary resources and objects in the event handler function. You can also specify their destroy-method at the place where Beans are defined in Spring XML. When the module is unloaded, Spring will automatically execute the destroy-method.

  5. When the base is started, all modules will be deployed. Therefore, when coding the base, make sure to be compatible with all modules, otherwise the base deployment will fail. If there are incompatible changes that cannot be bypassed (usually there will be many incompatible changes between the base and modules during the module splitting process), please refer to Incompatible Base and Module Upgrade。

Knowledge Points

Module Slimming (Important)
Module-to-Module and Module-to-Base Communication (Important)
Module Testing (Important)
Reuse Base Interceptors in Modules
Reuse Base Data Sources in Modules
Introduction to the Principle of Class Delegation Between Base and Modules Multiple Configurations for Modules



3.2 - 4.3.2 Module Slimming

Koupleless Module Slimming

Why Slimming?

Using the underlying SOFAArk framework, Koupleless achieves class isolation between modules and between modules and the base. When the module starts, it initializes various objects and prioritizes using the module’s class loader to load classes, resources, and JAR files from the FatJar build artifact. Classes that cannot be found will be delegated to the base’s class loader for retrieval.

Based on this class delegation loading mechanism, the common classes, resources, and JAR files shared by the base and modules all sink into the base, allowing the module build artifact to be very small, resulting in very low memory consumption for the module and very fast startup.

Furthermore, after the module starts, many objects will be created in the Spring context. If module hot-unloading is enabled, complete recycling may not be possible, and excessive installations can cause high overhead in the Old generation and Metaspace, triggering frequent FullGC. Therefore, it is necessary to control the size of individual module packages to be < 5MB. In this way, the base can hot deploy and hot unload hundreds of times without replacement or restarting.

The so-called “module slimming” means that the JAR dependencies already present in the base do not participate in the module packaging and construction, thus achieving the two benefits mentioned above:

  • Increase the speed of module installation, reduce module package size, reduce startup dependencies, and control module installation time < 30 seconds, or even < 5 seconds.
  • In the hot deploy and hot unload scenario, the base can hot deploy and hot unload hundreds of times without replacement or restart.

Slimming Principles

The principle of building the ark-biz jar package is to place common packages such as frameworks and middleware in the base as much as possible while ensuring the functionality of the module, and reuse the base packages in the module, making the resulting ark-biz jar more lightweight.

In different scenarios, complex applications can choose different slimming methods.

Scenarios and Corresponding Slimming Methods

Scenario 1: The base and the module have close cooperation, such as the middle platform mode/shared library mode

In the case of close cooperation between the base and modules, the modules should perceive some facade classes of the base and the dependency versions currently used by the base during development, and import the required dependencies as needed. During module packaging, only two types of dependencies should be included: dependencies that the base does not have, and dependencies whose versions are inconsistent with those of the base.

Therefore, the base needs to:

  1. Unified control over module dependency versions to let module developers know which dependencies the base has during development, to mitigate risks, and allow module developers to import part of the dependencies as needed without specifying versions.

The module needs to:

  1. Only include dependencies that are not in the base and dependencies whose versions are inconsistent with those of the base during packaging to reduce the cost of slimming the module

Step 1: Packaging “base-dependencies-starter”

Objective

This step will produce “base-dependencies-starter” for unified control of module dependency versions.

Pom configuration for base bootstrap:

Note: The dependencyArtifactId in the following configuration needs to be modified, generally to ${baseAppName}-dependencies-starter

<build>
<plugins>
    <plugin>
        <groupId>com.alipay.sofa.koupleless</groupId>
        <artifactId>koupleless-base-build-plugin</artifactId>
        <!--        koupleless.runtime.version >= 1.3.0 -->
        <version>${koupleless.runtime.version}</version>
        <configuration>
            <!-- Generate the artifactId of the starter (groupId consistent with the base), which needs to be modified here!! -->
            <dependencyArtifactId>${baseAppName}-dependencies-starter</dependencyArtifactId>
            <!-- Generate the version number of the jar -->
            <dependencyVersion>0.0.1-SNAPSHOT</dependencyVersion>
            <!-- For debugging, change to true to see the intermediate products of the packaging -->
            <cleanAfterPackageDependencies>false</cleanAfterPackageDependencies>
        </configuration>
    </plugin>
  </plugins>
</build>

Local test

  1. Pack the base-dependencies-starter jar: execute the command in the root directory of the base:
mvn com.alipay.sofa.koupleless:koupleless-base-build-plugin::packageDependency -f ${Relative path of the base bootstrap pom to the root directory of the base}

The constructed pom will be in the outputs directory and will be automatically installed in the local Maven repository.

Note, this step will not upload “base-dependencies-starter” to the maven repository. We welcome further discussion to supplement the solution of “uploading to the maven repository”.

Step 2: Module modification packaging plugin and parent

Objective

  1. When developing the module, use the “base-dependencies-starter” from Step 1 as the parent of the module project for unified management of dependency versions;
  2. Modify the module packaging plug-in to only include “dependencies not in the base” and “dependencies whose versions are inconsistent with those of the base” when packaging the module, eliminating the need to manually configure “provided” and achieving automatic slimming of the module.

In addition: For some dependencies, even if the module and base use the same dependency version, the dependency needs to be retained when the module is packaged, i.e., the module slimming dependency whitelist needs to be configured. This feature will be launched at the end of July.

Configure the parent in the module’s root directory pom:

<parent>
   <groupId>com.alipay</groupId>
   <artifactId>${baseAppName}-dependencies-starter</artifactId>
   <version>0.0.1-SNAPSHOT</version>
</parent>

Configure plugin in the module’s packaging pom:

<build>
   <plugins>
       <plugin>
           <groupId>com.alipay.sofa</groupId>
           <artifactId>sofa-ark-maven-plugin</artifactId>
           <!--           since ${sofa.ark.version} >= 2.2.13    -->
           <version>${sofa.ark.version}</version>
           <executions>
               <execution>
                   <id>default-cli</id>
                   <goals>
                       <goal>repackage</goal>
                   </goals>
               </execution>
           </executions>
           <configuration>
               <!-- Configure the identifier of "base-dependencies-starter", standardized as '${groupId}:${artifactId}':'version' -->
               <baseDependencyParentIdentity>com.alipay:${baseAppName}-dependencies-starter:0.0.1-SNAPSHOT</baseDependencyParentIdentity>
           </configuration>
       </plugin>
   </plugins>
</build>

Step 3: Configure Module Dependency Whitelist

For some dependencies, even if the module and base use the same version of the dependency, the dependency needs to be retained when the module is packaged. This requires configuring a module slimming dependency whitelist. This feature will be launched by the end of July.

Step 4: Package Building

Scenario 2: The base and the module have loose cooperation, such as resource saving in multi-application merge deployment

In the case of loose cooperation between the base and the module, the module should not perceive the dependency versions currently used by the base during development, so the module needs to focus more on the low-cost access to module slimming. Dependencies that need to be excluded from module packaging can be configured.

Method 1: SOFAArk Configuration File Combining

Step 1

SOFAArk Module Slimming reads configuration from two places:

  • “Module Project Root Directory/conf/ark/bootstrap.properties”, such as: my-module/conf/ark/bootstrap.properties
  • “Module Project Root Directory/conf/ark/bootstrap.yml”, such as: my-module/conf/ark/bootstrap.yml

Configuration

Configure the common package of frameworks and middleware that need to be sunk to the base in “Module Project Root Directory/conf/ark/bootstrap.properties” in the following format, such as:

# excludes config ${groupId}:{artifactId}:{version}, split by ','
excludes=org.apache.commons:commons-lang3,commons-beanutils:commons-beanutils
# excludeGroupIds config ${groupId}, split by ','
excludeGroupIds=org.springframework
# excludeArtifactIds config ${artifactId}, split by ','
excludeArtifactIds=sofa-ark-spi

Configure the common package of frameworks and middleware that need to be sunk to the base in “Module Project Root Directory/conf/ark/bootstrap.yml” in the following format, such as:

# excludes config ${groupId}:{artifactId}:{version}, split by '-'
# excludeGroupIds config ${groupId}, split by '-'
# excludeArtifactIds config ${artifactId}, split by '-'
excludes:
  - org.apache.commons:commons-lang3
  - commons-beanutils:commons-beanutils
excludeGroupIds:
  - org.springframework
excludeArtifactIds:
  - sofa-ark-spi

Step 2

Upgrade the module packaging plug-in sofa-ark-maven-plugin version >= 2.2.12

    <!-- Plugin 1: Packaging plug-in for sofa-ark biz to package as ark biz jar -->
    <plugin>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>sofa-ark-maven-plugin</artifactId>
        <version>${sofa.ark.version}</version>
        <executions>
            <execution>
                <id>default-cli</id>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <skipArkExecutable>true</skipArkExecutable>
            <outputDirectory>./target</outputDirectory>
            <bizName>biz1</bizName>
            <webContextPath>biz1</webContextPath>
            <declaredMode>true</declaredMode>
        </configuration>
    </plugin>

Step 3

Simply build the module ark-biz jar package, and you will see a significant difference in the size of the slimmed ark-biz jar package.

You can click here to view the complete example project for module slimming.

3.3 - 4.3.3 Module Startup

Module Startup

Module Startup Parameters

Modules can be deployed in two ways: static merged deployment and hot deployment.
Static merged deployment does not support configuration startup parameters. Most of the startup parameters for the module can be placed in the module configuration (application.properties); for example, when configuring the profile: change the startup parameter --spring.profiles.active=dev to spring.profiles.active=true in the application.properties file.
Hot deployment modules support configuration of startup parameters. For example, when using Arklet to install a module via a web request, you can configure startup parameters and environment variables:

curl --location --request POST 'localhost:1238/installBiz' \
--header 'Content-Type: application/json' \
--data '{
    "bizName": "${Module Name}",
    "bizVersion": "${Module Version}",
    "bizUrl": "file:///path/to/ark/biz/jar/target/xx-xxxx-ark-biz.jar",
    "args": ["--spring.profiles.active=dev"],
    "env": {
        "XXX": "YYY"
    }
}'

Module Startup Acceleration

Design Concept for Module Startup Acceleration

The overall idea for module startup acceleration is:

  1. The base platform should start the services in advance, which only requires the base platform to pre-import the dependencies.
  2. The module can reuse the base platform’s services in various ways. The methods for reusing the base services include, but are not limited to, analyzing the specific use case; if there are any questions, feel free to discuss in the community group:
    1. Reuse through sharing class static variables.
    2. Reuse by having the base platform encapsulate some service interface APIs, allowing the module to call these APIs directly.
    3. Obtain proxy objects of base platform objects through annotations, using tools provided by Koupleless like @AutowiredFromBase, @AutowiredFromBiz, SpringServiceFinder, and some annotations supporting JVM service calls provided by Dubbo or SOFARpc.
    4. Find objects across modules to directly obtain base platform objects, using tools like SpringBeanFinder provided by Koupleless. There is an implicit issue here: for modules to successfully invoke base platform services, they need to use certain model classes. Therefore, modules typically need to import the dependencies corresponding to those services, leading to these service configurations being scanned during module startup, which may result in reinitializing these services. This can cause unnecessary services to start and slow down the module startup, increasing memory consumption. Thus, to accelerate module startup, three tasks must be completed:
  3. The base platform should start the services in advance.
  4. The module should prohibit the startup of these services, which is the focus of this article.
  5. The module should reuse base platform services.

How Modules Can Prohibit Startup of Certain Services

Starting from version 1.1.0, Koupleless provides the following configuration capability:

koupleless.module.autoconfigure.exclude # Services that do not need to start during module startup
koupleless.module.autoconfigure.include # Services that need to start during module startup. If a service is configured with both include and exclude, the service will start.

This configuration can be set in the base platform or in the module. If configured in the base platform, it applies to all modules. If configured in the module, it only applies to that module and will override the configuration in the base platform.

Benchmark

Detailed benchmark information is yet to be added.

3.4 - 4.3.4 Module Communication Module to Module and Module to Base Communication

Koupleless Module Communication Module to Module and Module to Base Communication

Between the base and modules, and among modules, there is spring context isolation, meaning their beans do not conflict and are not visible to each other. However, in many scenarios such as the middleware mode and independent module mode, there are situations where the base calls the module, the module calls the base, and modules call each other. Currently, three methods are supported for invocation: @AutowiredFromBiz, @AutowiredFromBase, and SpringServiceFinder method calls. Note that the usage of these three methods varies.

Spring Environment

Importing Dependencies in Modules

<dependency>
    <groupId>com.alipay.koupleless</groupId>
    <artifactId>koupleless-app-starter</artifactId>
    <version>0.5.6</version>
    <scope>provided</scope>
</dependency>

Base Calling Module

Only SpringServiceFinder can be used.

@RestController
public class SampleController {

    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {

        Provider studentProvider = SpringServiceFinder.getModuleService("biz", "0.0.1-SNAPSHOT",
                "studentProvider", Provider.class);
        Result result = studentProvider.provide(new Param());

        Provider teacherProvider = SpringServiceFinder.getModuleService("biz", "0.0.1-SNAPSHOT",
                "teacherProvider", Provider.class);
        Result result1 = teacherProvider.provide(new Param());
        
        Map<String, Provider> providerMap = SpringServiceFinder.listModuleServices("biz", "0.0.1-SNAPSHOT",
                Provider.class);
        for (String beanName : providerMap.keySet()) {
            Result result2 = providerMap.get(beanName).provide(new Param());
        }

        return "hello to ark master biz";
    }
}

Module Calling Base

Method 1: Annotation @AutowiredFromBase

@RestController
public class SampleController {

    @AutowiredFromBase(name = "sampleServiceImplNew")
    private SampleService sampleServiceImplNew;

    @AutowiredFromBase(name = "sampleServiceImpl")
    private SampleService sampleServiceImpl;

    @AutowiredFromBase
    private List<SampleService> sampleServiceList;

    @AutowiredFromBase
    private Map<String, SampleService> sampleServiceMap;

    @AutowiredFromBase
    private AppService appService;

    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {

        sampleServiceImplNew.service();

        sampleServiceImpl.service();

        for (SampleService sampleService : sampleServiceList) {
            sampleService.service();
        }

        for (String beanName : sampleServiceMap.keySet()) {
            sampleServiceMap.get(beanName).service();
        }

        appService.getAppName();

        return "hello to ark2 dynamic deploy";
    }
}

Method 2: Programming API SpringServiceFinder

@RestController
public class SampleController {

    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {

        SampleService sampleServiceImplFromFinder = SpringServiceFinder.getBaseService("sampleServiceImpl", SampleService.class);
        String result = sampleServiceImplFromFinder.service();
        System.out.println(result);

        Map<String, SampleService> sampleServiceMapFromFinder = SpringServiceFinder.listBaseServices(SampleService.class);
        for (String beanName : sampleServiceMapFromFinder.keySet()) {
            String result1 = sampleServiceMapFromFinder.get(beanName).service();
            System.out.println(result1);
        }

        return "hello to ark2 dynamic deploy";
    }
}

Module Calling Module

Referencing the module calling the base, the annotation is used with @AutowiredFromBiz and the programming API is supported by SpringServiceFinder.

Method 1: Annotation @AutowiredFromBiz

@RestController
public class SampleController {

    @AutowiredFromBiz(bizName = "biz", bizVersion = "0.0.1-SNAPSHOT", name = "studentProvider")
    private Provider studentProvider;

    @AutowiredFromBiz(bizName = "biz", name = "teacherProvider")
    private Provider teacherProvider;

    @AutowiredFromBiz(bizName = "biz", bizVersion = "0.0.1-SNAPSHOT")
    private List<Provider> providers;

    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {

        Result provide = studentProvider.provide(new Param());

        Result provide1 = teacherProvider.provide(new Param());

        for (Provider provider : providers) {
            Result provide2 = provider.provide(new Param());
        }

        return "hello to ark2 dynamic deploy";
    }
}

Method 2: Programming API SpringServiceFinder

@RestController
public class SampleController {

    @RequestMapping(value = "/", method = RequestMethod.GET)
    public String hello() {

        Provider teacherProvider1 = SpringServiceFinder.getModuleService("biz", "0.0.1-SNAPSHOT", "teacherProvider", Provider.class);
        Result result1 = teacherProvider1.provide(new Param());

        Map<String, Provider> providerMap = SpringServiceFinder.listModuleServices("biz", "0.0.1-SNAPSHOT", Provider.class);
        for (String beanName : providerMap.keySet()) {
            Result result2 = providerMap.get(beanName).provide(new Param());
        }

        return "hello to ark2 dynamic deploy";
    }
}

Complete Example

SOFABoot Environment

Please refer to this documentation



3.5 - 4.3.5 Module Local Development and Debugging

Local development and debugging of the Koupleless module

Arkctl Tool Installation

The Arkctl module installation mainly provides automated packaging and deployment capabilities, including invoking the mvn command to automatically build the module as a JAR file and calling the API interface provided by Arklet for completion of deployment. The installation method for Arkctl can refer to the documentation: arkctl Installation in the Local Environment Development Verification section.

Installation Method 1: Using the Golang Toolchain

  1. Download the corresponding version of Golang from the Golang official website; the version must be above 1.21.
  2. Execute the command go install github.com/koupleless/arkctl@v0.2.1 to install the Arkctl tool.

Installation Method 2: Downloading Binary Files

  1. Download Arkctl based on the actual operating system. Download Arkctl.
  2. Unzip the corresponding binary file and place it in a directory that is included in the system’s PATH variable.
  3. After the base and module have been modified and the base has been started, the Arkctl tool can be used to quickly complete the build and deployment of the module into the base.
     

How to Find the PATH Value on Linux/Mac?

Execute in the terminal:

echo $PATH  
# Choose a directory and place arkctl in that directory

How to Find the PATH Value on Windows?

Press Windows + R, type cmd, and then press Enter to open the command prompt. In the command prompt window, enter the following command and press Enter:

echo %PATH%

Note: In the Windows environment, if Windows Defender is enabled, it may falsely report issues when downloading binaries through the browser, as shown below:


You can refer to the [Go official documentation](https://go.dev/doc/faq#virus) for the reason behind the error. This error can be ignored; feel free to download. > Since Arkctl deployment is actually completed by calling the API, if you prefer not to use the command-line tool, you can directly use the Arklet [API interface](/docs/contribution-guidelines/arklet/architecture) to complete the deployment operation. We also provide a telnet method for module deployment; [detailed instructions can be found here](https://www.sofastack.tech/projects/sofa-boot/sofa-ark-ark-telnet/).

Local Quick Deployment

You can use the Arkctl tool to quickly build and deploy modules, improving the efficiency of local debugging and development.

Scenario 1: Building a Module JAR and Deploying to a Locally Running Base.

Preparation:

  1. Start a base locally.
  2. Open a module project repository.
    Execute the command:
# This needs to be executed in the root directory of the repository.  
# For example, if it is a Maven project, execute it in the directory where the root pom.xml is located.  
arkctl deploy

Once the command completes, it is successfully deployed, and the user can debug and validate the relevant module functionalities.

Scenario 2: Deploying a Locally Built JAR to a Locally Running Base.

Preparation:

  1. Start a base locally.
  2. Prepare a built JAR file.
    Execute the command:
arkctl deploy /path/to/your/pre/built/bundle-biz.jar

Once the command completes, it is successfully deployed, and the user can debug and validate the relevant module functionalities.

Scenario 3: Deploying a Locally Unbuilt JAR to a Locally Running Base.

Preparation:

  1. Start a base locally.
    Execute the command:
arkctl deploy ./path/to/your/biz/

Note: This command is applicable if the module can be built independently (e.g., if commands like mvn package can be successfully executed in the biz directory), the command will automatically build the module and deploy it to the base.

Scenario 4: Building and Deploying Submodule JARs in a Multi-Module Maven Project from the Root.

Preparation:

  1. Start a base locally.
  2. Open a multi-module Maven project repository.
    Execute the command:
# This needs to be executed in the root directory of the repository.  
# For example, if it is a Maven project, execute it in the directory where the root pom.xml is located.  
arkctl deploy --sub ./path/to/your/sub/module

Once the command completes, it is successfully deployed, and the user can debug and validate the relevant module functionalities.

Scenario 5: Building a Module JAR and Deploying to a Remote Running K8s Base.

Preparation:

  1. Ensure that a base pod is already running remotely.
  2. Open a module project repository.
  3. You must have a K8s certificate with exec permissions and the kubectl command-line tool available locally.
    Execute the command:
# This needs to be executed in the root directory of the repository.  
# For example, if it is a Maven project, execute it in the directory where the root pom.xml is located.  
arkctl deploy --pod {namespace}/{podName}

Once the command completes, it is successfully deployed, and the user can debug and validate the relevant module functionalities.

Scenario 6: How to Use This Command More Quickly

You can create a Shell Script in IDEA, set the running directory, and then enter the corresponding Arkctl command as shown in the image below.

Local Module Debugging

Module and Base in the Same IDEA Project

Since the IDEA project can see the module code, debugging the module is no different from normal debugging. Just set breakpoints in the module code and start the base in debug mode.

Module and Base in Different IDEA Projects

  1. Add the debug configuration to the base startup parameters: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000, then start the base.
  2. Add remote JVM debug to the module, setting host to localhost: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000.
  3. Set breakpoints in the module.
  4. After installing the module, you can begin debugging.

Checking Deployment Status

Scenario 1: Querying Modules Already Deployed in the Current Base.

Preparation:

  1. Start a base locally.
    Execute the command:
arkctl status

Scenario 2: Querying Modules Already Deployed in the Remote K8s Environment Base.

Preparation:

  1. Start a base in the remote K8s environment.
  2. Ensure you have Kube certificates and the necessary permissions locally.
    Execute the command:
arkctl status --pod {namespace}/{name}

Viewing Runtime Module Status and Information Using Arthas

Retrieve All Biz Information

vmtool -x 1 --action getInstances --className com.alipay.sofa.ark.container.model.BizModel --limit 100

For example:

image.png

Retrieve Specific Biz Information

# Please replace ${bizName}  
vmtool -x 1 --action getInstances --className com.alipay.sofa.ark.container.model.BizModel --limit 100 | grep ${bizName} -A 4

For example:

image.png

Retrieve Biz Information Corresponding to a Specific BizClassLoader

# Please replace ${BizClassLoaderHashCode}  
vmtool -x 1 --action getInstances --className com.alipay.sofa.ark.container.model.BizModel --limit 100 | grep ${BizClassLoaderHashCode} -B 1 -A 3

For example:

image.png

3.6 - 4.3.6 Reusing Base Data Source

Koupleless Module Reusing Base Data Source

Recommendation

It is highly recommended to use the approach outlined in this document to reuse the base data source within the module whenever possible. Failing to do so may result in repeated creation and consumption of data source connections during module deployments, leading to slower module publishing and operations, as well as increased memory usage.

SpringBoot Solution

Simply create a MybatisConfig class in the module’s code. This way, the transaction template is reused from the base, and only the Mybatis SqlSessionFactoryBean needs to be newly created. Refer to the demo: /koupleless/samples/springboot-samples/db/mybatis/biz1

Use SpringBeanFinder.getBaseBean to obtain the base Bean object, and then register it as the module’s Bean:


@Configuration
@MapperScan(basePackages = "com.alipay.sofa.biz1.mapper", sqlSessionFactoryRef = "mysqlSqlFactory")
@EnableTransactionManagement
public class MybatisConfig {

    // Note: Do not initialize a base DataSource, as it will be destroyed when the module is uninstalled. 
    // However, resources such as transactionManager, transactionTemplate, and mysqlSqlFactory can be safely destroyed.

    @Bean(name = "transactionManager")
    public PlatformTransactionManager platformTransactionManager() {
        return (PlatformTransactionManager) getBaseBean("transactionManager");
    }

    @Bean(name = "transactionTemplate")
    public TransactionTemplate transactionTemplate() {
        return (TransactionTemplate) getBaseBean("transactionTemplate");
    }

    @Bean(name = "mysqlSqlFactory")
    public SqlSessionFactoryBean mysqlSqlFactory() throws IOException {
        // The data source cannot be declared as a bean in the module's Spring context, as it will be closed when the module is uninstalled.

        DataSource dataSource = (DataSource) getBaseBean("dataSource");
        SqlSessionFactoryBean mysqlSqlFactory = new SqlSessionFactoryBean();
        mysqlSqlFactory.setDataSource(dataSource);
        mysqlSqlFactory.setMapperLocations(new PathMatchingResourcePatternResolver()
                .getResources("classpath:mappers/*.xml"));
        return mysqlSqlFactory;
    }
}

SOFABoot Solution

If the SOFABoot base does not enable multi-bundle (there is no MANIFEST.MF file in the Package), the solution is identical to the SpringBoot solution mentioned above. If there is a MANIFEST.MF file, you need to call BaseAppUtils.getBeanOfBundle to obtain the base Bean, where BASE_DAL_BUNDLE_NAME is theModule-Name in the MANIFEST.MF file:
image.png


@Configuration
@MapperScan(basePackages = "com.alipay.koupleless.dal.dao", sqlSessionFactoryRef = "mysqlSqlFactory")
@EnableTransactionManagement
public class MybatisConfig {

    // Note: Do not initialize a base DataSource, as it will be destroyed when the module is uninstalled. 
    // However, resources such as transactionManager, transactionTemplate, and mysqlSqlFactory can be safely destroyed

    private static final String BASE_DAL_BUNDLE_NAME = "com.alipay.koupleless.dal"

    @Bean(name = "transactionManager")
    public PlatformTransactionManager platformTransactionManager() {
        return (PlatformTransactionManager) BaseAppUtils.getBeanOfBundle("transactionManager",BASE_DAL_BUNDLE_NAME);
    }

    @Bean(name = "transactionTemplate")
    public TransactionTemplate transactionTemplate() {
        return (TransactionTemplate) BaseAppUtils.getBeanOfBundle("transactionTemplate",BASE_DAL_BUNDLE_NAME);
    }

    @Bean(name = "mysqlSqlFactory")
    public SqlSessionFactoryBean mysqlSqlFactory() throws IOException {
        // The data source cannot be declared as a bean in the module's Spring context, as it will be closed when the module is uninstalled.
        ZdalDataSource dataSource = (ZdalDataSource) BaseAppUtils.getBeanOfBundle("dataSource",BASE_DAL_BUNDLE_NAME);
        SqlSessionFactoryBean mysqlSqlFactory = new SqlSessionFactoryBean();
        mysqlSqlFactory.setDataSource(dataSource);
        mysqlSqlFactory.setMapperLocations(new PathMatchingResourcePatternResolver()
                .getResources("classpath:mapper/*.xml"));
        return mysqlSqlFactory;
    }
}


3.7 - 4.3.7 Reusing Base Interceptors

Koupleless Module Reusing Base Interceptors

Objective

In the base, many Aspect interceptors (Spring interceptors) are defined, and you may want to reuse them in the module. However, the Spring contexts of the module and the base are isolated, which means that Aspect interceptors will not take effect in the module.

Solution

Create a proxy object for the original interceptor class, allowing the module to invoke this proxy object. Then, the module initializes this proxy object through the AutoConfiguration annotation. The complete steps and example code are as follows:

Step 1:

The base code defines an interface that defines the execution method of the interceptor. This interface needs to be visible to the module (referenced in the module dependencies):

public interface AnnotionService {
    Object doAround(ProceedingJoinPoint joinPoint) throws Throwable;
}

Step 2:

Write the specific implementation of the interceptor in the base. This implementation class needs to be annotated with @SofaService (SOFABoot) or @SpringService (SpringBoot, under construction):

@Service
@SofaService(uniqueId = "facadeAroundHandler")
public class FacadeAroundHandler implements AnnotionService {

    private final static Logger LOG = LoggerConst.MY_LOGGER;

    public Object doAround(ProceedingJoinPoint joinPoint) throws Throwable {
        log.info("Start execution")
        joinPoint.proceed();
        log.info("Execution completed")
    }
}

Step 3:

In the module, use the @Aspect annotation to implement an Aspect. SOFABoot injects the FacadeAroundHandler on the base via @SofaReference. If it is SpringBoot, then use @AutowiredFromBase to inject FacadeAroundHandler on the base.

Note: Do not declare this as a bean, do not add @Component or @Service annotation, only @Aspect annotation is needed.

// Note: Do not declare this as a bean, do not add @Component or @Service annotation
@Aspect
public class FacadeAroundAspect {

    // If it is SOFABoot, use @SofaReference; if it is SpringBoot, use @AutowiredFromBase.
    @SofaReference(uniqueId = "facadeAroundHandler")
    //@AutowiredFromBase
    private AnnotionService facadeAroundHandler;

    @Pointcut("@annotation(com.alipay.linglongmng.presentation.mvc.interceptor.FacadeAround)")
    public void facadeAroundPointcut() {
    }

    @Around("facadeAroundPointcut()")
    public Object doAround(ProceedingJoinPoint joinPoint) throws Throwable {
        return facadeAroundHandler.doAround(joinPoint);
    }
}

Step 4:

Use the @Configuration annotation to create a Configuration class, and declare the aspectj objects needed by the module as Spring Beans.
Note: This Configuration class needs to be visible to the module, and related Spring Jar dependencies need to be imported with provided.

@Configuration
public class MngAspectConfiguration {
    @Bean
    public FacadeAroundAspect facadeAroundAspect() {
        return new FacadeAroundAspect();
    }
    @Bean
    public EnvRouteAspect envRouteAspect() {
        return new EnvRouteAspect();
    }
    @Bean
    public FacadeAroundAspect facadeAroundAspect() {
        return new FacadeAroundAspect();
    }
    
}

Step 5:

Explicitly depend on the Configuration class MngAspectConfiguration created in step 4 in the module code.

@SpringBootApplication
@ImportResource("classpath*:META-INF/spring/*.xml")
@ImportAutoConfiguration(value = {MngAspectConfiguration.class})
public class ModuleBootstrapApplication {
    public static void main(String[] args) {
        SpringApplicationBuilder builder = new SpringApplicationBuilder(ModuleBootstrapApplication.class)
        	.web(WebApplicationType.NONE);
        builder.build().run(args);
    }
}


3.8 - 4.3.8 Thread Pool Usage

Koupleless Thread Pool Usage

Background

When multiple modules or a module and a base share the same thread pool, the Classloader used by the thread executing a task in the thread pool may differ from the Classloader that was used when the task was created. This can lead to a ClassNotFoundException when the thread pool executes the task.

As a result, when multiple modules or a module and a base share the same thread pool, in order to ensure consistency between the Classloader used during task execution and the Classloader used at the creation of the task, we need to make some modifications to the thread pool.

⚠️Note: There will be no such issue if each module uses its own thread pool.

There are 4 common ways to use thread pools in Java:

  1. Directly create thread tasks and submit them to the thread pool, such as: Runnable, Callable, ForkJoinTask
  2. Customize ThreadPoolExecutor and submit tasks to ThreadPoolExecutor
  3. Use ThreadPoolExecutor or ScheduledThreadPoolExecutor from the third-party libraries.
  4. Create thread pools through Executors and submit tasks to ExecutorService, ScheduledExecutorService, ForkJoinPool
  5. For SpringBoot users, submit tasks to ThreadPoolTaskExecutor, SchedulerThreadPoolTaskExecutor

This article will introduce how each method is used on Koupleless.

How to Use

1. Directly create thread tasks and submit them to the thread pool

The original method:


threadPool.execute(new Runnable(){
    public void run() {
        //do something
    }
});


threadPool.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

If the threadPool remains unchanged, then it is necessary to wrap Runnable as KouplelessRunnable and Callable as KouplelessCallable, as follows:

// Runnable
// wrap function:
threadPool.execute(KouplelessRunnable.wrap(new Runnable(){
    public void run() {
        //do something
    }
});

// or new KouplelessRunnable:
threadPool.execute(new KouplelessRunnable(){
    public void run() {
        //do something
    }
});

// Runnable
// wrap function:
threadPool.execute(KouplelessCallable.wrap(new Callable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

// or new KouplelessRunnable
threadPool.execute(new KouplelessCallable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

2. Customize ThreadPoolExecutor

The original method:

ThreadPoolExecutor threadPool = new ThreadPoolExecutor(5, 5, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());

threadPool.execute(new Runnable(){
    public void run() {
        //do something
    }
});


threadPool.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

To keep Runnable and Callable unchanged, there are two ways to modify:

  1. Change threadPool to KouplelessThreadPoolExecutor
  2. Or use kouplelessExecutorService.

First, let’s take an example of the first modification method: change threadPool to KouplelessThreadPoolExecutor. As follows:

// modify ThreadPoolExecutor as KouplelessThreadPoolExecutor
ThreadPoolExecutor threadPool = new KouplelessThreadPoolExecutor(5, 5, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());

threadPool.execute(new Runnable(){
    public void run() {
        //do something
    }
});


threadPool.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

Then, illustrate the second method of modification: using kouplelessExecutorService. As follows:

// use kouplelessExecutorService
ExecutorService executor        = new KouplelessExecutorService(new ThreadPoolExecutor(5, 5, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>()));

// use executor to execute task
executor.execute(new Runnable(){
    public void run() {
        //do something
    }
});
executor.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

3. Use ThreadPoolExecutor or ScheduledThreadPoolExecutor from the third-party libraries.

The original method:

ThreadPoolExecutorA executorService = new ThreadPoolExecutorA();

executorService.execute(new Runnable(){
    public void run() {
        //do something
    }
});


executorService.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

ScheduledThreadPoolExecutorA scheduledExecutorService = new ScheduledThreadPoolExecutorA();

scheduledExecutorService.execute(new Runnable(){
    public void run() {
        //do something
    }
});

scheduledExecutorService.execute(new Callable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

To keep Runnable and Callable unchanged, it is necessary to use kouplelessExecutorService and kouplelessScheduledExecutorService, as follows:

// use KouplelessExecutorService
        ExecutorService executor        = new KouplelessExecutorService(new ThreadPoolExecutorA());

// use executor to execute tasks
executor.execute(new Runnable(){
    public void run() {
        //do something
    }
});
executor.execute(new Callable<String>(){
    public String call() {
        //do something
    return "mock";
    }
});

// use scheduledExecutorService 
ScheduledExecutorService scheduledExecutor = new KouplelessScheduledExecutorService(new ScheduledThreadPoolExecutorA());

// use scheduledExecutor to execute tasks
scheduledExecutor.execute(new Runnable(){
    public void run() {
        //do something
    }
});
scheduledExecutor.execute(new Callable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

4. Create thread pools through Executors

The original method:

ExecutorService executorService = Executors.newFixedThreadPool(6);

executorService.execute(new Runnable(){
    public void run() {
        //do something
    }
});


executorService.execute(new Callable<String>(){
public String call() {
        //do something
        return "mock";
 }
});

ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor();

scheduledExecutorService.execute(new Runnable(){
    public void run() {
        //do something
    }
});

scheduledExecutorService.execute(new Callable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

To keep Runnable and Callable unchanged, it is necessary to use kouplelessExecutorService and kouplelessScheduledExecutorService, as follows:

// use KouplelessExecutorService
ExecutorService executor        = new KouplelessExecutorService(Executors.newFixedThreadPool(6));

// use executor to execute tasks
executor.execute(new Runnable(){
    public void run() {
        //do something
    }
});
executor.execute(new Callable<String>(){
    public String call() {
        //do something
    return "mock";
    }
});

// use KouplelessScheduledExecutorService
ScheduledExecutorService scheduledExecutor = new KouplelessScheduledExecutorService(Executors.newSingleThreadScheduledExecutor());

// use scheduledExecutor to execute tasks
scheduledExecutor.execute(new Runnable(){
    public void run() {
        //do something
    }
});
scheduledExecutor.execute(new Callable<String>(){
    public String call() {
        //do something
        return "mock";
    }
});

5. For SpringBoot users, submit tasks to ThreadPoolTaskExecutor, SchedulerThreadPoolTaskExecutor

Due to koupeless having already adapted ThreadPoolTaskExecutor and SchedulerThreadPoolTaskExecutor for springboot (2.3.0-2.7.x), they can be used directly.

@Autowired
private ThreadPoolTaskExecutor threadPoolTaskExecutor;

@Autowired
private SchedulerThreadPoolTaskExecutor schedulerThreadPoolTaskExecutor;

threadPoolTaskExecutor.execute(new Runnable(){
    public void run() {
        //do something
    }
});

schedulerThreadPoolTaskExecutor.execute(new Runnable(){
    public void run() {
        //do something
    }
});

3.9 - 4.3.9 Multiple Configurations for Modules

Why Use Multiple Configurations

In different scenarios, a module’s code may be deployed to different applications but require different configurations.

How to Use

Step 1: When packaging a module’s code for different scenarios, configure different bizName, such as biz1, biz2.

<plugin>
    <groupId>com.alipay.sofa</groupId>
    <artifactId>sofa-ark-maven-plugin</artifactId>
    <version>${sofa.ark.version}</version>
    <executions>
        <execution>
            <id>default-cli</id>
            <goals>
                <goal>repackage</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <!-- Configure different bizName for different scenarios, such as biz1, biz2 -->
        <bizName>biz1</bizName>
        <!-- ... Other properties -->
    </configuration>
</plugin>

Step 2: In the resources directory of the module, add the following files, where config , biz1 and biz2 are folders:

  • config/biz1/application.properties
  • config/biz2/application.properties Step 3: Package two different ark-biz files with different bizName values (biz1, biz2):
  • biz1-0.0.1-SNAPSHOT-ark-biz.jar
  • biz2-0.0.1-SNAPSHOT-ark-biz.jar Step 4: Install the corresponding ark-biz module for different scenarios. When the module starts, it will read the configuration files based on the bizName value:
  • config/biz1/application.properties
  • config/biz2/application.properties

Principle

When the module starts, it reads the following files as property sources based on the module name and spring.profiles.active field:

  • config/${bizName}/application-${profile}.properties
  • config/${bizName}/application.properties If spring.profiles.active is not set, it reads the following file as the property source:
  • config/${bizName}/application.properties

3.10 - 4.3.10 Multimodule Integration Testing

[English](./README.md) | 简体中文

Why Do We Need an Integration Testing Framework?

Without an integration testing framework, the validation steps for developers when verifying koupleless module logic can be cumbersome and involve the following steps:

  1. Start a base process.
  2. Build the module JAR package.
  3. Install the module.
  4. Call the module’s HTTP interface (or other methods) to validate the logic. If the logic does not meet expectations, developers need to repeat the above steps, making such a validation process highly inefficient. To improve the validation efficiency for developers, we decided to provide the koupleless integration testing framework, allowing developers to start both the base and the module within a single process.

Integration Testing Framework

Principle

The integration testing framework simulates a multi-module deployment scenario by enhancing the class loading behavior of the base and the modules. The specific source code can be referenced in koupleless-test-suite.

How to Use

Taking webflux-samples as an example, the project structure of webflux-samples is as follows:

We create a new Maven module:

First, this module needs to add the integration testing framework dependency:

<dependency>
    <groupId>com.alipay.sofa.koupleless</groupId>
    <artifactId>koupleless-test-suite</artifactId>
    <version>${koupleless.runtime.version}</version>
</dependency>

Next, we need to add the dependencies for the base and the module:

<!-- Base Dependency -->
<dependency>
    <groupId>com.alipay.sofa.web.webflux</groupId>
    <artifactId>demowebflux</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <classifier>lib</classifier>
</dependency>
<!-- Module Dependency -->
<dependency>
    <groupId>com.alipay.sofa.web.webflux</groupId>
    <artifactId>bizwebflux</artifactId>
    <version>0.0.1-SNAPSHOT</version>
</dependency>

Then, we need to write the integration test case:

public static void setUp() {
    TestMultiSpringApplication multiApp = new TestMultiSpringApplication(
            MultiSpringTestConfig
                    .builder()
                    .baseConfig(
                            BaseSpringTestConfig
                                    .builder()
                                    // Pass in the base application's startup class.
                                    .mainClass(DemoWebfluxApplication.class)
                                    .build()
                    )
                    .bizConfigs(
                            Lists.newArrayList(
                                    BizSpringTestConfig
                                            .builder()
                                            .bizName("biz")
                                            // Pass in the module's startup class.
                                            .mainClass(BizWebfluxApplication.class)
                                            .build()))
                    .build());
    multiApp.run();
}

Finally, by starting the tests in IDEA, we will find that both the base and module’s Spring containers are up and running. This allows us to validate the multi-module logic within a single process.

Thus, we have completed an integration test case.

Summary

Through the above experiment, we have validated that the koupleless integration testing framework can quickly verify multi-module logic, improving developers’ validation efficiency.

3.11 - 4.3.11 Static Merge Deployment

Static Merge Deployment of Koupleless Module

Introduction

SOFAArk provides the capability of static merge deployment, where the Base package (foundation application) can start already constructed Biz package (module application) during startup. The default way of obtaining the module is through local directory, local file URL, and remote URL.

In addition, SOFAArk also provides an extension interface for static merge deployment, where developers can customize the way of obtaining the Biz package (module application).

Usage

Step 1: Package Module Application into Ark Biz

If developers wish for their application’s Ark Biz package to be used as a Jar package dependency by other applications, running on the same SOFAArk container, they need to package and publish the Ark Biz package. For details, see Ark Biz Introduction. The Ark Biz package is generated using the Maven plugin sofa-ark-maven-plugin.

<build>
    <plugin>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>sofa-ark-maven-plugin</artifactId>
        <version>${sofa.ark.version}</version>
        <executions>
            <execution>
                <id>default-cli</id>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <!-- Default is 100, larger values indicate later installation, with Koupleless runtime version greater than or equal to 1.2.2 -->
            <priority>200</priority>
        </configuration>
    </plugin>
</build>

Step 2: Base Fetching Ark Biz for Merge Deployment

Requirements:

  • JDK8
    • sofa.ark.version >= 2.2.12
    • koupleless.runtime.version >= 1.2.3
  • JDK17/JDK21
    • sofa.ark.version >= 3.1.5
    • koupleless.runtime.version >= 2.1.4

Method 1: Using Official Default Retrieval Method, Supporting Local Directory, Local File URL, Remote URL

1. Base Configuration of Local Directory, Local File URL, Remote URL

Developers need to specify the Ark Biz package that needs to be merged and deployed in the base’s ark configuration file (conf/ark/ark.properties or conf/ark/ark.yml), supporting:

  • Local directory
  • Local file URL (windows system as file:\\, linux system as file://)
  • Remote URL (supporting http://,https://) In integrateBizURLs field for local file URL and remote URL, and integrateLocalDirs field for local directory. The configuration is as follows:
integrateBizURLs=file://${xxx}/koupleless_samples/springboot-samples/service/biz1/biz1-bootstrap/target/biz1-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar,\
  file://${xxx}/koupleless_samples/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar,\
  https://oss.xxxxx/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
integrateLocalDirs=/home/${xxx}/sofa-ark/biz,\
  /home/${xxx}/sofa-ark/biz2

or

integrateBizURLs:
  - file://${xxx}/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
  - file://${xxx}/koupleless_samples/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
integrateLocalDirs:
  - /home/${xxx}/sofa-ark/biz
  - /home/${xxx}/sofa-ark/biz2
2. Base Configuration of Packaged Plugin Target integrate-biz

Add the integrate-biz to koupleless-base-build-plugin in the base’s bootstrap pom, as shown below:

<plugin>
    <groupId>com.alipay.sofa.koupleless</groupId>
    <artifactId>koupleless-base-build-plugin</artifactId>
    <version>${koupleless.runtime.version}</version>
    <executions>
        <execution>
            <goals>
                <goal>add-patch</goal>
                <!-- Used for static merge deployment -->
                <goal>integrate-biz</goal>
            </goals>
        </execution>
    </executions>
</plugin>

After the build, if the packaged jar file is manually unpacked, the specified module ark-biz package can be seen in classPath/SOFA-ARK/biz.

Method 2: Using Custom Retrieval Method

1. Ark Extension Mechanism Principle

Refer to Ark Extension Mechanism Introduction

2. Implement AddBizToStaticDeployHook Interface

In the base/third-party package, implement the AddBizToStaticDeployHook interface, using AddBizInResourcesHook as an example, as shown below:

@Extension("add-biz-in-resources-to-deploy")
public class AddBizInResourcesHook implements AddBizToStaticDeployHook {
    @Override
    public List<BizArchive> getStaticBizToAdd() throws Exception {
        List<BizArchive> archives = new ArrayList<>();
        // ...
        archives.addAll(getBizArchiveFromResources());
        return archives;
    }
    protected List<BizArchive> getBizArchiveFromResources() throws Exception {
        // ... Read Ark Biz package in resources
        return archives;
    }
}
3. Configure SPI

Add the /META-INF/services/sofa-ark/ directory in the resources directory, then add a file named com.alipay.sofa.ark.spi.service.biz.AddBizToStaticDeployHook in /META-INF/services/sofa-ark/ directory, where the file contains the fully qualified name of the hook class:

com.alipay.sofa.ark.support.common.AddBizInResourcesHook

Rebuild the base.

Step 3: Start the Base

Add the JVM parameter configuration: -Dsofa.ark.embed.static.biz.enable=true

3.12 - 4.3.12 Officially Supported Middleware Clients in Modules

Koupleless Module Officially Supported Middleware Clients

Compatibility Relationships Across Different Versions of the Framework

Users can choose to import the Koupleless version as needed, based on actual JDK and SpringBoot versions.

JDKSpringBootSOFA-ARKKoupleless
1.82.x2.x.x1.x.x
173.0.x, 3.1.x3.0.7 (不再更新)2.0.4（不再更新）
17 & 213.2.x and above3.1.x2.1.x

For Koupleless SDK latest versions, please refer to https://github.com/koupleless/runtime/releases

In Koupleless modules, the official support currently includes and is compatible with common middleware clients.
Note: Here, “already supported” needs to be included in the base POM by importing the relevant client dependencies (strongly recommended to use the SpringBoot Starter method to import the dependencies), and also in the module POM by importing the relevant dependencies and setting <scope>provided</scope> to delegate the dependencies to the base for loading.


Compatibility Report for Various Components

Middleware ClientVersionRemarks
JDK8.x
17.x		Already Supported
SpringBoot>= 2.3.0 or 3.xAlready Supported
Base and module complete usage examples for JDK17 + SpringBoot3.x can be seen here
SpringBoot Cloud>= 2.7.xAlready Supported
Complete usage examples for base and modules can be seen here
SOFABoot>= 3.9.0 or 4.xAlready Supported
JMXN/AAlready Supported
Requires adding the -Dspring.jmx.default-domain=${spring.application.name} startup parameter to the base
log4j2AnyAlready Supported. Import log4j2 in the base and module, and additionally import the dependency:
<dependency>
  <groupId>com.alipay.koupleless</groupId>
  <artifactId>koupleless-adapter-log4j2</artifactId>
  <version>${latest Koupleless version}</version>
  <scope>provided</scope> <!– Module needs provided –>
  </dependency>
Complete usage examples for base and modules seen here
slf4j-api1.x and >= 1.7Already Supported
tomcat7.x, 8.x, 9.x, 10.x
and above		Already Supported
Complete usage examples for base and modules seen here
netty4.xAlready Supported
Complete usage examples for base and modules seen here
sofarpc>= 5.8.6Already Supported
dubbo3.xAlready Supported
Complete usage examples and considerations for base and modules can be seen here
grpc1.x and >= 1.42Already Supported
Complete usage examples and considerations for base and modules can be seen here
protobuf-java3.x and >= 3.17Already Supported
Complete usage examples and considerations for base and modules can be seen here
apollo1.x and >= 1.6.0Already Supported
Complete usage examples and considerations for base and modules can be seen here
nacos2.1.xAlready Supported
Complete usage examples and considerations for base and modules can be seen here
kafka-client>= 2.8.0 or
>= 3.4.0		Already Supported
Complete usage examples for base and modules can be seen here
rocketmq4.x and >= 4.3.0Already Supported
Complete usage examples for base and modules can be seen here
jedis3.xAlready Supported
Complete usage examples for base and modules can be seen here
xxl-job2.x and >= 2.1.0Already Supported
Needs to be declared as a compile dependency for use in modules
mybatis>= 2.2.2 or
>= 3.5.12		Already Supported
Complete usage examples for base and modules can be seen here
druid1.xAlready Supported
Complete usage examples for base and modules can be seen here
mysql-connector-java8.xAlready Supported
Complete usage examples for base and modules can be seen here
postgresql42.x and >= 42.3.8Already Supported
mongodb4.6.1Already Supported
Complete usage examples for base and modules can be seen here
hibernate5.x and >= 5.6.15Already Supported
j2cacheAnyAlready Supported
Needs to be declared as a compile dependency for independent use in modules
opentracing0.x and >= 0.32.0Already Supported
elasticsearch7.x and >= 7.6.2Already Supported
jaspyt1.x and >= 1.9.3Already Supported
OKHttp-Already Supported
Needs to be placed in the base, please use module automatic slimming capability
io.kubernetes:client10.x and >= 10.0.0Already Supported
net.java.dev.jna5.x and >= 5.12.1Already Supported
prometheus-Support to be verified
skywalking-The official does not support multiple service_names for one process. Tracing isolation can only be achieved by having each module print logs to separate directories. Please refer to the logging samples

Compatibility Relationships among Framework Versions

Note: Users can choose Koupleless versions according to their actual JDK and SpringBoot versions.

JDKSpringBootSOFAARKKoupleless
1.82.x2.2.101.x.x
173.0.x, 3.1.x3.0.x2.0.x
173.2.x and above3.1.x2.1.x

3.13 - 4.3.13 Koupleless Configuration

Koupleless configuration

Packaging Phase

Base Packaging Plugin Configuration

Plugin Parameter Configuration

The complete koupleless-base-build-plugin plugin configuration template is as follows:

<plugin>
  <groupId>com.alipay.sofa.koupleless</groupId>
  <artifactId>koupleless-base-build-plugin</artifactId>
  <version>${koupleless.runtime.version}</version>
  <executions>
    <execution>
      <goals>
        <goal>add-patch</goal>
        <!-- Used for static merger deployment-->
        <goal>integrate-biz</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
      <!--Base packaging directory, default is the project build directory-->
      <outputDirectory>./target</outputDirectory>
      
      <!--The groupId of the starter to be packaged, which defaults to the groupId of the project-->
      <dependencyGroupId>${groupId}</dependencyGroupId>
      
      <!--ArtifactId of the starter to be packaged-->
      <dependencyArtifactId>${baseAppName}-dependencies-starter</dependencyArtifactId>
      
      <!--Version number of the starter to be packaged-->
      <dependencyVersion>0.0.1-SNAPSHOT</dependencyVersion>
      
      <!--For debugging, change to true to see the intermediate products of the packaged starter-->
      <cleanAfterPackageDependencies>false</cleanAfterPackageDependencies>
  </configuration>
</plugin>

Static Integration Deployment Configuration

Developers need to specify the Ark Biz package that needs to be integrated and deployed in the ark configuration file of the base (conf/ark/bootstrap.properties or conf/ark/bootstrap.yml), with support for:

  • Local directory
  • Local file URL (File path for Windows is file:\\, and for Linux it is file://)
  • Remote URL (supports http://,https://) The configuration is as follows:
integrateBizURLs=file://${xxx}/koupleless_samples/springboot-samples/service/biz1/biz1-bootstrap/target/biz1-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar,\
  file://${xxx}/koupleless_samples/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar,\
  https://oss.xxxxx/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
integrateLocalDirs=/home/${xxx}/sofa-ark/biz,\
  /home/${xxx}/sofa-ark/biz2

or

integrateBizURLs:
  - file://${xxx}/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
  - file://${xxx}/koupleless_samples/springboot-samples/service/biz2/biz2-bootstrap/target/biz2-bootstrap-0.0.1-SNAPSHOT-ark-biz.jar
integrateLocalDirs:
  - /home/${xxx}/sofa-ark/biz
  - /home/${xxx}/sofa-ark/biz2

Module Packaging Plugin Configuration

Plugin Parameter Configuration

The complete sofa-ark-maven-plguin plugin configuration template is as follows:

<plugins>
    <plugin>
        <groupId>com.alipay.sofa</groupId>
        <artifactId>sofa-ark-maven-plugin</artifactId>
        <version>${sofa.ark.version}</version>
        <executions>
            <execution>
                <id>default-cli</id>
                <goals>
                    <goal>repackage</goal>
                </goals>
                <configuration>
                    <!--Ark package and ark biz packaging directory, default is the project build directory-->
                    <outputDirectory>./target</outputDirectory>
                    <!--Set the root directory of the application for reading the ${base.dir}/conf/ark/bootstrap.application configuration file, default to ${project.basedir}-->
                    <baseDir>./</baseDir>
                    <!--Generated ark package file name, default is ${artifactId}-->
                    <finalName>demo-ark</finalName>
                    <!--Whether to skip the goal:repackage execution, default to false-->
                    <skip>false</skip>
                    <!--Whether to package, install and publish ark biz, details please refer to the Ark Biz document, default to false-->
                    <attach>true</attach>
                    <!--Set the classifier of the ark package, default to empty-->
                    <arkClassifier>ark</arkClassifier>
                    <!--Set the classifier of the ark biz, default to ark-biz-->
                    <bizClassifier>ark-biz</bizClassifier>
                    <!--Set the biz name of the ark biz, default to ${artifactId}-->
                    <bizName>demo-ark</bizName>
                    <!--Set the biz version of the ark biz, default to ${artifactId}-->
                    <bizVersion>0.0.1</bizVersion>
                    <!--Set the startup priority of the ark biz, smaller priority has higher priority, ${artifactId}-->
                    <priority>100</priority>
                    <!--Set the startup entry of the ark biz, it will automatically search for the entry class that contains the main method and has the org.springframework.boot.autoconfigure.SpringBootApplication annotation-->
                    <mainClass>com.alipay.sofa.xx.xx.MainEntry</mainClass>
                    <!--Set whether to package dependencies with scope=provided, default to false-->
                    <packageProvided>false</packageProvided>
                    <!--Set whether to generate the Biz package, default to true-->
                    <keepArkBizJar>true</keepArkBizJar>
                    <!--For web applications, set the context path, default to /, each module should configure its own webContextPath, e.g.: biz1-->
                    <webContextPath>/</webContextPath>
                    <!--When packaging ark biz, exclude specified package dependencies; format: ${groupId:artifactId} or ${groupId:artifactId:classifier}-->
                    <excludes>
                        <exclude>org.apache.commons:commons-lang3</exclude>
                    </excludes>
                    <!--When packaging ark biz, exclude dependencies with the specified groupId-->
                    <excludeGroupIds>
                        <excludeGroupId>org.springframework</excludeGroupId>
                    </excludeGroupIds>
                    <!--When packaging ark biz, exclude dependencies with the specified artifactId-->
                    <excludeArtifactIds>
                        <excludeArtifactId>sofa-ark-spi</excludeArtifactId>
                    </excludeArtifactIds>
                    <!--When packaging ark biz, configure classes not covered by the ark plugin index; by default, ark biz will prioritize indexing all exported classes of ark plugin, which means it will only load the class locally, rather than delegating ark plugin to load-->
                    <denyImportClasses>
                        <class>com.alipay.sofa.SampleClass1</class>
                        <class>com.alipay.sofa.SampleClass2</class>
                    </denyImportClasses>
                    <!--Corresponding to the denyImportClasses configuration, package level can be configured-->
                    <denyImportPackages>
                        <package>com.alipay.sofa</package>
                        <package>org.springframework.*</package>
                    </denyImportPackages>
                    <!--When packaging ark biz, configure resources not covered by the ark plugin index; by default, ark biz will prioritize indexing all exported resources of the ark plugin, adding that configuration means that ark biz will only search for the resources internally without searching from the ark plugin-->
                    <denyImportResources>
                        <resource>META-INF/spring/test1.xml</resource>
                        <resource>META-INF/spring/test2.xml</resource>
                    </denyImportResources>
                  
                     <!--Isolates the dependencies that the ark biz has declared in its pom, default to false-->
                    <declaredMode>true</declaredMode>
                    <!--When packaging ark biz, only package dependencies that the base does not have, or dependencies of modules that are different from the base. This parameter specifies the "dependency management" identifier of the base, and is required to be a parent of module pom with the format ${groupId}:${artifactId}:${version}-->
                    <baseDependencyParentIdentity>${groupId}:${artifactId}:${version}</baseDependencyParentIdentity>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

Module Slimming Configuration

SOFAArk module slimming reads configuration from two places:

  • module project root directory/conf/ark/bootstrap.properties, e.g.: my-module/conf/ark/bootstrap.properties
  • module project root directory/conf/ark/bootstrap.yml, e.g.: my-module/conf/ark/bootstrap.yml bootstrap.properties In the module project root directory/conf/ark/bootstrap.properties, configure the commonly used packages of the framework and middleware that need to be down to the base as follows:
# excludes config ${groupId}:{artifactId}:{version}, split by ','
excludes=org.apache.commons:commons-lang3,commons-beanutils:commons-beanutils
# excludeGroupIds config ${groupId}, split by ','
excludeGroupIds=org.springframework
# excludeArtifactIds config ${artifactId}, split by ','
excludeArtifactIds=sofa-ark-spi

bootstrap.yml In the module project root directory/conf/ark/bootstrap.yml, configure the commonly used packages of the framework and middleware that need to be down to the base as follows:

# excludes 中配置 ${groupId}:{artifactId}:{version}, 不同依赖以 - 隔开
# excludeGroupIds 中配置 ${groupId}, 不同依赖以 - 隔开
# excludeArtifactIds 中配置 ${artifactId}, 不同依赖以 - 隔开
excludes:
  - org.apache.commons:commons-lang3
  - commons-beanutils:commons-beanutils
excludeGroupIds:
  - org.springframework
excludeArtifactIds:
  - sofa-ark-spi

Development Phase

Arklet Configuration

Port Configuration

When the base is started, configure the port in the JVM parameters, default is 1238

-Dkoupleless.arklet.http.port=XXXX

Module Runtime Configuration

Configuration of Health Check

Configuration in the application.properties of the base:

# Or do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
# If all information needs to be displayed, configure the following content
management.endpoint.health.show-components=always
management.endpoint.health.show-details=always
# Do not ignore module startup status
koupleless.healthcheck.base.readiness.withAllBizReadiness=true

Web Gateway Configuration

When traditional applications are split into modules, each module has its own webContextPath, and the upstream caller needs to modify the request path. To avoid the modification, you can configure Web Gateway forwarding rules in the application.properties or application.yaml, allowing the upstream caller not to modify. In the configuration, three strategies can be configured:

  • Domain matching: specifies that requests that meet HostA are forwarded to ModuleA
  • Path matching: specifies that requests that meet PathA are forwarded to specific PathB of ModuleA
  • Domain and path matching: specifies that requests that meet HostA and PathA will be forwarded to specific PathB of ModuleA application.yaml Configuration example:
koupleless:
  web:
    gateway:
      forwards:
# host in [a.xxx,b.xxx,c.xxx] path /${anyPath} --forward to--> biz1/${anyPath}
        - contextPath: biz1
        - hosts:
            - a
            - b
            - c
# /idx2/** -> /biz2/**, /t2/** -> /biz2/timestamp/**
        - contextPath: biz2
        - paths:
            - from: /idx2
            - to: /
            - from: /t2
            - to: /timestamp
# /idx1/** -> /biz1/**, /t1/** -> /biz1/timestamp/**
        - contextPath: biz1
        - paths:
            - from: /idx1
            - to: /
            - from: /t1
            - to: /timestamp

application.properties Configuration example:

# host in [a.xxx,b.xxx,c.xxx] path /${anyPath} --forward to--> biz1/${anyPath}
koupleless.web.gateway.forwards[0].contextPath=biz1
koupleless.web.gateway.forwards[0].hosts[0]=a
koupleless.web.gateway.forwards[0].hosts[1]=b
koupleless.web.gateway.forwards[0].hosts[2]=c
# /idx2/** -> /biz2/**, /t2/** -> /biz2/timestamp/**
koupleless.web.gateway.forwards[1].contextPath=biz2
koupleless.web.gateway.forwards[1].paths[0].from=/idx2
koupleless.web.gateway.forwards[1].paths[0].to=/
koupleless.web.gateway.forwards[1].paths[1].from=/t2
koupleless.web.gateway.forwards[1].paths[1].to=/timestamp
# /idx1/** -> /biz1/**, /t1/** -> /biz1/timestamp/**
koupleless.web.gateway.forwards[2].contextPath=biz1
koupleless.web.gateway.forwards[2].paths[0].from=/idx1
koupleless.web.gateway.forwards[2].paths[0].to=/
koupleless.web.gateway.forwards[2].paths[1].from=/t1
koupleless.web.gateway.forwards[2].paths[1].to=/timestamp

3.14 - 4.3.14 SOFAArk Key User Documentation

Module Lifecycle

Ark Event Mechanism

Ark Logging



4 - 5. Module Controller V2 Operation and Maintenance

Operation and Maintenance of Modules under the Koupleless Module Controller V2 Architecture

4.1 - 5.1 Module Release

Koupleless Module Online and Offline Procedures

Note: The current ModuleController v2 has only been tested on Kubernetes (K8S) version 1.24, with no testing on other versions. ModuleController V2 relies on certain Kubernetes (K8S) features; thus, the K8S version must not be lower than V1.10.

Module Release

ModuleController V2 supports deploying modules using any Pod deployment method, including but not limited to bare Pod deployment, Deployments, DaemonSets, and StatefulSets. Below, we demonstrate the release process using Deployment as an example; configurations for other methods can refer to the template configuration in Deployment:

kubectl apply -f samples/module-deployment.yaml --namespace yournamespace

The complete content is as follows:

apiVersion: apps/v1  # Specifies the API version, which must be listed in `kubectl api-versions`
kind: Deployment  # Specifies the role/type of resource to create
metadata:  # Metadata/attributes of the resource
  name: test-module-deployment  # Name of the resource, must be unique within the same namespace
  namespace: default # Namespace where it will be deployed
spec:  # Specification field of the resource
  replicas: 1
  revisionHistoryLimit: 3 # Retains historical versions
  selector: # Selector
    matchLabels: # Matching labels
      app: test-module-deployment
  strategy: # Strategy
    rollingUpdate: # Rolling update
      maxSurge: 30% # Maximum additional replicas that can exist, can be a percentage or an integer
      maxUnavailable: 30% # Maximum number of Pods that can become unavailable during the update, can be a percentage or an integer
    type: RollingUpdate # Rolling update strategy
  template: # Template
    metadata: # Metadata/attributes of the resource
      labels: # Sets resource labels
        module-controller.koupleless.io/component: module # Required, declares Pod type for management by module controller
        # Unique ID for Deployment
        app: test-module-deployment-non-peer
    spec: # Specification field of the resource
      containers:
        - name: biz1 # Required, declares the module's bizName, must match the artifactId declared in pom.xml
          image: https://serverless-opensource.oss-cn-shanghai.aliyuncs.com/module-packages/stable/biz1-web-single-host-0.0.1-SNAPSHOT-ark-biz.jar
          env:
            - name: BIZ_VERSION # Required, declares module's biz_version, value must match the version declared in pom.xml
              value: 0.0.1-SNAPSHOT
      affinity:
        nodeAffinity: # Required, declares the base selector to ensure modules are scheduled onto designated bases
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: base.koupleless.io/stack
                    operator: In
                    values:
                      - java # Mandatory in a multi-language environment, specifies the tech stack
                  - key: base.koupleless.io/version
                    operator: In
                    values:
                      - 1.1.1 # Specified base version, mandatory, at least one required
                  - key: base.koupleless.io/name
                    operator: In
                    values:
                      - base  # Specified base bizName, mandatory, at least one required
      tolerations: # Required, allows pods to be scheduled onto base nodes
        - key: "schedule.koupleless.io/virtual-node"
          operator: "Equal"
          value: "True"
          effect: "NoExecute"

All configurations align with a regular Deployment, except for mandatory fields; additional Deployment configurations can be added for custom functionality.

Subsequent module updates can be achieved by updating the module Deployment’s Container Image and BIZ_VERSION, utilizing the Deployment’s RollingUpdate for phased updates.

The Module Controller ensures lossless module traffic during rolling updates by controlling the Pod status update sequence on the same base. The process is as follows:

  1. After updating the Deployment, new version module Pods are created based on the update strategy.
  2. The K8S Scheduler schedules these Pods to the VNode, where the old module version is still installed.
  3. The Module Controller detects the successful scheduling of Pods and initiates the installation of the new module version.
  4. Once the installation is complete, the Module Controller checks the status of all modules on the current base, sorts the associated Pods by creation time, and updates their statuses in sequence. This causes the Pods corresponding to the old module version to become Not Ready first, followed by the new version Pods becoming Ready.
  5. The Deployment controller, upon detecting that the new Pods are Ready, begins cleaning up old version Pods. It prioritizes deleting Pods that are Not Ready. At this point, the old version Pods on the same base are already Not Ready and are deleted, preventing Ready state old version Pods on other bases from being deleted first.

Throughout this process, there is no instance where a base lacks a module, ensuring lossless traffic during the module update.

Checking Module Status

This requirement can be met by examining Pods with nodeName corresponding to the base’s node. First, understand the mapping between base services and nodes.

In the design of Module Controller V2, each base generates a globally unique UUID at startup as the identifier for the base service. The corresponding node’s Name includes this ID.

Additionally, the IP of the base service corresponds one-to-one with the node’s IP, allowing selection of the corresponding base Node via IP.

Therefore, you can use the following command to view all Pods (modules) installed on a specific base and their statuses:

kubectl get pod -n <namespace> --field-selector status.podIP=<baseIP>

Or

kubectl get pod -n <namespace> --field-selector spec.nodeName=virtual-node-<baseUUID>

Module Offline

Removing the module’s Pod or other controlling resources in the K8S cluster completes the module offline process. For instance, in a Deployment scenario, you can directly delete the corresponding Deployment to offline the module:

kubectl delete yourmoduledeployment --namespace yournamespace

Replace yourmoduledeployment with your ModuleDeployment name and yournamespace with your namespace.

For customizing module release and operation strategies (such as grouping, Beta testing, pausing, etc.), refer to Module Operation and Scheduling Strategies.

The demonstrated example uses kubectl; directly calling the K8S API Server to delete Deployment also achieves module group offline.

Module Scaling

Since ModuleController V2 fully leverages K8S’s Pod orchestration scheme, scaling only occurs on ReplicaSets, Deployments, StatefulSets, etc. Scaling can be implemented according to the respective scaling methods; below, we use Deployment as an example:

kubectl scale deployments/yourdeploymentname --namespace=yournamespace --replicas=3

Replace yourdeploymentname with your Deployment name, yournamespace with your namespace, and set the replicas parameter to the desired scaled quantity.

Scaling strategies can also be implemented through API calls.

Module Replacement

In ModuleController v2, modules are tightly bound to Containers. To replace a module, you need to execute an update logic, updating the module’s Image address on the Pod where the module resides.

The specific replacement method varies slightly depending on the module deployment method; for instance, directly updating Pod information replaces the module in-place, while Deployment executes the configured update strategy (e.g., rolling update, creating new version Pods before deleting old ones). DaemonSet also executes the configured update strategy but with a different logic – deleting before creating, which might cause traffic loss.

Module Rollback

Being compatible with native Deployments, rollback can be achieved using Deployment’s rollback method.

To view deployment history:

kubectl rollout history deployment yourdeploymentname

To rollback to a specific version:

kubectl rollout undo deployment yourdeploymentname --to-revision=<TARGET_REVISION>

Other Operational Issues

Module Traffic Service Implementation

A native Service can be created for the module, which can provide services only when the base and ModuleController are deployed within the same VPC.

As bases and ModuleController may not be deployed in the same VPC currently, interaction between them is realized through MQTT message queues. Base nodes integrate the IP of the Pod where the base resides, and module Pods integrate the IP of the base node. Therefore, when the base itself and ModuleController are not in the same VPC, the IP of the module is actually invalid, preventing external service provision.

A potential solution involves forwarding at the Load Balancer (LB) layer of the Service, redirecting the Service’s traffic to the base service on the corresponding IP of the K8S where the base resides. Further evaluation and optimization of this issue will be based on actual usage scenarios.

Incompatible Base and Module Release

  1. Deploy a module’s Deployment first, specifying the latest version of the module code package address in Container and the name and version information of the new version base in nodeAffinity. This Deployment will create corresponding Pods, but they won’t be scheduled until new version bases are created.
  2. Update the base Deployment to release the new version image, triggering the replacement and restart of the base. Upon startup, the base informs the ModuleController V2 controller, creating a corresponding version node.
  3. After the creation of the corresponding version base node, the K8S scheduler automatically triggers scheduling, deploying the Pods created in step 1 onto the base node for installation of the new version module, thus achieving simultaneous release.


4.2 - 5.2 Module Release Operations Strategy

Koupleless Module Release Operations Strategy

Operations Strategy

To achieve zero-downtime changes in the production environment, the module release operations leverage Kubernetes (K8S) native scheduling capabilities to provide secure and reliable update functionality. Users can deploy module Pods according to business requirements.

Scheduling Strategy

Dispersion Scheduling: Achieved through native Deployment controls, with PodAffinity configurations facilitating dispersion scheduling.

Peer and Non-Peer Deployment

Peer and non-peer deployment strategies can be realized by selecting different deployment methods.

Peer Deployment

Two implementation methods are provided:

  1. Using DaemonSet: Modules can be deployed as DaemonSets, where a DaemonSet controller automatically creates a module Pod for each base node upon its addition, ensuring peer deployment.

    Note that DaemonSet rolling updates occur by uninstalling before reinstalling; choose based on actual business needs.

  2. Via Deployment: Unlike DaemonSet, an additional component is required to maintain module replica count equivalent to the number of base nodes (under development, expected in the next release). Supports install-before-uninstall, avoiding backend traffic loss in a microservices architecture.

    While Deployments strive for dispersion, they do not guarantee complete dispersion; modules might be deployed multiple times to the same base. For strong dispersion, add Pod anti-affinity settings in the Deployment, as shown below:

apiVersion: apps/v1
kind: Deployment
metadata:
    name: test-module-deployment
    namespace: default
    labels:
        module-controller.koupleless.io/component: module-deployment
spec:
    replicas: 1
    revisionHistoryLimit: 3
    selector:
        matchLabels:
            module.koupleless.io/name: biz1
            module.koupleless.io/version: 0.0.1
    strategy:
        rollingUpdate:
            maxSurge: 30%
            maxUnavailable: 30%
        type: RollingUpdate
    template:
        metadata:
            labels:
                module-controller.koupleless.io/component: module
                module.koupleless.io/name: biz1
                module.koupleless.io/version: 0.0.1
        spec:
            containers:
            - name: biz1
              image: https://serverless-opensource.oss-cn-shanghai.aliyuncs.com/module-packages/test_modules/biz1-0.0.1-ark-biz.jar
              env:
              - name: BIZ_VERSION
                value: 0.0.1
            affinity:
                nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                        nodeSelectorTerms:
                        - matchExpressions:
                          - key: base.koupleless.io/stack
                            operator: In
                            values: ["java"]
                          - key: base.koupleless.io/version
                            operator: In
                            values: ["1.0.0"] # If modules can only be scheduled to specific node versions, this field is mandatory.
                          - key: base.koupleless.io/name
                            operator: In
                            values: ["base"]
                podAntiAffinity: # Core configuration for dispersion scheduling
                    requiredDuringSchedulingIgnoredDuringExecution:
                    - labelSelector:
                        matchLabels:
                            module.koupleless.io/name: biz1
                            module.koupleless.io/version: 0.0.1
                      topologyKey: topology.kubernetes.io/zone
            tolerations:
            - key: "schedule.koupleless.io/virtual-node"
              operator: "Equal"
              value: "True"
              effect: "NoExecute"

Non-Peer Deployment

Achieved by deploying modules as Deployments or ReplicaSets, with deployments based on the replica count setting.

Batch Updates

The strategy for batch updates requires custom control logic. ModuleController V2 introduces a capability where, when different versions of the same-named module are installed sequentially on a base, the Pod of the earlier-installed module enters BizDeactivate status and transitions to the Failed phase. Exploit this logic to implement batch update strategies.



4.3 - 5.3 Health Check

Background

The purpose of health checks is to obtain the status of an application throughout its lifecycle, including the operational and runtime phases, so that users can make decisions based on this status. For instance, if the application status is DOWN, it indicates a malfunction in the application, and the user may choose to restart or replace the machine.

In the case of a single application, health checks are relatively simple:

  • Operational phase status:
    • If it’s starting up, the status is UNKNOWN;
    • If startup fails, the status is DOWN;
    • If startup is successful, the status is UP.
  • Runtime phase status:
    • If all health checkpoints of the application are healthy, the status is UP;
    • If any health checkpoint of the application is not healthy, the status is DOWN.

In multi-application scenarios, the situation can be much more complex. We need to consider the impact of the multi-application’s status during both the operational phase and the runtime phase on the overall application health. When designing health checks, we need to consider the following two issues:

  • During the module operational phase, should the module start-up status affect the overall application health status?

    In different operational scenarios, users have different expectations. koupleless module operations have three scenarios:

    ScenarioImpact of the Module on the Overall Application Health Status
    Module Hot-DeploymentProvide configuration to let users decide whether the hot-deployment result should affect the overall application health status (default configuration is: does not affect the original health status of the application)
    Static Merge DeploymentModule deployment occurs during the base startup, so the module startup status should directly affect the overall health status of the application
    Module ReplayModule replay occurs during the base startup, thus the module startup status should directly affect the overall health status of the application

  • During the module runtime phase, should the module running status affect the overall application health status?

    The module runtime phase status should have a direct impact on the overall application health status.

Under this context, we have designed a health check approach for multi-application scenarios.

Usage

Requirements

  • Koupleless version >= 1.1.0
  • sofa-ark version >= 2.2.9

Obtain the overall health status of the application

There are 3 types of health status for the base:

StatusMeaning
UPHealthy, indicating readiness
UNKNOWNCurrently starting up
DOWNUnhealthy (may be due to startup failure or unhealthy running state)

Since Koupleless supports hot deployment of modules, while obtaining the overall health status of the application, users may wish for the module deployment result to impact the overall application health status or not.

Module launch result does not affect the overall application health status (default)

  • Features: For a healthy base, if the module installation fails, it will not affect the overall application health status.
  • Usage: Same as the health check configuration for regular Spring Boot, configure in the base’s application.properties:
# or do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
# If you need to display all information, configure the following content
management.endpoint.health.show-components=always
management.endpoint.health.show-details=always
  • Access: {baseIp:port}/actuator/health
  • Result:
{
    // Overall health status of the application
    "status": "UP",
    "components": {
        // Aggregated health status of the modules
        "arkBizAggregate": {
            "status": "UP",
            "details": {
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP",
                    // Can see the health status of all active HealthIndicators in the modules
                    "details": {
                        "diskSpace": {
                          "status": "UP",
                          "details": {
                            "total": 494384795648,
                            "free": 272435396608,
                            "threshold": 10485760,
                            "exists": true
                            }
                        },
                        "pingHe": {
                          "status": "UP",
                          "details": {}
                        }
                    }
                }
            }
        },
        // Startup health status of base and modules
        "masterBizStartUp": {
            "status": "UP",
            // Including the startup status of each module.
            "details": {
                "base:1.0.0": {
                    "status": "UP"
                },
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP"
                },
                "biz2:0.0.1-SNAPSHOT": {
                    "status": "DOWN"
                }
            }
        }
    }
}

Overall Health Status in Different Scenarios

Scenario 1: Start base

StatusMeaning
UPBase is healthy
UNKNOWNBase is starting up
DOWNBase is unhealthy

Scenario 2: Start base and install modules with static merge deployment

StatusMeaning
UPBase and module are healthy
UNKNOWNBase or module is starting up
DOWNBase startup failed / base is unhealthy / module startup failed / module is unhealthy

Scenario 3: After base starts, install modules with hot deployment

Provide configuration to let users decide whether the result of module hot deployment affects the overall health status of the application (The default configuration is: Does not affect the original health status of the application)

Default Configuration: In the scenario of hot deployment, whether or not a module is successfully installed does not affect the overall health status of the application, as follows:

StatusMeaning
UPBase and module are healthy
UNKNOWNBase is starting up
DOWNBase startup failed / base is unhealthy / module is unhealthy

Scenario 4: Base running

StatusMeaning
UPBase and module are healthy
UNKNOWN-
DOWNBase is unhealthy or module is unhealthy

Scenario 5: After base started, reinstall module

Reinstall module refers to automatically pulling the module baseline and installing the module after the base is started.

Reinstall module is not supported at the moment

StatusMeaning
UPBase and module are healthy
UNKNOWNBase or module is starting up
DOWNBase is unhealthy or module startup failed or module is unhealthy

Module launch result affects the overall application health status

  • Features: For a healthy base, if a module installation fails, the overall application health status will also fail.
  • Usage: In addition to the above configuration, you need to configure koupleless.healthcheck.base.readiness.withAllBizReadiness=true, that is, configure in the base’s application.properties:
# Alternatively, do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
# If you need to display all information, configure the following content
management.endpoint.health.show-components=always
management.endpoint.health.show-details=always
# Do not ignore module startup status
koupleless.healthcheck.base.readiness.withAllBizReadiness=true
  • Access: {baseIp:port}/actuator/health
  • Result:
{
    // Overall health status of the application
    "status": "UP",
    "components": {
        // Aggregated health status of the modules
        "arkBizAggregate": {
            "status": "UP",
            "details": {
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP",
                    // Can see the health status of all active HealthIndicators in the modules
                    "details": {
                        "diskSpace": {
                          "status": "UP",
                          "details": {
                            "total": 494384795648,
                            "free": 272435396608,
                            "threshold": 10485760,
                            "exists": true
                            }
                        },
                        "pingHe": {
                          "status": "UP",
                          "details": {}
                        }
                    }
                }
            }
        },
        // Startup health status of base and modules
        "masterBizStartUp": {
            "status": "UP",
            // Including the startup status of each module.
            "details": {
                "base:1.0.0": {
                    "status": "UP"
                },
                "biz1:0.0.1-SNAPSHOT": {
                    "status": "UP"
                }
            }
        }
    }
}

Overall Health Status in Different Scenarios

Scenario 1: Start base

StatusMeaning
UPBase is healthy
UNKNOWNBase is starting up
DOWNBase is unhealthy

Scenario 2: Start base and install modules with static merge deployment

StatusMeaning
UPBase and module are healthy
UNKNOWNBase or module is starting up
DOWNBase startup failed / base is unhealthy / module startup failed / module is unhealthy

Scenario 3: After base starts, install modules with hot deployment

Provide configuration to let users decide whether the result of module hot deployment affects the overall health status of the application (The default configuration is: Does not affect the original health status of the application)

When configuring as koupleless.healthcheck.base.readiness.withAllBizReadiness=true:

StatusMeaning
UPBase and module are healthy
UNKNOWNBase or module is starting up
DOWNBase startup failed / Module startup failed / base is unhealthy / module is unhealthy

Scenario 4: Base running

StatusMeaning
UPBase and module are healthy
UNKNOWN-
DOWNBase is unhealthy or module is unhealthy

Scenario 5: After base started, reinstall module

Reinstall module refers to automatically pulling the module baseline and installing the module after the base is started.

Reinstall module is not supported at the moment.

Obtaining the Health Status of a Single Module

  • Usage: Consistent with the normal springboot health check configuration, enable the health node, i.e. configure in the module’s application.properties:
# or do not configure management.endpoints.web.exposure.include
management.endpoints.web.exposure.include=health
  • Access: {baseIp:port}/{bizWebContextPath}/actuator/info
  • Result:
{
    "status": "UP",
    "components": {
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 494384795648,
                "free": 270828220416,
                "threshold": 10485760,
                "exists": true
            }
        },
        "ping": {
            "status": "UP"
        }
    }
}

Get information about base, modules, and plugins

  • Usage: Same as the regular springboot health check configuration, enable the info endpoint, i.e., configure in the base’s application.properties:
# Note: If the user configures management.endpoints.web.exposure.include on their own, they need to include the health endpoint, otherwise the health endpoint cannot be accessed
management.endpoints.web.exposure.include=health,info
  • Access: {baseIp:port}/actuator/info
  • Result:
{
    "arkBizInfo": [
      {
        "bizName": "biz1",
        "bizVersion": "0.0.1-SNAPSHOT",
        "bizState": "ACTIVATED",
        "webContextPath": "biz1"
      },
      {
        "bizName": "base",
        "bizVersion": "1.0.0",
        "bizState": "ACTIVATED",
        "webContextPath": "/"
      }
    ],
    "arkPluginInfo": [
        {
            "pluginName": "koupleless-adapter-log4j2",
            "groupId": "com.alipay.sofa.koupleless",
            "artifactId": "koupleless-adapter-log4j2",
            "pluginVersion": "1.0.1-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/koupleless/koupleless-adapter-log4j2/1.0.1-SNAPSHOT/koupleless-adapter-log4j2-1.0.1-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.koupleless.adapter.Log4j2AdapterActivator"
        },
        {
            "pluginName": "web-ark-plugin",
            "groupId": "com.alipay.sofa",
            "artifactId": "web-ark-plugin",
            "pluginVersion": "2.2.9-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/web-ark-plugin/2.2.9-SNAPSHOT/web-ark-plugin-2.2.9-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.ark.web.embed.WebPluginActivator"
        },
        {
            "pluginName": "koupleless-base-plugin",
            "groupId": "com.alipay.sofa.koupleless",
            "artifactId": "koupleless-base-plugin",
            "pluginVersion": "1.0.1-SNAPSHOT",
            "pluginUrl": "file:/Users/lipeng/.m2/repository/com/alipay/sofa/koupleless/koupleless-base-plugin/1.0.1-SNAPSHOT/koupleless-base-plugin-1.0.1-SNAPSHOT.jar!/",
            "pluginActivator": "com.alipay.sofa.koupleless.plugin.ServerlessRuntimeActivator"
        }
    ]
}

4.4 - 5.4 Deployment of Module Controller V2

Deployment methodology for Koupleless Module Controller V2

Note: ModuleController V2 has only been tested on K8S version 1.24 and relies on certain K8S features. Therefore, the K8S version should not be lower than V1.10.

Resource File Locations

  1. Role Definition
  2. RBAC Definition
  3. ServiceAccount Definition
  4. ModuleControllerV2 Deployment Definition

Deployment Method

Use the kubectl apply command to sequentially apply the above four resource files to complete the deployment of a single-instance ModuleController.

For using the Module Controller’s sharded cluster capability, modify the above deployment definition to a Deployment version, placing the Pod Spec content into the Deployment template.

To use load balancing in a sharded cluster, set the IS_CLUSTER parameter to true in the Module Controller ENV configuration.

Configurable Parameter Explanation

Environment Variable Configuration

Below are some configurable environment variables and their explanations:

  • ENABLE_MQTT_TUNNEL

    • Meaning: Flag to enable MQTT operations pipeline. Set to true to enable. If enabled, configure the related environment variables below.

  • MQTT_BROKER

    • Meaning: URL of the MQTT broker.

  • MQTT_PORT

    • Meaning: MQTT port number.

  • MQTT_USERNAME

    • Meaning: MQTT username.

  • MQTT_PASSWORD

    • Meaning: MQTT password.

  • MQTT_CLIENT_PREFIX

    • Meaning: MQTT client prefix.

  • ENABLE_HTTP_TUNNEL

    • Meaning: Flag to enable HTTP operations pipeline. Set to true to enable. Optionally configure the environment variables below.

  • HTTP_TUNNEL_LISTEN_PORT

    • Meaning: Module Controller HTTP operations pipeline listening port, default is 7777.

  • REPORT_HOOKS

    • Meaning: Error reporting links. Supports multiple links separated by ;. Currently only supports DingTalk robot webhooks.

  • ENV

    • Meaning: Module Controller environment, set as VNode label for operations environment isolation.

  • IS_CLUSTER

    • Meaning: Cluster flag. If true, Virtual Kubelet will start with cluster configuration.

  • WORKLOAD_MAX_LEVEL

    • Meaning: Cluster configuration indicating the maximum workload level for workload calculation in Virtual Kubelet. Default is 3. Refer to Module Controller architecture design for detailed calculation rules.

  • ENABLE_MODULE_DEPLOYMENT_CONTROLLER

    • Meaning: Flag to enable the Module Deployment Controller. If true, the deployment controller will start to modify Module deployment replicas and baselines.

  • VNODE_WORKER_NUM

    • Meaning: Number of concurrent processing threads for VNode Modules. Set to 1 for single-threaded.

  • CLIENT_ID

    • Meaning: Optional, Module Controller instance ID. need to be unique in one env, will generate a random UUID in default.

Documentation Reference

For detailed structure and implementation, refer to the documentation.

4.5 - 5.5 Module Information Retrieval

Koupleless Module Information Retrieval

View the names and statuses of all installed modules on a base instance

kubectl get module -n <namespace> -l koupleless.alipay.com/base-instance-ip=<pod-ip> -o custom-columns=NAME:.metadata.name,STATUS:.status.status

or

kubectl get module -n <namespace> -l koupleless.alipay.com/base-instance-name=<pod-name> -o custom-columns=NAME:.metadata.name,STATUS:.status.status

View detailed information of all installed modules on a base instance

kubectl describe module -n <namespace> -l koupleless.alipay.com/base-instance-ip=<pod-ip>

or

kubectl describe module -n <namespace> -l koupleless.alipay.com/base-instance-name=<pod-name>

Replace <pod-ip> with the IP of the base instance you want to view, <pod-name> with the name of the base instance you want to view, and <namespace> with the namespace of the resources you want to view.

4.6 - 5.6 Error Codes

This article mainly introduces the error codes of Arklet, ModuleController, and KouplelessBoard.

ErrorCode Rules

Two-level error codes, support dynamic combination, using PascalCase, different levels of error codes can only be separated by “."
such as Arklet.InstallModuleFailed
Level 1: Error Source
Level 2: Error Type

Suggestion

Briefly explain the solution for upstream operations to refer to.

Arklet Error Codes

Level 1: Error Source

CodeMeaning
UserErrors caused by the user
ArkletExceptions from Arklet itself
ModuleControllerExceptions caused by specific upstream components
OtherUpstreamExceptions caused by unknown upstream

Level 2: Error Type

Business TypeError SourceError TypeMeaningSolution
GeneralArkletUnknownErrorUnknown error (default)Please check

ModuleControllerInvalidParameterParameter validation failedPlease check the parameters
ModuleControllerInvalidRequestInvalid operation typePlease check the request
OtherUpstreamDecodeURLFailedURL parsing failedPlease check if the URL is valid
Query RelatedArkletNoMatchedBizModule query failed, no target biz exists-
ArkletInvalidBizNameModule query failed, query parameter bizName cannot be emptyPlease add the query parameter bizName
Installation RelatedArkletInstallationRequirementNotMetModule installation conditions are not metPlease check the necessary parameters for module installation
ArkletPullBizErrorPackage pulling failedPlease retry
ArkletPullBizTimeOutPackage pulling timed outPlease retry
UserDiskFullDisk full when pulling the packagePlease replace the base
UserMachineMalfunctionMachine malfunctionPlease restart the base
UserMetaspaceFullMetaspace exceeds the thresholdPlease restart the base
ArkletInstallBizExecutingModule is being installedPlease retry

ArkletInstallBizTimedOutUninstalling old module failed during module installationPlease check
ArkletInstallBizFailedNew module installation failed during module installationPlease check
UserInstallBizUserErrorModule installation failed, business exceptionPlease check the business code
Uninstallation RelatedArkletUninstallBizFailedUninstallation failed, current biz still exists in the containerPlease check
ArkletUnInstallationRequirementNotMetModule uninstallation conditions are not metThe current module has multiple versions, and the version to be uninstalled is in the active state, which is not allowed to be uninstalled

ModuleController Error Codes

Level 1: Error Source

CodeMeaning
UserErrors caused by the user
ModuleControllerExceptions from ModuleController itself
KouplelessBoardExceptions caused by specific upstream components
ArkletExceptions caused by specific downstream components
OtherUpstreamExceptions caused by unknown upstream
OtherDownstreamExceptions caused by unknown downstream

Level 2: Error Type

Business TypeError SourceError TypeMeaningSolution
GeneralModuleControllerUnknownErrorUnknown error (default)Please check

OtherUpstreamInvalidParameterParameter validation failedPlease check the parameters
ArkletArkletServiceNotFoundBase service not foundPlease ensure that the base has Koupleless dependency
ArkletNetworkErrorNetwork call exceptionPlease retry
OtherUpstreamSecretAKErrorSignature exceptionPlease confirm that there are operation permissions
ModuleControllerDBAccessErrorDatabase read/write failedPlease retry
OtherUpstreamDecodeURLFailedURL parsing failedPlease check if the URL is valid
ModuleControllerRetryTimesExceededMultiple retries failedPlease check
ModuleControllerProcessNodeMissedLack of available working nodesPlease retry later
ModuleControllerServiceMissedService missingPlease check if ModuleController version contains the template type
ModuleControllerResourceConstranedResource limited (thread pool, queue, etc. full)Please retry later
Installation RelatedArkletInstallModuleTimedOutModule installation timed outPlease retry
Arklet / UserInstallModuleFailedModule installation failedPlease check the failure reason
ArkletInstallModuleExecutingModule is being installedThe same module is being installed, please retry later
UserDiskFullDisk fullPlease replace
Uninstallation RelatedOtherUpstreamEmptyIPListIP list is emptyPlease enter the IP to be uninstalled

ArkletUninstallBizTimedOutModule uninstallation timed outPlease retry

ArkletUninstallBizFailedModule uninstallation failedPlease check
Base RelatedModuleControllerBaseInstanceNotFoundBase instance not foundPlease ensure that the base instance exists

KubeAPIServerGetBaseInstanceFailedFailed to query base informationPlease ensure that the base instance exists
ModuleControllerBaseInstanceInOperationBase instance is under operationPlease retry later
ModuleControllerBaseInstanceNotReadyBase data not read or base is not availablePlease ensure that the base is available
ModuleControllerBaseInstanceHasBeenReplacedBase instance has been replacedAdditional base instances will be added later, please wait
ModuleControllerInsufficientHealthyBaseInstanceInsufficient healthy base instancesPlease scale out
Scaling RelatedModuleControllerRescaleRequirementNotMetScaling conditions are not metPlease check if there are enough machines for scaling/Check the scaling ratio

⚠️ Note: The base runs on different base instances, such as pods. Therefore, BaseInstanceInOperation, BaseInstanceNotReady, BaseInstanceHasBeenReplaced, InsufficientHealthyBaseInstance error codes may refer to both the application status of the base and the status of the base instance.

DashBoard Error Codes

Level 1: Error Source

CodeMeaning
KouplelessBoardExceptions from KouplelessBoard itself
ModuleControllerExceptions caused by specific downstream components
OtherUpstreamExceptions caused by unknown upstream
OtherDownstreamExceptions caused by unknown downstream

Level 2: Error Type

Business TypeError SourceError TypeMeaningSolution
GeneralKouplelessBoardUnknownErrorUnknown error (default)

OtherUpstreamInvalidParameterParameter validation failedPlease check the parameters
Work OrderKouplelessBoardOperationPlanNotFoundWork order not foundPlease check
KouplelessBoardOperationPlanMutualExclusionWork order mutual exclusionPlease retry
Internal ErrorKouplelessBoardInternalErrorInternal system errorPlease check
KouplelessBoardThreadPoolErrorThread pool call exceptionPlease check
Operation and MaintenanceModuleControllerBaseInstanceOperationFailedOperation failedPlease check
ModuleControllerBaseInstanceUnderOperationUnder operationPlease retry
ModuleControllerBaseInstanceOperationTimeOutOperation timed outPlease retry
ModuleControllerOverFiftyPercentBaseInstancesUnavaliableMore than 50% of machine traffic is unreachablePlease check the base instance
KouplelessBoardBaselineInconsistencyConsistency check failed (inconsistent baseline)Please check
External Service Call ErrorOtherDownstreamExternalErrorExternal service call errorPlease check
KouplelessBoardNetworkErrorExternal service call timed outPlease retry