6.5.3 Runtime Adaptation or Best Practices for Multi-Module Deployment

• 1: 6.5.3.1 Koupleless Multi-Application Governance Patch Management
• 2: 6.5.3.2 Introduction to Multi-Module Integration Testing Framework
• 3: 6.5.3.3 Adapting to Multi-Module with Dubbo 2.7
• 4: 6.5.3.4 Best Practices for Multi-Module with ehcache
• 5: 6.5.3.5 Logback's adaptation for multi-module environments
• 6: 6.5.3.6 log4j2 Multi-Module Adaptation
• 7: 6.5.3.7 Module Use Bes
• 8: 6.5.3.8 Module Using Dubbo
• 9: 6.5.3.10 Introduction to the Principle of Class Delegation Loading between Foundation and Modules
• 10: 6.3.5.11 What happens if a module independently introduces part of the SpringBoot framework?

1 - 6.5.3.1 Koupleless Multi-Application Governance Patch Management

Why Koupleless Needs Multi-Application Governance Patching?

Koupleless is a multi-application architecture, and traditional middleware may only consider scenarios for a single application. Therefore, in some cases, it is incompatible with multi-application coexistence, leading to problems such as shared variable contamination, classloader loading exceptions, and unexpected class judgments. Thus, when using Koupleless middleware, we need to patch some potential issues, covering the original middleware implementation, allowing open-source middleware to be compatible with the multi-application mode.

Research on Multi-Application Governance Patching Solutions for Koupleless

In multi-application compatibility governance, we not only consider production deployment but also need to consider compatibility with local user development (IDEA click Debug), compatibility with unit testing (e.g., @SpringbootTest), and more.

Below is a comparison table of different solutions.

Solution Comparison

Conclusion

Overall, it is not possible to achieve user 0 perception access completely, and each method requires minor business refactoring. Among many solutions, A and D can achieve full compatibility. However, the A solution does not require business code changes, nor does it intrude into runtime logic. It only requires users to add the following dependency at the beginning of the maven dependency:

<dependency>
  <groupId>com.alipay.koupleless</groupId>
  <artifactId>koupleless-base-starter</artifactId>
  <version>${koupleless.runtime.version}</version>
  <type>pom</type>
</dependency>

Therefore, we will adopt solution A.
If you have more ideas or input, welcome to discuss them with the open-source community!

2 - 6.5.3.2 Introduction to Multi-Module Integration Testing Framework

This article focuses on the design concepts, implementation details, and usage of the multi-module integration testing framework.

Why Do We Need a Multi-Module Integration Testing Framework?

Assuming there is no integration testing framework, when developers want to verify whether the deployment process of multiple modules behaves correctly, they need to follow these steps:

1. Build the base and JAR packages for all modules.
2. Start the base process.
3. Install the module JAR packages into the base.
4. Invoke HTTP/RPC interfaces.
5. Verify whether the returned results are correct.

Although the above workflow appears simple, developers face several challenges:

1. Constantly switching back and forth between the command line and the code.
2. If the validation results are incorrect, they need to repeatedly modify the code and rebuild + remote debug.
3. If the app only provides internal methods, they must modify the code to expose interfaces via HTTP/RPC to validate the behavior of the multi-module deployment.

These challenges lead to low efficiency and an unfriendly experience for developers. Therefore, we need an integration testing framework to provide a one-stop validation experience.

What Problems Should the Integration Testing Framework Solve?

The integration testing framework needs to simulate the behavior of multi-module deployment in the same process with a single start. It should also allow developers to directly call code from the modules/base to verify module behavior.

The framework needs to solve the following technical problems:

1. Simulate the startup of the base Spring Boot application.
2. Simulate the startup of module Spring Boot applications, supporting loading modules directly from dependencies instead of JAR packages.
3. Simulate the loading of Ark plugins.
4. Ensure compatibility with Maven’s testing commands.

By default, Sofa-ark loads modules through executable JAR packages and Ark plugins. Therefore, developers would need to rebuild JAR packages or publish to repositories during each validation, reducing validation efficiency. The framework needs to intercept the corresponding loading behavior and load modules directly from Maven dependencies to simulate multi-module deployment.

The code that accomplishes these tasks includes:

1. TestBizClassLoader: Simulates loading the biz module and is a derived class of the original BizClassLoader, solving the problem of loading classes on demand to different ClassLoaders within the same JAR package.
2. TestBiz: Simulates starting the biz module and is a derived class of the original Biz, encapsulating the logic for initializing TestBizClassLoader.
3. TestBootstrap: Initializes ArkContainer and loads Ark plugins.
4. TestClassLoaderHook: Controls the loading order of resources via a hook mechanism. For instance, application.properties in the biz JAR package will be loaded first.
5. BaseClassLoader: Simulates normal base ClassLoader behavior and is compatible with testing frameworks like Surefire.
6. TestMultiSpringApplication: Simulates the startup behavior of multi-module Spring Boot applications.

How to Use the Integration Testing Framework?

Start Both Base and Module Spring Boot Applications in the Same Process

Sample code is as follows:

public void demo() {
    new TestMultiSpringApplication(MultiSpringTestConfig
            .builder()
            .baseConfig(BaseSpringTestConfig
                    .builder()
                    .mainClass(BaseApplication.class) // Base startup class
                    .build())
            .bizConfigs(Lists.newArrayList(
                    BizSpringTestConfig
                            .builder()
                            .bizName("biz1") // Name of module 1
                            .mainClass(Biz1Application.class) // Startup class of module 1
                            .build(),
                    BizSpringTestConfig
                            .builder()
                            .bizName("biz2") // Name of module 2
                            .mainClass(Biz2Application.class) // Startup class of module 2
                            .build()
            ))
            .build()
    ).run();
}

Write Assert Logic

You can retrieve module services using the following method:

public void getService() {
    StrategyService strategyService = SpringServiceFinder.
            getModuleService(
                    "biz1-web-single-host",
                    "0.0.1-SNAPSHOT",
                    "strategyServiceImpl",
                    StrategyService.class
            );
}

After obtaining the service, you can write assert logic.

Reference Use Cases

For more comprehensive use cases, you can refer to Tomcat Multi-Module Integration Testing Cases.

3 - 6.5.3.3 Adapting to Multi-Module with Dubbo 2.7

Why Adaptation is Needed

The native Dubbo 2.7 cannot support module publishing its own Dubbo services in multi-module scenarios, leading to a series of issues such as serialization and class loading exceptions during invocation.

Multi-Module Adaptation Solutions

Dubbo 2.7 Multi-Module Adaptation SDK will be included when building by koupleless-base-build-plugin, the adapter mainly from aspects such as class loading, service publishing, service unloading, service isolation, module-level service management, configuration management, serialization, etc.

1. AnnotatedBeanDefinitionRegistryUtils Unable to Load Module Classes Using the Base Classloader

com.alibaba.spring.util.AnnotatedBeanDefinitionRegistryUtils#isPresentBean

public static boolean isPresentBean(BeanDefinitionRegistry registry, Class<?> annotatedClass) {
    ...

    //        ClassLoader classLoader = annotatedClass.getClassLoader(); // Original logic
    ClassLoader classLoader = Thread.currentThread().getContextClassLoader();   // Changed to use tccl to load classes

    for (String beanName : beanNames) {
        BeanDefinition beanDefinition = registry.getBeanDefinition(beanName);
        if (beanDefinition instanceof AnnotatedBeanDefinition) {
            ...
            String className = annotationMetadata.getClassName();
            Class<?> targetClass = resolveClassName(className, classLoader);
            ...
        }
    }

    return present;
}

2. Module-Level Service and Configuration Resource Management

1. com.alipay.sofa.koupleless.support.dubbo.ServerlessServiceRepository Replaces the Native org.apache.dubbo.rpc.model.ServiceRepository

The native service uses the interfaceName as the cache key. When both the base and the module publish services with the same interface but different groups, it cannot distinguish between them. Replacing the native service caching model, using the Interface Class type as the key, and using the path containing the group as the key to support scenarios where the base and the module publish services with the same interface but different groups.

private static ConcurrentMap<Class<?>, ServiceDescriptor> globalClassServices = new ConcurrentHashMap<>();

private static ConcurrentMap<String, ServiceDescriptor>   globalPathServices  = new ConcurrentHashMap<>();

2. com.alipay.sofa.koupleless.support.dubbo.ServerlessConfigManager Replaces the Native org.apache.dubbo.config.context.ConfigManager

   Adds a classloader dimension key to the original config to isolate different configurations according to classloader in different modules.

final Map<ClassLoader, Map<String, Map<String, AbstractConfig>>> globalConfigsCache = new HashMap<>();

public void addConfig(AbstractConfig config, boolean unique) {
    ...
    write(() -> {
        Map<String, AbstractConfig> configsMap = getCurrentConfigsCache().computeIfAbsent(getTagName(config.getClass()), type -> newMap());
        addIfAbsent(config, configsMap, unique);
    });
}
private Map<String, Map<String, AbstractConfig>> getCurrentConfigsCache() {
    ClassLoader contextClassLoader = Thread.currentThread().getContextClassLoader();   // Based on the current thread classloader to isolate different configuration caches
    globalConfigsCache.computeIfAbsent(contextClassLoader, k -> new HashMap<>());
    return globalConfigsCache.get(contextClassLoader);
}

ServerlessServiceRepository and ServerlessConfigManager both depend on the dubbo ExtensionLoader’s extension mechanism to replace the original logic. For specific principles, please refer to org.apache.dubbo.common.extension.ExtensionLoader.createExtension.

3. Module-Level Service Install and Uninstall

override DubboBootstrapApplicationListener to prevent the original Dubbo module from starting or uninstalling when publishing or uninstalling services

• com.alipay.sofa.koupleless.support.dubbo.BizDubboBootstrapListener

The native Dubbo 2.7 only publishes Dubbo services after the base module is started. In the case of multi-modules, it cannot support module-level service publishing. Ark listens for module startup events using a listener and manually calls Dubbo to publish module-level services.

private void onContextRefreshedEvent(ContextRefreshedEvent event) {
  try {
      ReflectionUtils.getMethod(DubboBootstrap.class, "exportServices")
          .invoke(dubboBootstrap);
      ReflectionUtils.getMethod(DubboBootstrap.class, "referServices").invoke(dubboBootstrap);
  } catch (Exception e) {
      
  }
}

The original Dubbo 2.7 unexports all services in the JVM when a module is uninstalled, leading to the unexporting of services from the base and other modules after the module is uninstalled. Ark listens for the spring context closing event of the module and manually unexports Dubbo services of the current module, retaining Dubbo services of the base and other modules.

private void onContextClosedEvent(ContextClosedEvent event) {
        // DubboBootstrap.unexportServices unexports all services, only need to unexport services of the current biz
        Map<String, ServiceConfigBase<?>> exportedServices = ReflectionUtils.getField(dubboBootstrap, DubboBootstrap.class, "exportedServices");

        Set<String> bizUnexportServices = new HashSet<>();
        for (Map.Entry<String, ServiceConfigBase<?>> entry : exportedServices.entrySet()) {
            String serviceKey = entry.getKey();
            ServiceConfigBase<?> sc = entry.getValue();
            if (sc.getRef().getClass().getClassLoader() == Thread.currentThread().getContextClassLoader()) {   // Distinguish module services based on the classloader of ref service implementation
                bizUnexportServices.add(serviceKey);
                configManager.removeConfig(sc);   // Remove service configuration from configManager
                sc.unexport();   // Unexport service
                serviceRepository.unregisterService(sc.getUniqueServiceName());   // Remove from serviceRepository
            }
        }
        for (String service : bizUnexportServices) {
            exportedServices.remove(service);    // Remove service from DubboBootstrap
        }
    }

4. Service Routing

• com.alipay.sofa.koupleless.support.dubbo.ConsumerRedefinePathFilter

When invoking Dubbo services, the service model (including interface, param, return types, etc.) is obtained from the ServiceRepository based on the path to perform service invocation, parameter, and return value serialization. The original Dubbo 2.7 uses interfaceName as the path to find the service model, which cannot support the scenario where the base module and other modules publish services with the same interface. Ark adds group information to the path on the consumer side through a custom filter to facilitate correct service routing on the provider side.

public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
  if (invocation instanceof RpcInvocation) {
      RpcInvocation rpcInvocation = (RpcInvocation) invocation;
      // Original path is interfaceName, such as com.alipay.sofa.rpc.dubbo27.model.DemoService
      // Modified path is serviceUniqueName, such as masterBiz/com.alipay.sofa.rpc.dubbo27.model.DemoService
      rpcInvocation.setAttachment("interface", rpcInvocation.getTargetServiceUniqueName());   // Original path is interfaceName, such as
  }
  return invoker.invoke(invocation);
}

5. Serialization

• org.apache.dubbo.common.serialize.java.JavaSerialization
• org.apache.dubbo.common.serialize.java.ClassLoaderJavaObjectInput
• org.apache.dubbo.common.serialize.java.ClassLoaderObjectInputStream

When obtaining the serialization tool JavaSerialization, use ClassLoaderJavaObjectInput instead of the original JavaObjectInput and pass provider-side service classloader information.

// org.apache.dubbo.common.serialize.java.JavaSerialization
public ObjectInput deserialize(URL url, InputStream is) throws IOException {
    return new ClassLoaderJavaObjectInput(new ClassLoaderObjectInputStream(null, is));   // Use ClassLoaderJavaObjectInput instead of the original JavaObjectInput, pass provider-side service classloader information
}

// org.apache.dubbo.common.serialize.java.ClassLoaderObjectInputStream
private ClassLoader classLoader;

public ClassLoaderObjectInputStream(final ClassLoader classLoader, final InputStream inputStream) {
  super(inputStream);
  this.classLoader = classLoader;
}

• org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcResult Client-side deserialization of return values

// patch begin
if (in instanceof ClassLoaderJavaObjectInput) {
   InputStream is = ((ClassLoaderJavaObjectInput) in).getInputStream();
   if (is instanceof ClassLoaderObjectInputStream) {
      ClassLoader cl = serviceDescriptor.getServiceInterfaceClass().getClassLoader();  // Set provider-side service classloader information to ClassLoaderObjectInputStream
      ((ClassLoaderObjectInputStream) is).setClassLoader(cl);
   }
}
// patch end

• org.apache.dubbo.rpc.protocol.dubbo.DecodeableRpcResult Client-side deserialization of return values

// patch begin
if (in instanceof ClassLoaderJavaObjectInput) {
   InputStream is = ((ClassLoaderJavaObjectInput) in).getInputStream();
   if (is instanceof ClassLoaderObjectInputStream) {
       ClassLoader cl = invocation.getInvoker().getInterface().getClassLoader(); // Set consumer-side service classloader information to ClassLoaderObjectInputStream
       ((ClassLoaderObjectInputStream) is).setClassLoader(cl);
   }
}
// patch end

Example of Using Dubbo 2.7 in a Multi-Module Environment

Example of Using Dubbo 2.7 in a Multi-Module Environment

dubbo2.7 Multi-Module Adaptation SDK Source Code

4 - 6.5.3.4 Best Practices for Multi-Module with ehcache

Why Best Practices are Needed

During CacheManager initialization, there are shared static variables causing issues when multiple applications use the same Ehcache name, resulting in cache overlap.

Requirements for Best Practices

1. Base module must include Ehcache, and modules should reuse the base.

In Spring Boot, Ehcache initialization requires creating it through the EhCacheCacheConfiguration defined in Spring, which belongs to Spring and is usually placed in the base module.

During bean initialization, the condition check will lead to class verification, if net.sf.ehcache.CacheManager is found, it will use a Java native method to search for the net.sf.ehcache.CacheManager class in the ClassLoader where the class belongs. Therefore, the base module must include this dependency; otherwise, it will result in ClassNotFound errors.

2. Modules should exclude the included Ehcache (set scope to provided or utilize automatic slimming capabilities).

When a module uses its own imported Ehcache, theoretically, it should avoid sharing static variables in the base CacheManager class, thus preventing potential errors. However, in our actual testing, during the module installation process, when initializing the EhCacheCacheManager, we encountered an issue where, during the creation of a new object, it required obtaining the CacheManager belonging to the class of the object, which in turn should be the base CacheManager. Importantly, we cannot include the CacheManager dependency in the module’s compilation, as it would lead to conflicts caused by a single class being imported by multiple different ClassLoaders.

When a module uses its own imported Ehcache, theoretically, it should avoid sharing static variables in the base CacheManager class, thus preventing potential errors. However, in our actual testing, during the module installation process, when initializing the EhCacheCacheManager, we encountered an issue where, during the creation of a new object, it required obtaining the CacheManager belonging to the class of the object, which in turn should be the base CacheManager. Importantly, we cannot include the CacheManager dependency in the module’s compilation, as it would lead to conflicts caused by a single class being imported by multiple different ClassLoaders.

Therefore, all loading should be delegated to the base module.

Best Practice Approach

1. Delegate module Ehcache slimming to the base.
2. If multiple modules have the same cacheName, modify cacheName to be different.
3. If you don’t want to change the code to modify cache name, you can dynamically replace cacheName through packaging plugins.

 <plugin>
    <groupId>com.google.code.maven-replacer-plugin</groupId>
    <artifactId>replacer</artifactId>
    <version>1.5.3</version>
    <executions>
        <!-- Perform replacement before packaging -->
        <execution>
            <phase>prepare-package</phase>
            <goals>
                <goal>replace</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <!-- Automatically recognize the project's target folder -->
        <basedir>${build.directory}</basedir>
        <!-- Directory rules for replacement files -->
        <includes>
            <include>classes/j2cache/*.properties</include>
        </includes>
        <replacements>
            <replacement>
                <token>ehcache.ehcache.name=f6-cache</token>
                <value>ehcache.ehcache.name=f6-${parent.artifactId}-cache</value>
            </replacement>

        </replacements>
    </configuration>
</plugin>

4. Set the shared property of the FactoryBean to false.

@Bean
    public EhCacheManagerFactoryBean ehCacheManagerFactoryBean() {
        EhCacheManagerFactoryBean factoryBean = new EhCacheManagerFactoryBean();

        // Set the factoryBean's shared property to false
        factoryBean.setShared(true);
//        factoryBean.setShared(false);
        factoryBean.setCacheManagerName("biz1EhcacheCacheManager");
        factoryBean.setConfigLocation(new ClassPathResource("ehcache.xml"));
        return factoryBean;
    }

Otherwise, it will enter this logic, initializing the static variable instance of CacheManager. If this variable has a value, and if shared is true in the module, it will reuse the CacheManager’s instance, leading to errors.

Example of Best Practices

For an example project, pleaserefer to here

5 - 6.5.3.5 Logback's adaptation for multi-module environments

Why Adaptation is Needed

The native logback framework only provides a default logging context, making it impossible to isolate log configurations between different modules. Consequently, in scenarios involving deploying multiple modules together, modules can only utilize the logging configuration of the base application, causing inconvenience when logging from individual modules.

Multi-Module Adaptation Solution

Logback supports native extension ch.qos.logback.classic.selector.ContextSelector, which allows for a custom context selector. Ark provides a default implementation of ContextSelector to isolate LoggerContext for multiple modules (refer to com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector). Each module uses its independent LoggerContext, ensuring log configuration isolation.

During startup, the log configuration and context initialization are handled by Spring’s log system LogbackLoggingSystem.

Specify the context selector as com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector and add the JVM startup parameter:

-Dlogback.ContextSelector=com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector

When using SLF4J as the logging facade with logback as the logging implementation framework, during the base application startup, when the SLF4J static binding is first performed, the specific ContextSelector is initialized. If no custom context selector is specified, the DefaultContextSelector will be used. However, when we specify a context selector, the ArkLogbackContextSelector will be initialized as the context selector.

ch.qos.logback.classic.util.ContextSelectorStaticBinder.init

public void init(LoggerContext defaultLoggerContext, Object key) {
  ...

  String contextSelectorStr = OptionHelper.getSystemProperty(ClassicConstants.LOGBACK_CONTEXT_SELECTOR);
  if (contextSelectorStr == null) {
  contextSelector = new DefaultContextSelector(defaultLoggerContext);
  } else if (contextSelectorStr.equals("JNDI")) {
  // if jndi is specified, let's use the appropriate class
  contextSelector = new ContextJNDISelector(defaultLoggerContext);
  } else {
  contextSelector = dynamicalContextSelector(defaultLoggerContext, contextSelectorStr);
  }
}

static ContextSelector dynamicalContextSelector(LoggerContext defaultLoggerContext, String contextSelectorStr) {
  Class<?> contextSelectorClass = Loader.loadClass(contextSelectorStr);
  Constructor cons = contextSelectorClass.getConstructor(new Class[] { LoggerContext.class });
  return (ContextSelector) cons.newInstance(defaultLoggerContext);
}

In the ArkLogbackContextSelector, we utilize ClassLoader to differentiate between different modules and cache the LoggerContext of each module based on its ClassLoader

When obtaining the LoggerContext based on the ClassLoader, during the startup of the Spring environment, the logging context is initialized via the Spring logging system. This is achieved by calling org.springframework.boot.logging.logback.LogbackLoggingSystem.getLoggerContext, which returns the LoggerContext specific to each module using the custom context selector implemented by Ark, com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector.getLoggerContext().

public LoggerContext getLoggerContext() {
  ClassLoader classLoader = this.findClassLoader();
  if (classLoader == null) {
      return defaultLoggerContext;
  }
  return getContext(classLoader);
}

When obtaining the classloader, first, the thread’s context classloader is retrieved. If it is identified as the classloader of the module, it is returned directly. If the TCCL (thread context classloader) is not the classloader of the module, the call stack of the Class objects is traversed through the ClassContext. When encountering the classloader of the module in the call stack, it is returned directly. This approach is taken to accommodate scenarios where the TCCL is not guaranteed to be the classloader of the module. For example, when logging is performed in module code, and the current class is loaded by the module’s classloader itself, traversing the ClassContext allows us to eventually obtain the classloader of the module, ensuring the use of the module-specific LoggerContext.

private ClassLoader findClassLoader() {
  ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
  if (classLoader != null && CONTAINER_CLASS_LOADER.equals(classLoader.getClass().getName())) {
      return null;
  }
  if (classLoader != null && BIZ_CLASS_LOADER.equals(classLoader.getClass().getName())) {
      return classLoader;
  }

  Class<?>[] context = new SecurityManager() {
      @Override
      public Class<?>[] getClassContext() {
          return super.getClassContext();
      }
  }.getClassContext();
  if (context == null || context.length == 0) {
      return null;
  }
  for (Class<?> cls : context) {
      if (cls.getClassLoader() != null
          && BIZ_CLASS_LOADER.equals(cls.getClassLoader().getClass().getName())) {
          return cls.getClassLoader();
      }
  }

  return null;
}

Once the appropriate ClassLoader is obtained, different LoggerContext instances are selected. All module contexts are cached in com.alipay.sofa.ark.common.adapter.ArkLogbackContextSelector.CLASS_LOADER_LOGGER_CONTEXT with the ClassLoader as the key.

private LoggerContext getContext(ClassLoader cls) {
  LoggerContext loggerContext = CLASS_LOADER_LOGGER_CONTEXT.get(cls);
  if (null == loggerContext) {
      synchronized (ArkLogbackContextSelector.class) {
          loggerContext = CLASS_LOADER_LOGGER_CONTEXT.get(cls);
          if (null == loggerContext) {
              loggerContext = new LoggerContext();
              loggerContext.setName(Integer.toHexString(System.identityHashCode(cls)));
              CLASS_LOADER_LOGGER_CONTEXT.put(cls, loggerContext);
          }
      }
  }
  return loggerContext;
}

Sample Usage of Multi-Module Logback

Sample Usage of Multi-Module Logback

View Source Code of ArkLogbackContextSelector

6 - 6.5.3.6 log4j2 Multi-Module Adaptation

Why Adaptation is Needed

In its native state, log4j2 does not provide individual log directories for modules in a multi-module environment. Instead, it logs uniformly to the base directory, which makes it challenging to isolate logs and corresponding monitoring for each module. The purpose of this adaptation is to enable each module to have its own independent log directory.

Initialization of log4j2 in Regular Applications

Before Spring starts, log4j2 initializes various logContexts and configurations using default values. During the Spring startup process, it listens for Spring events to finalize initialization. This process involves invoking the Log4j2LoggingSystem.initialize method via org.springframework.boot.context.logging.LoggingApplicationListener.

The method determines whether it has already been initialized based on the loggerContext.

Here, a problem arises in a multi-module environment.

The getLoggerContext method retrieves the LoggerContext based on the classLoader of org.apache.logging.log4j.LogManager. Relying on the classLoader of a specific class to extract the LoggerContext can be unstable in a multi-module setup. This instability arises because some classes in modules can be configured to delegate loading to the base, so when a module starts, it might obtain the LoggerContext from the base. Consequently, if isAlreadyInitialized returns true, the log4j2 logging for the module cannot be further configured based on user configuration files.

If it hasn’t been initialized yet, it enters super.initialize, which involves two tasks:

1. Retrieving the log configuration file.
2. Parsing the variable values in the log configuration file. Both of these tasks may encounter issues in a multi-module setup. Let’s first examine how these two steps are completed in a regular application.

Retrieving the Log Configuration File

You can see that the location corresponding to the log configuration file’s URL is obtained through ResourceUtils.getURL. Here, the URL is obtained by retrieving the current thread’s context ClassLoader, which works fine in a multi-module environment (since each module’s startup thread context is already its own ClassLoader).

Parsing Log Configuration Values

The configuration file contains various variables, such as these:

These variables are parsed in the specific implementation of org.apache.logging.log4j.core.lookup.AbstractLookup, including:

Based on the above analysis, configuring via bundle method might not be feasible in a multi-module setup because ResourceBundleLookup might only exist in the base module, leading to always obtaining application.properties from the base module. Consequently, the logging configuration path of the modules would be the same as that of the base module, causing all module logs to be logged into the base module. Therefore, it needs to be modified to use ContextMapLookup.

Expected Logging in a Multi-Module Consolidation Scenario

Both the base module and individual modules should be able to use independent logging configurations and values, completely isolated from each other. However, due to the potential issues identified in the analysis above, which could prevent module initialization, additional adaptation of log4j2 is required.

Multi-Module Adaptation Points

1. Ensure getLoggerContext() can retrieve the LoggerContext of the module itself.

2. It’s necessary to adjust to use ContextMapLookup so that module logs can retrieve the module application name and be logged into the module directory.

   a. Set the values of application.properties to ThreadContext when the module starts. b. During logging configuration, only use the ctx:xxx:xxx configuration format.

Module Refactoring Approach

Check the source code for detailed information

7 - 6.5.3.7 Module Use Bes

koupleless-adapter-bes

koupleless-adapter-bes is used to adapt to the BaoLande (BES) container, the warehouse address is koupleless-adapter-bes (thanks to the community student Chen Jian for his contribution).

The project is currently only verified in BES 9.5.5.004 version, and other versions need to be verified by themselves, and necessary adjustments need to be made according to the same logic.

If multiple BIZ modules do not need to use the same port to publish services, only need to pay attention to the precautions mentioned in the installation dependency section below, and do not need to introduce the dependencies related to this project.

Quick Start

1. Install Dependencies

First, make sure that BES-related dependencies have been imported into the maven repository. (There is a key point here. Due to the conflicting package structure of BES’s dependency package with the recognition mechanism of the koupleless 2.2.9 project, users need to add the prefix sofa-ark- to the BES’s dependency package by themselves, and the specific recognition mechanism can refer to koupleless’ com.alipay.sofa.ark.container.model. BizModel class)

The reference import script is as follows:

mv XXX/BES-EMBED/bes-lite-spring-boot-2.x-starter-9.5.5.004.jar XXX/BES-EMBED/sofa-ark-bes-lite-spring-boot-2.x-starter-9.5.5.004.jar
mvn install:install-file -Dfile=XXX/BES-EMBED/sofa-ark-bes-lite-spring-boot-2.x-starter-9.5.5.004.jar -DgroupId=com.bes.besstarter -DartifactId=sofa-ark-bes-lite-spring-boot-2.x-starter -Dversion=9.5.5.004 -Dpackaging=jar
mvn install:install-file -Dfile=XXX/BES-EMBED/bes-gmssl-9.5.5.004.jar -DgroupId=com.bes.besstarter -DartifactId=bes-gmssl -Dversion=9.5.5.004 -Dpackaging=jar
mvn install:install-file -Dfile=XXX/BES-EMBED/bes-jdbcra-9.5.5.004.jar -DgroupId=com.bes.besstarter -DartifactId=bes-jdbcra -Dversion=9.5.5.004 -Dpackaging=jar
mvn install:install-file -Dfile=XXX/BES-EMBED/bes-websocket-9.5.5.004.jar -DgroupId=com.bes.besstarter -DartifactId=bes-websocket -Dversion=9.5.5.004 -Dpackaging=jar

2. Compile and Install the Project Plugin

Enter the bes9-web-adapter directory of the project and execute the mvn install command.

The project will install the “bes-web-ark-plugin” and “bes-sofa-ark-springboot-starter” two modules.

3. Use the Project Components

First, according to the koupleless documentation, upgrade the project to Koupleless Base

Then, replace the coordinates mentioned in the dependencies

<dependency>
    <groupId>com.alipay.sofa</groupId>
    <artifactId>web-ark-plugin</artifactId>
    <version>${sofa.ark.version}</version>
</dependency>

with the coordinates of this project

<dependency>
    <groupId>com.alipay.sofa</groupId>
    <artifactId>bes-web-ark-plugin</artifactId>
    <version>2.2.9</version>
</dependency>
<dependency>
   <groupId>com.alipay.sofa</groupId>
   <artifactId>bes-sofa-ark-springboot-starter</artifactId>
   <version>2.2.9</version>
</dependency>

Introduce BES-related dependencies (also need to exclude the dependency of tomcat). The reference dependence is as follows:

       <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.bes.besstarter</groupId>
            <artifactId>sofa-ark-bes-lite-spring-boot-starter</artifactId>
            <version>9.5.5.004</version>
        </dependency>
        <dependency>
            <groupId>com.bes.besstarter</groupId>
            <artifactId>bes-gmssl</artifactId>
            <version>9.5.5.004</version>
        </dependency>
        <dependency>
            <groupId>com.bes.besstarter</groupId>
            <artifactId>bes-jdbcra</artifactId>
            <version>9.5.5.004</version>
        </dependency>
        <dependency>
            <groupId>com.bes.besstarter</groupId>
            <artifactId>bes-websocket</artifactId>
            <version>9.5.5.004</version>
        </dependency>

4. Finished

After completing the above steps, you can start the project in Koupleless using BES.

8 - 6.5.3.8 Module Using Dubbo

Module Interceptor (Filter)

A module can use interceptors defined within itself or those defined on the base.

⚠️Note: Avoid naming module interceptors the same as those on the base. If the names are identical, the interceptors from the base will be used.

9 - 6.5.3.10 Introduction to the Principle of Class Delegation Loading between Foundation and Modules

Class Delegation Loading between Multiple Modules

The SOFAArk framework is based on a multi-ClassLoader universal class isolation solution, providing class isolation and application merge deployment capabilities. This document does not intend to introduce the principles and mechanismsof SOFAArk class isolation. Instead, it mainly introduces the current best practices of multi-ClassLoader.
The ClassLoader model between the foundation and modules deployed on the JVM at present is as shown in the figure below:

Current Class Delegation Loading Mechanism

The classes searched by a module during startup and runtime currently come from two sources: the module itself and the foundation. The ideal priority order of these two sources is to search from the module first, and if not found, then from the foundation. However, there are some exceptions currently:

1. A whitelist is defined, and dependencies within the whitelist are forced to use dependencies in the foundation.
2. The module can scan all classes in the foundation:
   • Advantage: The module can introduce fewer dependencies.
   • Disadvantage: The module will scan classes that do not exist in the module code, such as some AutoConfigurations. During initialization, errors may occur due to the inability to scan corresponding resources.
3. The module cannot scan any resources in the foundation:
   • Advantage: It will not initialize the same beans as the foundation repeatedly.
   • Disadvantage: If the module needs resources from the foundation to start, errors will occur due to the inability to find resources unless the module is explicitly introduced (Maven dependency scope is not set to provided).
4. When the module calls the foundation, some internal processes pass the class names from the module to the foundation. If the foundation directly searches for the classes passed by the module from the foundation ClassLoader, it will not find them. This is because delegation only allows the module to delegate to the foundation, and classes initiated from the foundation will not search the module again.

Points to Note When Using

When a module needs to upgrade the dependencies delegated to the foundation, the foundation needs to be upgraded first, and then the module can be upgraded.

Best Practices for Class Delegation

The principle of class delegation loading is that middleware-related dependencies need to be loaded and executed in the same ClassLoader. There are two best practices to achieve this:

Mandatory Delegation Loading

Since middleware-related dependencies generally need to be loaded and executed in the same ClassLoader, we will specify a whitelist of middleware dependency, forcing these dependencies to be delegated to the foundation for loading.

Usage

Add the configuration sofa.ark.plugin.export.class.enable=true to application.properties.

Advantages

Module developers do not need to be aware of which dependencies belong to the middleware that needs to be loaded by the same ClassLoader.

Disadvantages

The list of dependencies to be forcibly loaded in the whitelist needs to be maintained. If the list is missing, the foundation needs to be updated. Important upgrades require pushing all foundation upgrades.

Custom Delegation Loading

In the module’s pom, set the scope of the dependency to provided to actively specify which dependencies to delegate to the foundation for loading. By slimming down the module, duplicate dependencies with the foundation are delegated to the foundation for loading, and middleware dependencies are pre-deployed in the foundation (optional, although the module may not use them temporarily, they can be introduced in advance in case they are needed by subsequent modules without the need to redeploy the foundation). Here:

1. The foundation tries to precipitate common logic and dependencies, especially those related to middleware named xxx-alipay-sofa-boot-starter.
2. Pre-deploy some common dependencies in the foundation (optional).
3. If the dependencies in the module are already defined in the foundation, the dependencies in the module should be delegated to the foundation as much as possible. This will make the module lighter (providing tools for automatic module slimming). There are two ways for the module to delegate to the foundation:
   1. Set the scope of the dependency to provided, and check whether there are other dependencies set to compile through mvn dependency:tree, and all places where dependencies are referenced need to be set to provided.
   2. Set excludeGroupIds or excludeArtifactIds in the sofa-ark-maven-plugin biz packaging plugin.

            <plugin>
                <groupId>com.alipay.sofa</groupId>
                <artifactId>sofa-ark-maven-plugin</artifactId>
                <configuration> 
                    <excludeGroupIds>io.netty,org.apache.commons,......</excludeGroupIds>
                    <excludeArtifactIds>validation-api,fastjson,hessian,slf4j-api,junit,velocity,......</excludeArtifactIds>
                    <declaredMode>true</declaredMode>
                </configuration>
            </plugin>

Using Method 2.b To ensure that all declarations are set to provided scope, it is recommended to use Method 2.b, where you only need to specify once.

4. Only dependencies declared by the module can be delegated to the foundation for loading.

During module startup, the Spring framework has some scanning logic. If these scans are not restricted, they will search for all resources of both the module and the foundation, causing some modules to attempt to initialize functions they clearly do not need, resulting in errors. Since SOFAArk 2.0.3, the declaredMode of modules has been added to limit only dependencies declared within the module can be delegated to the foundation for loading. Simply add <declaredMode>true</declaredMode> to the module’s packaging plugin configurations.

Advantages

• No need to maintain a forced loading list for plugins. When some dependencies that need to be loaded by the same ClassLoader are not set for uniform loading, you can fix them by modifying the module without redeploying the foundation (unless the foundation does require it).

Disadvantages

• Strong dependency on slimming down modules.

Comparison and Summary

Conclusion: Recommend Custom Delegation Loading Method

1. Module custom delegation loading + module slimming.
2. Module enabling declaredMode.
3. Preload dependencies in the base.

declaredMode 开启方式

declaredMode Activation Procedure

Activation Conditions

The purpose of declaredMode is to enable modules to be deployed to the foundation. Therefore, before activation, ensure that the module can start locally successfully.
If it is a SOFABoot application and involves module calls to foundation services, local startup can be skipped by adding these two parameters to the module’s application.properties (SpringBoot applications do not need to care):

# If it is SOFABoot, then:
# Configure health check to skip JVM service check
com.alipay.sofa.boot.skip-jvm-reference-health-check=true
# Ignore unresolved placeholders
com.alipay.sofa.ignore.unresolvable.placeholders=true

Activation Method

Add the following configuration to the module’s packaging plugin:

Side Effects After Activation

If the dependencies delegated to the foundation by the module include published services, then the foundation and the module will publish two copies simultaneously.

10 - 6.3.5.11 What happens if a module independently introduces part of the SpringBoot framework?

Since the logic of multi-module runtime is introduced and loaded in the base, such as some Spring Listeners. If the module starts using its own SpringBoot entirely, there may be some class conversion or assignment judgment failures, for example:

CreateSpringFactoriesInstances

name = ‘com.alipay.sofa.ark.springboot.listener.ArkApplicationStartListener’, ClassUtils.forName gets the class from the base ClassLoader.

However, the type is loaded when the module starts, which means it is loaded using the module’s BizClassLoader.

At this point, performing an isAssignable check here will cause an error.

com.alipay.sofa.koupleless.plugin.spring.ServerlessApplicationListener is not assignable to interface org.springframework.context.ApplicationListener

So the module framework part needs to be delegated to the base to load.