Infrared and visible image fusion based on multi-level detail enhancement and generative adversarial network

2.1. GANs

GANs are deep learning models proposed by Goodfellow et al. in 2014, evolving from the minimum binary zero-sum game in free game theory ^[13]. Unlike traditional deep neural networks, GANs consist of two components: generator (G) and discriminator (D). The generator receives random noise input and learns specific data feature distribution. The discriminator takes the generated results of the generator and real samples as input, determines whether the input data belongs to the real sample or the false sample, and feeds the judgment result back to the generator. These two components are trained alternately. In the constant confrontation game, they compete and drive each other to improve. During training, the parameters are continuously optimized until the generator and discriminator become indistinguishable, reaching a Nash equilibrium state.

In essence, the generator model belongs to the great likelihood estimation under the machine learning branch, which generates the original data distribution into the specified data by capturing the distribution of the sample data and adopting the parameter transformation calculation method in the great likelihood estimation. The generator model can be expressed as

where $$ z $$ is the random input noise, obeying a multivariate Gaussian distribution sampling. $$ \theta^{(G)} $$ are the generator network parameters, and the network is passed through a series of nonlinear computations to obtain the output sample $$ x $$. The essence of the discriminator is a binary classification model. It takes the generated sample $$ x $$ and real sample data as input, calculates the discriminant probability, and judges whether the sample is true or false. The discriminator can be expressed as

where $$ \theta^{(D)} $$ is the network parameter of the discriminator and $$ y $$ is the output label, that is, the probability of discrimination.

The ultimate goal of GAN is to realize that the generated sample is infinitely close to the real data, so that the two are indistinguishable. The confrontation process between the two is expressed as

where $$ x $$ is the input data, $$ P_{data} $$ is the real data distribution, $$ z $$ is the input noise, and $$ P_{z} $$ is the prior variable of the input noise. It can be seen from Equation (3) that the purpose of the generator is to minimize the objective function $$ V(G,D) $$, making it difficult for the discriminator to distinguish real samples from the generated samples. The discriminator aims to improve the ability to distinguish between true and false samples, thereby maximizing the objective function $$ V(G,D) $$. The two are trained alternately to optimize their own capabilities, and the final iteration is relatively stable.

2.2. Deep feature extraction

2.2.1. Residual dense block

As the depth of the network increases, the features of each convolutional layer have hierarchical structures with different receptive fields, and the traditional convolutional network structure cannot fully utilize the information. In order to fully utilize the hierarchical features of all convolutional layers, Zhang et al. proposed a new feature extraction structure, named residual dense block (RDB) ^[30]. The structure extracts rich local features through convolutional layers densely connected, establishes dense connections between the front and the back layers, and reuses features in the channel dimension. Combined with residual connections, it enhances the information flow and ensures the transmission and coherence of feature information, effectively slowing down the phenomenon of gradient disappearance and enabling the network to achieve better performance with fewer parameters and calculations. The structure of the RDB module is shown in Figure 1.

Figure 1. The overall structure of the dense residual block. 3 × 3: filter size. Conv: Convolutional layer; Concat: establish dense connections between the front and back layers; BN: batch normalization.

The RDB module contains both dense and residual structures. Firstly, through the dense layer with N convolution kernel sizes of 3 × 3, the local feature learning can be realized effectively through the jump connection between Concat and the convolution layer in front. Then, the 1 × 1 convolution kernel is used for dimension reduction to prepare for the subsequent feature fusion. Finally, the global residual connection is used to improve the information flow and realize the fusion of local features and residual features.

2.2.2. Multi-scale residual blocks

In many visual tasks, it is very important to extract multi-scale features. Gao et al. proposed a novel multi-scale feature extraction method which is multi-scale residual block Res2Net in 2021 ^[31]. Its structure is shown in Figure 2. As a variant of the residual network ResNet, Res2Net inserts more hierarchical residual connection structures into the residual unit. Unlike the typical multi-scale improvement of general architecture, this approach combines scales based on hierarchical structures. It performs multi-scale extensions with hierarchical and layered feature sets in a given block. This module divides the input into several equally sized groups. The first group is passed directly through, while the remaining groups are processed by the convolution kernel and combined with the next group. This process continues, and the processed features are finally concatenated.

Figure 2. Schematic diagram of the structure of a multi-scale residual block. 3 × 3: filter size. Conv: Convolutional layer; Concat: establish dense connections between the front and back layers; BN: batch normalization. ($$ xi $$ expresses the N subsets into which it is divided after the first 1 × 1 convolution and $$ yi $$ expresses the N set of features obtained after $$ xi $$ undergoes a 3 × 3 convolution operation.)

As shown in Figure 2, after the first 1 × 1 convolution, it is divided into N subsets. Each subset can potentially receive all the feature information on its left after a 3 × 3 convolution operation, increase the receptive field, and obtain feature combinations with different quantities and sizes of receptive fields, so that the receptive field can represent multi-scale features at a finer granularity level. Finally, N groups of features are spliced through Concat, sent to 1 × 1 convolution, and then combined with the front residual connection to realize the fusion of feature information at various scales.

3.1. Overall network framework

The overall framework of the fusion network is shown in Figure 3, which mainly includes three parts: image preprocessing, generator and discriminator.

Figure 3. The overall framework of the fusion network.

First of all, the image preprocessing part mainly uses a multi-level detail enhancement method to enhance the edge details and other information of infrared and visible images, so that the input source image has more detailed information, which is beneficial to neural network processing. Secondly, Concat pairs of the registered infrared and visible images are used as the input of the generator for feature extraction and fusion image reconstruction. And the dual discriminator form of infrared discriminator D-ir and visible light discriminator D-vis is used to distinguish the infrared and visible images from the fusion image output by the generator, respectively, perceiving the feature differences. Ultimately, by engaging in an adversarial game with the generator and under the constraints imposed by the loss function, we can iteratively enrich and integrate image details to produce high-quality fused images that align with our expectations.

3.2. Multi-level detail enhancement module

In order to improve the quality of input source images and enhance the ability of neural networks to learn details, we can use the multi-level detail enhancement module to enhance the detail edge information of input source images. A method of multi-scale detail enhancement was proposed in literature ^[32], which decomposed the image into multiple scales through Gaussian filter kernel of different sizes, and fused detail information into the original image through a certain combination. Although this method could improve local details, it had limitations in improving global details and enhancing the balance between them. On this basis, this paper proposed a multi-level detail enhancement method based on guided image filtering, which uses multiple levels to enhance image details without producing a gray saturation artifact phenomenon or excessive noise.

Guided filtering, as a nonlinear filtering technique, is widely used for edge preservation and image denoising. Unlike conventional denoising methods, which often struggle to distinguish between image edges and noise, guided filtering effectively preserves edges while suppressing noise. This is achieved by leveraging the structural information of a guidance image to adjust pixel weights within each region, thereby enabling image smoothing while retaining fine edge details. These characteristics make guided filtering highly effective in tasks such as image denoising, enhancement, and edge preservation.

In traditional guided filtering, the edge-preserving coefficient within any arbitrary window is fixed, which has a significant impact on the output image. A coefficient value that is too small may result in noticeable noise in the output image, whereas a value that is too large may lead to over-smoothing of the image. This study leverages the intrinsic properties of the edge-preserving coefficient in guided filtering by setting different filter sizes and edge-preserving coefficients to appropriate values. Through iterative and recursive filtering, this approach generates output images at various structural levels, enabling detailed and nuanced multi-level enhancement.

Firstly, the input image is processed through guided filtering and decomposed into different levels of background images, as given in

where $$ I $$ is the input image, and $$ B_{1} $$, $$ B_{2} $$ and $$ B_{3} $$ are background images at different levels. $$ \sigma $$ and $$ \varepsilon $$ determine the filter size and edge retention coefficient, respectively. The values of ($$ \sigma $$, $$ \varepsilon $$) of the corresponding background images $$ B_{1} $$, $$ B_{2} $$ and $$ B_{3} $$ are set to (3, 0.01), (9, 0.1) and (15, 0.5) respectively. The input image $$ I $$, along with background images $$ B_{1} $$ and $$ B_{2} $$, is used as guide images, and each undergoes filtering to produce smooth background images $$ B_{1} $$, $$ B_{2} $$ and $$ B_{3} $$.

Secondly, different layers of detail are extracted to prepare for subsequent enhancement; the detail layers are extracted by

where $$ I $$ is the input image, and $$ D_{1} $$, $$ D_{2} $$ and $$ D_{3} $$ represent high-, medium- and low-level details, respectively.

Finally, by using special weighting factors of set proportions, the details at levels $$ D_{1} $$, $$ D_{2} $$, and $$ D_{3} $$, which correspond to fine details, moderate details, and large-scale edge structure details, are integrated into the original image through

where $$ I^{*} $$ is the enhanced image, $$ w_{1} $$, $$ w_{2} $$ and $$ w_{3} $$ are weight coefficients, with values of 0.4, 0.25 and 0.15 through many tests, and sat ($$ \cdot $$) represents the saturation function, defined as

where $$ \mathrm{\Delta} $$ is the boundary threshold. As $$ \mathrm{\Delta} $$ approaches infinity, sat($$ \cdot $$) is approximated as a symbolic function sgn($$ \cdot $$). On the one hand, the sat($$ \cdot $$) function is used to balance the positive and negative components of fine details $$ D_{1} $$ to prevent artifacts caused by excessive grayscale saturation; on the other hand, it can also control the boundary threshold $$ \mathrm{\Delta} $$ to suppress a certain degree of noise caused by adding details, ensuring the balance of global and local details.

3.3. Generator and discriminator

3.3.1. Generator structure

The generator structure is shown in Figure 4. The overall structure of the model can be divided into two parts: the encoder (Encoder) and the decoder (Decoder). The encoder mainly extracts the feature information of the image; firstly, two ordinary convolutional layers are used to extract the shallow feature, and then, the shallow feature is sent into the RDB and the multi-scale residual block Res2Net. The RDB module adopts three dense layers with a convolution kernel size of 3 × 3; the number of subsets N divided by the Res2Net module is selected as 4. It achieved the extraction of multi-scale features and the fusion of deep features, ensuring the richness of local features while also enhancing the circulation and transmission of receptive fields and feature information. Finally, the output of the first ordinary convolutional layer is short-circuited with the output of the Res2Net module by using global residual connection. Through the application of three convolutional layers with kernel sizes of 1 × 1, 3 × 3, and 1 × 1, respectively, the receptive field is enlarged and the feature map is enhanced, facilitating the integration of local and global features. This enables the resultant image to preserve superficial structural features while incorporating deep-seated detailed features, thereby ensuring the continuity of feature information.

Figure 4. The overall architecture of the generator, including the number of layers of the encoder and decoder.

The decoder decodes and reconstructs the fused feature map, reduces the channel number of the feature map through four convolution layers, and finally outputs the reconstructed feature fusion image.

3.3.2. Discriminator structure

The task of the discriminator itself is to effectively distinguish between real and false data (fusion data). Given that infrared and visible light images have different structures and features, separate discriminators are designed for infrared and visible light data. The contrast of the fused image can be improved by adversarial training of the $$ D_{i} $$ discriminator, and the texture details can be enriched by learning the $$ D_{V} $$ discriminator. The adversarial network containing two discriminators enables the generator to produce fused images that not only contain rich detail information but also have significant contrast for better results.

The discriminator structure is shown in Figure 5, which consists of five convolutional layers with a convolution kernel size of 3 × 3 and one fully connected layer. These convolutional layers use the LeakyReLU activation function to adjust the problem of zero gradients caused by negative input. Convolutional layers 2, 3 and 4 use batch normalization to speed up network convergence. The fully connected layer FC employs the Tanh activation function and performs a linear transformation to produce a scalar, indicating the probability of the input being real data as opposed to fused data.

Figure 5. Discriminator structure diagram.

The dual discriminators adopt the same structure but the parameters are not shared. The discriminator and the generator play an adversarial game to guide the network to continuously supplement the visible detail gradient information and the infrared target intensity information into the fusion image. Figure 6 illustrates the specific process of image feature extraction and fusion in the generator. Firstly, the image feature extraction operation is carried out in the encoder, and the superimposed image is processed through two ordinary convolutional layers to achieve shallow feature extraction, and then passed to the multi-scale feature extraction module, which consists of RDB and ResNet modules. A global residual connection is used to fuse local and residual features. Finally, the decoder reduces the dimensionality of the fused image features.

Figure 6. Flowchart of image feature extraction and fusion.

3.4. Loss function design

The loss function of a fusion network includes two parts: generator loss function $$ L_{G} $$ and discriminator loss function $$ L_{D} $$. $$ L_{G} $$ is composed of content loss $$ L_{con} $$ and counter loss $$ L_{adv} $$, as give in

where $$ L_{G} $$ is the total loss of the generator, $$ L_{adv} $$ is the adversarial loss, $$ L_{con} $$ is the content loss, $$ \lambda $$ is the balance parameter used to balance the two losses, and $$ \lambda $$ = 0.6 in this article. Among them, $$ L_{con} $$ is the main loss function, so that the fused image can effectively retain the effective feature information of the source image, which is obtained as:

where $$ HW $$ is the product of the height and width of the input image, $$ G_{F} $$ is the fusion image output by the generator, $$ v $$ and $$ i $$ are the visible and infrared images, respectively, $$ \nabla $$ represents the gradient calculation, and $$ MSE $$ represents the mean square error loss, whose purpose is to constrain the difference in intensity distribution between fusion image $$ G_{F} $$ and infrared image $$ i $$. $$ TV $$ is the Total Variation norm, which is used as a regularization loss to compensate for the image blur problem that may be caused by using $$ MSE $$ alone. $$ F $$ is the Frobenius norm, which enhances the learning of detailed texture features by constraining the gradient information of the fused image $$ G_{F} $$ and the visible image $$ v $$. $$ MS\_SSIM $$ is the multi-scale structural similarity, which constrains the similarity between the fused image $$ G_{F} $$ and the visible image $$ v $$ from multiple scale directions. $$ \alpha $$, $$ \beta $$ and $$ \gamma $$ are the parameters of the balance loss, which are respectively taken as 0.5, 0.5 and 100 in this paper.

In the content loss formulation, as given in Equation (9), the first and second terms $$ \frac{1}{HW}\left\lbrack \left\| {G_{F} - i} \right\|_{MSE} + \alpha{\cdot \left\| {G_{F} - i} \right\|}_{TV}\right\rbrack $$ address the relationship between the fused and infrared images. The purpose of $$ MSE $$ loss is to constrain the intensity distribution differences between the fused and infrared images, enabling the network to effectively learn the infrared intensity distribution. However, relying solely on $$ MSE $$ loss may result in image blurring. To mitigate this, $$ TV $$ regularization is introduced as an additional term to preserve image sharpness and compensate for the limitations of MSE.

The third term $$ \frac{1}{HW}\cdot {}{}\beta \cdot \left\| {\nabla G_{F} - \nabla v} \right\|_{F}^{2} $$ in Equation (9) enforces a constraint on the gradient information between the fused image and the visible light image using the F-paradigm. This term aims to enhance the model's ability to learn detailed texture features from the visible image, as gradient information encapsulates critical edge and detail characteristics.

Finally, the incorporation of the $$ MS-SSIM $$ loss, based on human visual perception, enables a multi-scale evaluation of the similarity between the fused and visible light images. By considering structural features across multiple scales, $$ MS-SSIM $$ enables the model to capture fine-grained details at various levels, ensuring that these details are effectively integrated into the fused image. This multi-faceted approach ensures a more comprehensive representation of the structural and textural attributes in the fused output.

$$ L_{adv} $$ is the loss function of information interaction between the discriminator and generator, aiming to promote the fusion image to retain more useful information, guide the generator’s output, and reduce feature information loss in the counter. The process is expressed as

where $$ P_{G_{F}} $$ is the data distribution of the fused image, $$ D_{i} (G_{F}) $$ and $$ D_{v} (G_{F} ) $$ are the discrimination probability values of the fused image $$ G_{F} $$ walk by the infrared and visible light discriminators, respectively.

$$ L_{D} $$ is to enable the discriminator to effectively identify the source and generated images, laying the foundation for the confrontation between the discriminator and the generator, which is obtained by

where $$ D_{v} $$ and $$ D_{i} $$ are visible light and infrared discriminators, respectively.

4.1. Experimental settings

This paper selected 45 pairs of registered and corrected infrared and visible images under different scenes from the TNO dataset, which can be downloaded from the website https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029?file=1475454, and cropped them with a stride of 20, resulting in 37,164 pairs of infrared and visible images with a size of 108 × 108, which were normalized into the training set. During the training process, the Adam optimizer was used, the initial learning rate was set to 0.00016, the batch size was set to 36, and the iterations were 1,032 times, with a total of two rounds of training. Fusion algorithm model was built and trained under the Tensorflow framework, and the hardware platform is GeForce RTX 3090 GPU and Interi9-10900K CPU.

The test set selected 42 groups of infrared and visible images in the TNO data set, evaluated from both subjective and objective aspects. The fusion method in the paper is experimentally compared with six fusion methods: GAN with multiclassification constraints (GanMcC) method^[15], DDCGAN method^[16], U2Fusion method^[33], NestFuse method^[11], DIDFuse method^[34], and FusionGAN method^[35].

4.2. Comparative experiment and result analysis

The test set was used to conduct comparative experiments between the fusion method in the paper and other fusion methods, and three typical scenarios in the test set were selected for subjective evaluation analysis. Scenario 1 features dimly lit trees on both sides, making it difficult to distinguish the central figure from the background. Scenario 2 is affected by smoke interference, where soldiers are hard to identify, and the intricate details of interwoven background trees are challenging to discern. Scenario 3 focuses on identifying detailed features such as wheels and rooftops in low-light conditions. This analysis highlights the performance of the proposed method in addressing diverse and challenging visual fusion scenarios.

The experimental results of Scenario 1 are shown in Figure 7, in which the red box represents the images of infrared target extraction, the green box represents the visible light images, and the red and green arrows respectively indicate the enlarged details of the targets. Figure 7A and B represents the infrared and visible images, respectively. The infrared target in Figure 7C has insufficient thermal radiation information and low contrast. The images in Figure 7D and G are clearly distinguished from the background; the details of the trees are clear, but the overall tone is too dark, and the details of the grass only have a rough outline. As can be seen from Figure 7E and F, the fusion effect of the GanMcC method and the FusionGAN method is not good. The infrared character targets have a blur artifact phenomenon, and the grass of the visible light details enlarged by the green frame is also blurry. The details of the grass in Figure 7H have obvious outlines, but the edges of the details in the middle part are incoherent. Figure 7I shows the fusion algorithm of this paper. It not only has high-contrast infrared character targets, but also has the most prominent visual effect. The edges of the grass in visible light are clear, and the detailed textures are the most obvious and coherent, with strong recognition.

Figure 7. Comparison of the results of the fusion experiment on seven fusion methods in scenario 1. The red box represents the magnified image of infrared target extraction, and the green box represents the magnified image of visible light details.

The experimental results of Scenario 2 are shown in Figure 8, where the infrared soldier target is effectively distinguished from the background, but details such as the outline of the soldier and the trees are blurred. The texture of the trees and the edge of the soldier appearance in Figure 8C and E compared with Figure 8F are clearer, but its infrared radiation information is not retained enough, and the contrast is low, which cannot highlight the soldier's infrared target. The smoke outlines in Figure 8D and G produced some artifacts, and the soldiers and the smoke could not be effectively distinguished. The smoke edges in Figure 8H are clear, but the soldiers are also disturbed by the smoke causing blurry contours of soldiers. The method presented in Figure 8I in this paper not only effectively retains the edge details of trees and smoke, but makes soldiers which are the infrared key targets clearly visible. The fused image has high contrast and good visual effect, which effectively inhibits the interference of smoke and highlights the target, laying a good foundation for subsequent visual tasks such as target recognition and target positioning.

Figure 8. Comparison of the results of the fusion experiment on seven fusion methods in scenario 2. The red box represents the magnified image of infrared target extraction, and the green box represents the magnified image of visible light details.

The experimental results for Scene 3 are shown in Figure 9. From Figure 9E and F, it can be seen that the fusion effect of FusionGAN and GANMCc methods is not good, and the contours of tires and roofs are not clear. In Figure 9G, the roofs are clearer than in Figure 9E and F, but the tire contours are still fuzzy and some artifacts are generated, which have lost a lot of details. Infrared radiation information is not retained enough to highlight the tire target and cannot be effectively distinguished from the background [Figure 9C]. The tire can be distinguished from the background but the edge part is blurred, and the roof part lacks details [Figure 9H]. The contrast is high, and the contours of the tire and the roof are clear, but some details are lost [Figure 9D]. The method in this paper not only effectively preserves the edge details of the tire and the roof, but also has a high contrast and a better visual effect [Figure 9I], making it suitable for subsequent visual tasks such as target recognition, target location, and target positioning. These improvements lay a good foundation for further tasks.

Figure 9. Comparison of the results of the fusion experiment on seven fusion methods in scenario 3. The red box represents the magnified image of infrared target extraction, and the green box represents the magnified image of visible light details.

4.3. Index evaluation

Individual differences will affect the subjective evaluation results, so it is necessary to conduct objective quantitative data comparative analysis. We selected information entropy (EN), AG, SF, EI, standard deviation (SD) and visual information fidelity for fusion (VIFF) as objective evaluation indicators of fusion quality. Among them, the information EN represents the information richness of the image. AG and SF evaluate the clarity of the detailed texture of the image. EI reflects whether the edges of the image are obviously prominent. SD of high-contrast images is often higher, more eye-catching, and the visual effect is good. VIFF is used to evaluate image quality using the characteristics of the human vision system. The above six evaluation indicators are all positive indicators, and the larger the value, the better the fusion quality.

We conduct comparative experiments using the test set to obtain the objective evaluation index values of each fusion methods, with the average value used for quantitative analysis. The results are shown in Table 1. The horizontal coordinates of Figure 10 represent 42 image pairs of different scenes, which were selected for the comparative experiments. The values of each index were evaluated across all scenes, and a comprehensive analysis was subsequently conducted based on the average values presented in Table 1. In terms of information EN, the method in this paper yields the best result, because more details are added to the image by the fusion network, enriching its content. The three values of AG, SF and EI have all reached the highest, and have been greatly improved compared to other fusion methods, higher than the sub-optimal values of 65.41%, 65.09% and 55.22%, respectively, indicating that the proposed method has obtained more detailed edge information, clearer texture details, and more prominent edge transitions. In terms of SD, the proposed method performs similarly to NestFuse, ranking second only to DIDFuse, indicating its ability to clearly distinguish the foreground from the background and achieve higher infrared target contrast. The method in this paper attains the optimal value in the VIFF index, aligning with the properties of the human visual system, and offering an excellent visual experience. Figure 8 shows the line chart of the objective evaluation indicators of each fusion algorithm. The trend observed in this chart, when combined with the results from Table 1, more intuitively and clearly demonstrates the superiority of the proposed method, as evidenced by good fusion results in both overall performance and objective data.

Figure 10. The proposed method is used for quantitative comparison of infrared and visible image fusion with the other six different fusion algorithms. The legend shows the mean values of the indicators for the different methods.

4.4. Ablation experiment

In order to verify the function of the multi-level detail enhancement module, RDB and Res2Net modules in the algorithm network structure, the following three sets of ablation experiments were set up. Experiment 1: No multi-level detail enhancement was performed after the image input, and the source image was directly fused without preprocessing. Experiment 2: We removed the RDB feature extraction module, kept the other modules, and replaced them with ordinary convolutions of the same number of layers. Experiment 3: Instead of using the Res2Net feature extraction module, we replaced it with ordinary convolutions of the same number of layers. In the end, we used the test set to conduct objective evaluations by comparing the indicators of the three experimental methods, and the experimental results are shown in Table 2. Figure 11 visualizes the experimental results, and the specific values are presented in Table 2.

Figure 11. Schematic diagram of the results of the ablation experiment.

It can be seen from Experiment 1 that after using multi-level detail enhancement on the input image, the output fused image has higher values of various indicators, especially in the index of AG, SF and EI representing the edge of detail, proving that they make a lot of contributions to the retention of detail texture information. The enhanced image is also more informative. The contrast index SD is also enhanced to some extent, indicating that the quality of the fusion image has been greatly improved after the enhancement of multi-level details. The comparison results of Experiments 2 and 3 with the original algorithm show that the RDB and Res2Net modules can effectively extract multi-scale features of the image, making the local and global features more closely connected, which is beneficial to the final fusion effect.

4.5. Computational quantity experiment

In order to analyze the impact of the RDB and Res2Net modules on the speed of inference, we set up three models for comparative analysis and calculate the variation in the number of parameters and GFlops, respectively: The Proposed Model in the paper: Include RDB and Res2Net modules. RDB-Replaced Model: The RDB feature extraction module is replaced by ordinary convolution corresponding to the same number of layers. Res2Net-Replaced Model: The Res2Net feature extraction module is replaced by ordinary convolution corresponding to the same number of layers. The results are presented in Table 3. The inclusion of RDB and Res2Net modules reduced both parameters and GFlops, improving inference speed compared to using standard convolutions.

The number of parameters in the model proposed was reduced by 20.6% and 10.08% relative to the two sets of comparison experiments, respectively. Tables 4 and 5 show the results of comparing the number of input channels to the convolutional layer, the number of output channels, and the number of output parameters per layer for the three sets of models, respectively. The RDB module extracts rich local features through densely connected convolutional layers and uses global residual connectivity to improve information flow; the channel count is optimized through concatenation and 1 × 1 convolutions, as shown in Table 4, thereby reducing the computational load of subsequent convolution compared to using ordinary convolutional layers. The core of the Res2Net module is the use of group convolution for multi-scale expansion and thus multi-scale feature extraction. The number of convolution kernels within each group is determined by that of input feature channels, which helps reduce the number of parameters. If a standard convolution layer was used instead of the Res2Net module, the absence of the group convolution structure would significantly increase the number of parameters. Specifically, after applying group convolutions, the number of parameters becomes 1/n of that of a standard convolution, where n is the number of groups. As shown in Table 5, in the proposed model, n is set to 4, resulting in a fourfold difference in the number of parameters.

Both the RDB and Res2Net modules significantly reduce computational complexity, with GFlops decreasing by 52.7% and 59.07%, respectively. The RDB module minimizes redundant computations through feature reuse and dense connections, reducing the number of channels processed in each convolution. The Res2Net module, by using group convolutions and multi-scale feature fusion, efficiently reuses feature information, reducing computation while enhancing model expressiveness without a large increase in parameters.

Authors’ contributions

Made substantial contributions to conception and design of the study and performed data analysis and interpretation: Tian X, Xianyu X

Performed data acquisition and provided administrative, technical, and material support: Li Z, Jia Y, Xu T

Availability of data and materials

The TNO data set used in this article can be downloaded from the website https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029.

Financial support and sponsorship

This work was supported in part by the Natural Science Foundation of Jiangsu Province under Grant BK20210298 and the China Aeronautical Science Foundation (20240055052001).

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not appliable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2024.