SANet: scale-adaptive network for lightweight salient object detection

2.1. SOD

SOD is based on simulating a human visual attention mechanism, which enables machines to automatically discover and filter important information. Since Professor Itti pioneered the research field of SOD in 1999, countless researchers have been engaged in research in this field and produced many scientific research results. SOD's technical solutions have also shifted from traditional statistical methods, frequency domain conversion methods, and machine learning methods to the currently hot field of deep learning. Traditional methods ^[23] are mainly based on manually designed features. Although they are very efficient, manually designed features inherently lack the ability of high-level representation, which limits the performance of the model. The deep learning-based methods have shown incomparable advantages over traditional methods in characterizing salient objects. They have quickly occupied the forefront of SOD and raised the level of SOD to a new height. Early deep learning-based methods did not solve the problem of longitudinal transmission feature attenuation, and the model had problems of false positives or negatives ^[24]. To this end, the encoder network is applied to SOD. Refs.^[14] and ^[25] also designed a feature conversion module to improve the effectiveness of horizontal feature transmission. To embed semantic information into the encoding and decoding processes, Chen et al. and Jia et al. designed different global information enhancement modules respectively^[1,26]. The Transformer model has facilitated a further enhancement in the level of SOD. However, the early transformer-based SOD models ^[27] were relatively complex and were not very suitable for high-resolution SOD tasks. Although some lightweight transformer networks ^[28] have been proposed in recent years and have reduced the number of model parameters from level B to around 80 M, this is still not affordable for edge applications. Current SOD methods based on large models still have the problem of high model complexity.

In summary, relevant research on SOD has accumulated a lot of research results, and the detection effect has reached the level of practical application. However, this is achieved under ideal laboratory conditions. The complexity and real-time performance of the model cannot meet the requirements in weak computing and high real-time scenarios.

2.2. Model lightweighting

Lightweight models have attracted attention in various fields due to their low computing resource requirements. There are two main methods to build lightweight models. One is to use network pruning, model quantization, or knowledge distillation to make complex models lightweight. Network pruning reduces the size of a neural network by removing unnecessary connections or nodes ^[29]. Model quantization reduces the storage space and computing resources required by the model by reducing the number of bits of the parameters and representing the parameters as integers or fixed-point numbers with fewer bits ^[30]. The knowledge distillation method achieves model lightweighting by transferring knowledge between large models and small models ^[31]. The other approach is to consider lightweight from the network design stage, to design an efficient and lightweight backbone network. Lightweight network design has been a research hotspot in the field of deep learning in recent years, aiming to provide efficient neural network models for mobile devices and edge computing. Representative methods in this category include MobileNets ^[18,19], EfficientNets ^[32,33], and ShuffleNets ^[20,21]. The most prominent feature of MobileNets is the use of depthwise separable convolutions instead of ordinary convolutions to achieve model lightweighting. The characteristic of EfficientNets is that they use a compound scaling strategy to design the network, controlling the model complexity by adjusting the model depth, the network width, and the image resolution. ShuffleNets follow the design concept of sparse connectivity and reduce computation and parameters by using group pointwise convolution and channel shuffle. In addition, GhostNet ^[34] proposed by Huawei is also an excellent lightweight network, but the design ideas and technology used in the model are similar to those mentioned above and will not be elaborated on here.

The research on lightweight models for SOD is still in its infancy. Currently, the more representative ones include SAMNet and CSNet proposed by Professor Cheng et al. at Nankai University, and ELWNet ^[35] proposed by Professor Zhang et al. at Northeastern University. Among them, ELWNet is achieved through feature domain conversion, CSNet realizes model lightweighting based on the dynamic weight decay pruning method, and SAMNet is achieved by optimizing the network structure.

Currently, despite numerous studies on model lightweighting, there are still three problems: (1) Pruning, quantization, and knowledge distillation greatly influence model performance, making the model performance insufficient to meet actual needs; (2) The lightweight backbone network has a small feature domain and cannot cope with complex detection scenarios; (3) Research on lightweight models for SOD is still in its infancy. To address these issues, we propose a more efficient and lightweight SOD model - SANet.

2.3. CapsNet

CNN has a dominant position in solving computer vision-related problems. However, it discards much valuable information in the pooling process, such as the pose and position of the target. Another disadvantage of CNN is that it lacks rotation invariance and therefore requires a large amount of training data ^[36]. To this end, Hinton of Google Brain proposed the capsule network (CapsNet) that can capture structural information ^[37]. CapsNets designed a clever dynamic routing algorithm to capture the part-whole relationship in the image to enhance the equivalence of the network. Based on this advantage, related research and applications based on the CapsNet structure have been proposed one after another. Saqur et al. proposed a new algorithm CapsGAN by showing the weakness of the CNN-based generative adversarial network (GAN) architecture in generating 3D images^[38]. Cheng et al. proposed complex-valued dense CapsNet (Cv-CapsNet) and complex-valued diverse CapsNet (Cv-CapsNet++) for image classification^[39]. Sun et al. proposed a deep tensor capsule network that uses a new tensor capsule-based routing algorithm and the corresponding convolution operation^[40].

Due to the unique advantages of CapsNet, it has also been successfully applied to the task of SOD. For example, Liu et al. optimized deep unsupervised SOD by using the part-whole relationship characteristics of CapsNet^[41]. Zhang et al. used the attention mechanism to interact with CNN and CapsNet features to better detect salient objects^[42]. Liu et al. integrated the advantages of CNN and CapsNet, extracted different semantic information respectively, and interacted with each other to generate better saliency prediction maps^[43]. The design of the SAFE module in this paper also contains some ideas for CapsNet.

3.1. Overall network architecture

As shown in Figure 3, the overall network structure of SANet comprises a bottom-up encoder, a top-down decoder, and a lateral connection between them. The encoder is built with SAFE modules as units and is divided into five stages. In these five stages, we downsample the input using dilated DSConv3×3 with a stride of 2 and adjust the number of channels. Then, we use the proposed SAFE module for scale-adaptive learning. Since the resolution of the feature map is high in the first two stages (E₁ and E₂), only a single SAFE module is used to process the feature map to reduce the computational burden. In the third to fifth stages (E₃, E₄, and E₅), we stack multiple SAFE modules to increase the receptive field and enhance the deep network representation capability. The default parameter settings of the SANet backbone network are shown in Table 1. We pass the output of the last encoder stage (E₅) through a pyramid pooling module (PPM) ^[44] to further improve the network’s learning of global features. Different from the classic encoder-decoder network structure, this paper inputs the output features of PPM into the decoders of each stage for feature fusion, so as to make full use of the semantic information in the deep layer of the network.

Figure 3. Overall encoder-decoder architecture of the proposed SANet. E_i represents the encoder at the i-th stage. D_i indicates the decoder at the i-th stage. S_i and R_i denote the output feature maps of the encoder and the decoder at the i-th stage, respectively. P_i stands for the predicted saliency map, and P₁ is the final prediction result. G is the ground-truth saliency map. PPM: Pyramid pooling module; MFA: multi-scale feature aggregation.

3.2. SAFE module

The multi-scale information of images is important for SOD, and salient objects in natural scenes are scale-variable. To adaptively extract information from different scales of images and accurately characterize salient objects, we propose the SAFE module, which is mainly divided into two parts: multi-scale feature interaction and dynamic selection.

Multi-scale Feature Interaction: In this part, as shown in Figure 4, we first use multiple depthwise separable convolutions with different dilation rates to process the input feature map and divide the input features into various branches. Since each branch has distinct sensitivities to information of different scales, to improve the representation capabilities of different branches, we perform cross-scale feature interaction.

Figure 4. Illustration of the proposed SAFE module. SAFE: Scale-adaptive feature extraction.

Specifically, let $$ X\in {{\mathbb{R}}^{C\times H\times W}} $$ be the input feature map whose number of channels, height, and width are C, H, and W, respectively. So, we will get the feature map $$ {X_i} $$ of each branch, namely,

where $$ \mathit{＄}_i $$ denotes depthwise separable conv3×3 (DSConv3×3 for short) with different dilation rates at branch i, and N is the number of branches. Next, except for $$ X_N $$, each feature map $$ X_{i-1} $$ is first processed by the 3×3 average pooling operation, and then added to $$ X_i $$ to obtain $$ {X'_i} $$, so $$ {X'_i} $$ can be expressed as follows:

where AP denotes 3×3 average pooling operation. In this way, each feature map $$ {X'_i} $$ can receive the feature information of all its previous feature maps $$ \left\{ {{X_j},j \leqslant i} \right\} $$, which realizes feature embedding and improves the representation ability of the intra-layer branches.

Dynamic selection: Features of different scales have varying representation capabilities for salient objects. To measure this difference, we perform dynamic selection after completing the multi-scale feature interaction. We use an element-wise summation to integrate the feature maps output by different branches, namely,

Here we use element-wise summation instead of concatenation because the concatenation operation will greatly increase the number of channels, resulting in heavier computational complexity and more network parameters. Next, we process X' with a 3 × 3 convolution and then perform the dynamic measurement module (DMM) operation. DMM consists of a global average pooling (GAP) operation and an MLP. We gather the global contextual information with channelwise statistics by using GAP. This process embeds the input X' to a learnable latent vector $$ Z \in {\mathbb{R}^{C\times 1\times 1}} $$ by performing GAP on X' over the spatial dimension. Thus, the c-th component of Z can be given as follows:

where H stands for height, which represents the number of pixels of X' in the vertical direction, and W stands for width, which represents the number of pixels of X' in the horizontal direction. Because each element in Z indicates the importance of the corresponding feature slice of X', Z can be used as the channel-wise attention of all branches. Next, we shall perform an additional embedding regarding Z via a MLP consisting of two fully-connected layers, a non-linearity ReLU, and a softmax operation. After the MLP, a vector of size $$ \left( {N \times C} \right) \times 1 \times 1 $$ will be output, and then we will split it into N parts corresponding to N different branches through the split operation, and the i-th part is $$ {\omega _i} \in {\mathbb{R}^{C\times 1\times 1}} $$. Since MLP is learnable, different attention weights can be dynamically assigned to each scale feature. The dynamic attention weight $$ {\omega _i} $$ of the i-th branch is calculated as follows:

where $$ {\omega _i} \in {\mathbb{R}^{C\times 1\times 1}} $$. After that, we use channel-wise multiplication to combine $$ {\omega _i} $$ with the corresponding branch features and integrate the multiplied results by element-wise summation to obtain the feature map $$ \tilde X $$:

After $$ \tilde X $$ undergoes a Conv1×1, it is added to the original input feature map X through a residual connection to obtain the final output Y, namely,

where $$ \kappa $$ denotes a Conv1×1 operation.

As shown in Figure 3, the SAFE module is the basic unit that forms the backbone network of SANet. The number of branches N mentioned above is set as a hyperparameter. In the subsequent ablation experiments, we prove that the higher the resolution of the input feature map, the more branches are needed, so we will set different numbers of branches at multiple stages of the network.

3.3. MFA-based decoder

This paper first uses the backbone network composed of SAFE modules to extract image features, and then we design the decoder. As shown in Figure 3, the decoder consists of two parts: feature fusion and MFA modules.

The feature fusion module fuses features from three directions and uses a depthwise separable k×k dilated convolution to integrate the fused results. After feature fusion, it is not enough to simply use a layer of convolution for processing. In particular, the output features of the PPM module will go through a relatively large scale span (maximum span of 16 times) during the upsampling process, and it is necessary to establish connections across scales reasonably. Therefore, we designed a MFA module to process further the fused features.

As shown in Figure 5, we use the output feature map of the feature fusion module as the input of the MFA module and split it into four parts by channel, represented by $$ F_i $$, where $$ i \in \left\{ {1,2,3,4} \right\} $$. Then, information on different scales is extracted through a depth-separable convolution with a dilation rate of $$ {r'_i} $$, forming four different branches and obtaining feature maps $$ {F'_i} $$, where $$ {r'_i} \in \left\{ {1,2,3,4} \right\} $$. So, $$ {F'_i} $$ can be expressed as follows:

Figure 5. Illustration of the proposed MFA module. MFA: Multi-scale feature aggregation; DDS: dilated DSConv3×3 operation; ©: a concatenation operation.

where $$ {\mathit{＄}_i} $$ represents DSConv3×3 with different dilation rates at branch i. To achieve cross-scale feature aggregation, we use residual connections between different branches and get $$ {F''_i} $$ by element-wise summation:

Then, each branch's results are merged as output through a concatenation operation, namely,

Let $$ R_i $$ represent the feature maps output by the decoder at each stage, and $$ S_i $$ represent the feature maps output by the encoder at each stage, where $$ i \in \left\{ {1,2,3,4,5} \right\} $$. So,

where MFA represents MFA module, $$ \zeta _5^{k \times k} $$ means DSConvk×k at the fifth stage, and Up indicates upsampling operation. In summary, we have

where $$ \zeta _i^{k \times k} $$ represents DSConvk×k at the i-th stage.

3.4. Saliency reasoning

We use deep supervision to improve the transparency of the hidden layer learning process. As shown in Figure 3, for the fusion features at different stages, we use a Conv1×1 and sigmoid activation function to generate multiple predictions, namely $$ P_i $$, where $$ i \in \left\{ {1,2,3,4,5} \right\} $$. We adopt the standard binary cross-entropy loss for training, which is defined as follows:

where $$ {L_{BCE}} $$ is the standard binary cross-entropy loss function, and G denotes the ground-truth saliency map. $$ \lambda $$ denotes the weighting scalar for loss balance, which is set to 0.4.

4.1. Experimental setup

4.2. Performance analysis

In this section, we compare SANet with eighteen typical SOD methods, including fifteen heavyweight methods and three lightweight state-of-the-art methods. This paper uses the same method to evaluate the detection results of related models.

4.2.3. Comprehensive comparison in general scenarios

Figure 6 shows the comprehensive comparison results of this paper and other methods in terms of model performance and efficiency. In the sub-figures of Figure 6A and B, SANet lies at the top-left corner. In Figure 6C, it lies at the top-right corner. This shows that SANet achieves higher accuracy with fewer parameters and FLOPs and faster speed. Therefore, it achieves a good trade-off between performance and efficiency.

Figure 6. Illustration of the trade-off between performance and efficiency. The avgF is the average of the results on the six datasets. (A) avgF vs. #Param; (B) avgF vs. FLOPs; (C) avgF vs. FPS. FLOPs: Floating-point operations; FPS: frames per second.

4.2.4. Qualitative comparison in general scenarios

For practical application scenarios of SOD, good visual qualitative effects are sometimes more important than quantitative performance. In Figure 7, we provide visual SOD results in five typical scenarios to evaluate the model effect. It can be seen that in the simple scene [Figure 7A], the visual detection results of SANet are comparable to those of heavyweight methods, and the depiction of salient target details is more accurate than other lightweight models. In the small target scene [Figure 7B], heavyweight methods, boundary-aware SOD network (BASNet), PoolNet, and visual saliency transformer (VST), can accurately identify salient targets, while cascaded partial decoder (CPD), SOD using short connections (DSS), ICON, and MENet have false positives and false negatives. Our model can segment small targets, and the boundaries are clearer than other lightweight methods, without false positives and false negatives. This is also due to the SAFE module, which enables our model to adaptively capture salient objects of any size. In the low-contrast scene [Figure 7C], DSS and PoolNet have false positives, while other heavyweight methods can accurately identify salient objects. Among lightweight methods, CSNet has false positives, whereas SANet, SAMNet, and HVPNet do not. However, compared with SANet, SAMNet and HVPNet do not accurately depict the details of salient objects. In the confusing scene [Figure 7D], SANet can still accurately identify the target object, while other methods except ICON have false positives. In the complex scene [Figure 7E and F], other lightweight methods including some heavyweight methods have false positives and negatives, while our method has demonstrated excellent performance in the complex scene regardless of whether there is one or multiple salient targets.

Figure 7. Visual comparison with mainstream SOD methods in general scenarios. (A) Simple scene; (B) small target scene; (C) low contrast scene; (D) confusing scene; (E) complex scene; (F) complex scene. SOD: Salient object detection.

4.2.5. Comparison in traffic scenarios

We use three models in TSOD for comparative experiments, and they perform well in general scenarios. These models are trained on the TSOD dataset; the final test results are shown in Table 3 and Figure 8. From Table 3, we can see that our method is better than the comparison methods in terms of maxF, avgF, MAE, and S. As shown in Figure 8, we qualitatively compare our method with two other excellent lightweight SOD methods in seven different scenarios. Figure 8A is a simple scene. It can be seen that SANet can easily identify the salient target and depict its outline clearly. In contrast, the performance of the other two methods is unsatisfactory. Figure 8B and C are small target scenes during the day. It can be seen that SANet still performs well for such ultra-small targets, thanks to the fact that our proposed SAFE module is sensitive to objects of various scales. Figure 8D is a multi-target scene. It can be seen that SANet does not mistakenly regard the large truck next to it as a salient target, but accurately identifies the relatively small correct salient target in the distance. Figure 8E is a multi-target scene with a complex background. SANet can also identify the unique salient target, while SAMNet does not identify any salient objects, and CSNet includes other targets. Figure 8F is a night scene with low contrast and interference from vehicle lights. SANet can accurately identify road signs, while the other two methods do not identify any salient objects. Figure 8G is a small target scene at night. Although SANet did not accurately identify the outline of the small target, it still correctly identified the road sign. In general, compared with the other two excellent lightweight SOD methods, SANet can obtain better results in general scenes and various challenging scenes, which also proves the effectiveness of our proposed innovations.

Table 3

The Comparison of maxF, avgF, MAE, and S in traffic scenarios

Methods	TSOD
	maxF$$ \uparrow $$	avgF$$ \uparrow $$	MAE$$ \downarrow $$	S$$ \uparrow $$
MAE: Mean absolute error; TSOD: traffic salient object detection.
SAMNet	0.333	0.126	0.058	0.590
CSNet	0.233	0.055	0.062	0.535
Ours	0.650	0.347	0.036	0.700

Figure 8. Visual comparison of different methods in traffic scenarios. (A) Simple scene during daytime; (B) small target scene during daytime; (C) small target scene during daytime; (D) multi-target scene; (E) multi-target and complex background scene; (F) low contrast scene at night; (G) small target scene at night.

4.3. Failure cases

Although our proposed method achieves excellent performance in multiple scenarios, it does encounter some failure cases. As shown in Figure 9, we provide failure cases in different scenarios and conduct an in-depth analysis. From Figure 9A, we can see that SANet did not identify any significant targets, showing that SANet's recognition ability for ultra-small targets is still limited. In Figure 9B, due to the low overall contrast between the vehicle and the background, SANet did not identify the vehicle as a significant target, but mistakenly identified the roadside sign. Figure 9C and D are multi-target scenes. SANet both mistakenly identified multiple targets and the sizes of the acquired targets were somewhat different. This is because SANet has a strong ability to distinguish objects of different scales. We will improve it in future experiments. Figure 9E is a multi-target scene with a complex background. It can be seen that SANet still mistakenly identified multiple targets, and the recognition accuracy is not high when two vehicles overlap. Figure 9F is a small target scene at night. It can be seen that the headlights will have a certain impact on the detection results. Figure 9G is a scene with strong light interference at night. In this scene, the noise interference is strong, which has a certain impact on the detection results. We attribute these failure cases to the following factors: (1) The number of images in the training set is limited and cannot cover all traffic scenarios; (2) The actual traffic scenarios are complex and diverse, which poses a huge challenge to the performance of the model; (3) The representation ability of lightweight networks is limited, and the network's processing capabilities for overly complex traffic scenarios are still insufficient.

Figure 9. Failure cases of SANet. (A) Small target scene during daytime; (B) low contrast scene during daytime; (C) multi-target scene; (D) multi-target scene; (E) multi-target and complex background scene; (F) small target scene at night; (G) strong light interference scene at night.

The model we proposed currently performs poorly in traffic detection, especially in small object scenes and complex background scenes. To address this issue, we propose several possible solutions in the future: (1) The training set of the traffic scene SOD dataset TSOD used in this study has only 2,000 images, which is less than one-fifth of DUTS-TR. In the future, we can improve it by increasing the number of training set images; (2) Currently, computing power is developing rapidly, and the computing power of onboard computing is far superior to that of the past. We may be able to improve the model's representation capabilities for small target scenes and complex scenes by appropriately increasing the number of model parameters and model depth; (3) Use knowledge distillation to use large-scale networks or pre-trained models as teacher models to guide the learning of lightweight networks.

4.4. Ablation study

In this section, we conduct an ablation study on the proposed module components, the backbone network's effectiveness, and the SAFE module's configuration to demonstrate our proposed model's effectiveness. The relevant experimental settings are consistent with those outlined in Section 4.1.

4.5. Proposed module components

Table 4 shows the results of the ablation study of the model components in this paper. As the number of model components increases, the model performance improves progressively. Compared with Ver.0, the average values of maxF on six datasets of Ver.3 increased by 0.015 and MAE decreased by 0.014. There is no ImageNet pre-training between Ver.0 and Ver.3, and the difference in their experimental results also shows that the proposed model is effective.

4.6. The effectiveness of the backbone network

In addition to existing SOD methods, we also compared several widely used lightweight backbone networks, including MobileNet, MobileNetV2, ShuffleNetV2, and EfficientNet. To use these lightweight backbone networks for SOD tasks, we add the same decoder as SANet to these networks for ablation study.

In Table 5, we can see that directly applying the existing lightweight backbone network to the SOD task does not produce satisfactory results regarding accuracy. Taking EfficientNet as an example, we take the average values of maxF, avgF, and MAE of six data sets. The results showed that compared to EfficientNet, SANet achieved a 13.20% improvement in maxF, an 11.14% improvement in avgF, and a 44.72% reduction in MAE. This further verifies the correctness and rationality of our redesign of the backbone network structure for SOD.

4.7. Configuration of the SAFE module

Table 6 presents the ablation study results of the SAFE module with varying branch numbers and dilation rates. Increasing the number of branches in the E₁-E₄ stages improves some metrics, but also significantly increases computational complexity, which contradicts our goal of maintaining a lightweight model. The default settings of the SAFE module are selected after weighing the trade-off between model accuracy and complexity.

Authors’ contributions

Made substantial contributions to conception and design of the study, performed data analysis and interpretation, and wrote the manuscript: Liu Z

Provided technical guidance and reviewed the manuscript: Zhao W, Jia N

Reviewed the manuscript: Liu X, Yang J

Availability of data and materials

The raw data that support the findings of this study are available from the corresponding author Zhao W, upon reasonable request. The datasets used in the paper and other model-related image data have been appropriately cited in References with the standard referencing conventions.

Financial support and sponsorship

This work was supported by the National Natural Science Foundation of China (62403361).

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2024.