Recently, our laboratory’s work on parameter-efficient fine-tuning of vision foundation models, entitled“Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models” (Authors: Shufan Shen, Junshu Sun, Shuhui Wang, and Qingming Huang), has been accepted by T-PAMI. T-PAMI, which stands for IEEE Transactions on Pattern Analysis and Machine Intelligence, is a major international journal in the fields of pattern recognition, computer vision, and machine learning, with an impact factor of 20.4 as announced in 2026.
In recent years, parameter-efficient fine-tuning has become an important technique for adapting pre-trained vision models to downstream tasks. Among these approaches, sparse fine-tuning achieves strong task adaptation by updating only a small number of task-relevant weights, while reducing computational costs and the risk of overfitting. However, existing sparse fine-tuning methods typically adopt a two-stage “locate-then-tune” paradigm. They not only rely on static gradient information to locate task-relevant weights, overlooking the dynamic evolution of parameters during fine-tuning, but also still need to store the complete weight matrices in the optimizer, resulting in high memory overhead. To address these issues, this work proposes an end-to-end sparse fine-tuning framework, SNELLA (Sparse tuning with kerNELized LoRA and Adaptive bi-level sparsity allocation), which enables the dynamic localization and updating of task-relevant weights with low memory overhead. Specifically, we first extend LoRA from the perspective of kernel methods by mapping and merging low-rank matrices through a nonlinear kernel function to construct a high-rank update matrix, thereby overcoming the limitation of the low-rank constraint in conventional LoRA on the independent adjustment of weights. Furthermore, to address the trade-off between expressiveness and training stability in existing nonlinear kernel functions, we propose the Mix-K kernel, which combines a normalized RBF kernel with a piecewise linear kernel to improve the expressiveness of the low-rank learnable modules while effectively alleviating the gradient vanishing problem. To more accurately locate task-relevant weights, we further propose an adaptive bi-level sparsity allocation mechanism, which dynamically performs competition across and within layers to adaptively allocate the limited parameter budget to more important layers and weights, thereby enabling the end-to-end localization and updating of sparse parameters.
Extensive experiments demonstrate that SNELLA achieves better performance than existing methods across a variety of tasks, including image classification, medical image segmentation, and text-to-image generation. For example, on the FGVC benchmark, SNELLA improves the average Top-1 accuracy by 1.8% (91.9% vs. 90.1%) compared with SPT-LoRA, while achieving 31.1%–39.9% memory reduction across models of different scales, demonstrating its strong performance and memory efficiency.
Download: