AIGNLAI/GOLD

## 黄金子空间：持续测试时自适应中效率与泛化的交汇点

Guannan Lai · Da-Wei Zhou · Zhenguo Li · Han-Jia Ye

GOLD 的官方实现，这是一个持续测试时自适应框架，能在低秩的分类器敏感子空间中执行高效的在线自适应。

## 🎉 简介 持续测试时自适应（CTTA）旨在不访问源数据的情况下，持续地将预训练模型自适应于不断变化的目标环境。尽管最近的方法展现出了可喜的成效，但它们往往面临着一个根本的权衡：更强的自适应能力通常需要更新更多的参数，这不仅增加了在线优化的成本，甚至在分布持续变化的情况下可能会损害长期的鲁棒性。因此，在 CTTA 中同时实现有效性和高效性仍然是一项挑战。 在这项工作中，我们从自适应空间的视角重新审视了 CTTA。我们不去更新高维特征或大量的模型参数，而是提出了一个简单的问题：*是否存在一个紧凑的子空间，足以实现有效的测试时自适应？* 我们的答案是肯定的。如下图所示，尽管目标数据可能会随时间持续发生漂移，但有效的自适应可以在一个虽小却极具表达能力的子空间内进行，我们将其称为**黄金子空间**。基于这一洞见，我们提出了 **GOLD**，这是一个轻量且高效的 CTTA 框架，它能够动态识别并利用该子空间进行在线自适应，从而以最少的可训练参数和极低的计算开销实现强大的性能。 ## ✨ GOLD GOLD 建立在一个关键观察之上：有效的持续测试时自适应并不需要更新整个特征空间。相反，存在一个紧凑的、足以完成自适应的子空间，我们将其称为**黄金子空间**。基于这一洞见，GOLD 通过将目标特征投影到这个低维子空间中，并仅学习一个轻量级的缩放向量来执行在线自适应，在保持强大自适应能力的同时，显著降低了自适应成本。 具体而言，GOLD 由两个交替的阶段组成：**自适应**和**更新**。在自适应阶段，冻结的 backbone 特征会被投影到黄金子空间中，然后通过由缩放向量参数化的残差低秩变换进行重新校准。在更新阶段，GOLD 利用平均梯度外积（AGOP）从传入的目标数据中动态估计黄金子空间，并通过自训练和基于原型的对比目标来更新缩放向量。通过这种方式，GOLD 能够在不断演变的测试分布下实现高效且有效的自适应。 ## 📚 引用 如果您觉得此代码库有用，请考虑引用： ``` @inproceedings{lai2026golden, title = {The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation}, author = {Lai, Guannan and Zhou, Da-Wei and Li, Zhenguo and Ye, Han-Jia}, booktitle = {CVPR}, year = {2026} #### } ## ☄️ 如何使用 To use the repository, we provide a conda environment. ```bash conda update conda conda env create -f environment.yml #### conda activate tta #### Classification

Features

This repository contains an extensive collection of different methods, datasets, models, and settings, which we evaluate in a comprehensive benchmark (see below). We also provide a tutorial on how to use this repository in combination with CLIP-like models [here](classification/tutorials/tutorial_clip.md). A brief overview of the repository's main features is provided below: - **Datasets** - `cifar10_c` [CIFAR10-C](https://zenodo.org/record/2535967#.ZBiI7NDMKUk) - `cifar100_c` [CIFAR100-C](https://zenodo.org/record/3555552#.ZBiJA9DMKUk) - `imagenet_c` [ImageNet-C](https://zenodo.org/record/2235448#.Yj2RO_co_mF) - `imagenet_a` [ImageNet-A](https://github.com/hendrycks/natural-adv-examples) - `imagenet_r` [ImageNet-R](https://github.com/hendrycks/imagenet-r) - `imagenet_v2` [ImageNet-V2](https://huggingface.co/datasets/vaishaal/ImageNetV2/tree/main) - `imagenet_k` [ImageNet-Sketch](https://github.com/HaohanWang/ImageNet-Sketch) - `imagenet_d` [ImageNet-D](https://github.com/bethgelab/robustness/tree/main/examples/imagenet_d) - `imagenet_d109` - `domainnet126` [DomainNet (cleaned)](http://ai.bu.edu/M3SDA/) - `Continually Changing Corruptions` [CCC](https://github.com/oripress/CCC) - **Models** - For adapting to ImageNet variations, all pre-trained models available in [Torchvision](https://pytorch.org/vision/stable/models.html) or [timm](https://github.com/huggingface/pytorch-image-models?tab=readme-ov-file#models) can be used. - For the corruption benchmarks, pre-trained models from [RobustBench](https://github.com/RobustBench/robustbench) can be used. - For the DomainNet-126 benchmark, there is a pre-trained model for each domain. - Further models include [ResNet-26 GN](https://github.com/zhangmarvin/memo). - It is also possible to use the models provided by [OpenCLIP](https://github.com/mlfoundations/open_clip/tree/main). - **Settings** - `reset_each_shift` Reset the model state after the adaptation to a domain. - `continual` Train the model on a sequence of domains without knowing when a domain shift occurs. - `gradual` Train the model on a sequence of gradually increasing/decreasing domain shifts without knowing when a domain shift occurs. - `mixed_domains` Train the model on one long test sequence where consecutive test samples are likely to originate from different domains. - `correlated` Same as the continual setting but the samples of each domain are further sorted by class label. - `mixed_domains_correlated` Mixed domains and sorted by class label. - Combinations like `gradual_correlated` or `reset_each_shift_correlated` are also possible. - **Mixed Precision Training** - Almost all of the aforementioned methods (except SAR and GTTA) can be trained with mixed precision. This greatly speeds up your experiments and requires less memory. However, all benchmark results are generated with fp32. - **Modular Design** - Adding new methods should be rather simple, thanks to the modular design.

Get Started

To run one of the following benchmarks, the corresponding datasets need to be downloaded. - *CIFAR10-to-CIFAR10-C*: the data is automatically downloaded. - *CIFAR100-to-CIFAR100-C*: the data is automatically downloaded. - *ImageNet-to-ImageNet-C*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php) and [ImageNet-C](https://zenodo.org/record/2235448#.Yj2RO_co_mF). - *ImageNet-to-ImageNet-A*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php) and [ImageNet-A](https://github.com/hendrycks/natural-adv-examples). - *ImageNet-to-ImageNet-R*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php) and [ImageNet-R](https://github.com/hendrycks/imagenet-r). - *ImageNet-to-ImageNet-V2*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php) and [ImageNet-V2](https://huggingface.co/datasets/vaishaal/ImageNetV2/tree/main). - *ImageNet-to-ImageNet-Sketch*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php) and [ImageNet-Sketch](https://github.com/HaohanWang/ImageNet-Sketch). - *ImageNet-to-ImageNet-D*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php). For [ImageNet-D](https://openreview.net/pdf?id=LiC2vmzbpMO), see the download instructions for DomainNet-126 below. ImageNet-D is created by symlinks, which are set up at the first use. - *ImageNet-to-ImageNet-D109*: see instructions for DomainNet-126 below. - *DomainNet-126*: download the 6 splits of the [cleaned version](http://ai.bu.edu/M3SDA/). Following [MME](https://arxiv.org/abs/1904.06487), DomainNet-126 only uses a subset that contains 126 classes from 4 domains. - *ImageNet-to-CCC*: for non source-free methods, download [ImageNet](https://www.image-net.org/download.php). CCC is integrated as a webdataset and does not need to be downloaded! Please note that it cannot be combined with settings such as correlated. After downloading the missing datasets, you may need to adapt the path to the root directory `_C.DATA_DIR = "./data"` located in the file `conf.py`. For the individual datasets, the directory names are specified in `conf.py` as a dictionary (see function `complete_data_dir_path`). In case your directory names deviate from the ones specified in the mapping dictionary, you can simply modify them.

Run Experiments

We provide config files for all experiments and methods. Simply run the following Python file with the corresponding config file. ```bash #### python test_time.py --cfg cfgs/[ccc/cifar10_c/cifar100_c/imagenet_c/imagenet_others/domainnet126]/[source/norm_test/norm_alpha/tent/memo/rpl/eta/eata/rdumb/sar/cotta/rotta/adacontrast/lame/gtta/rmt/roid/tpt/tca].yaml For imagenet_others, the argument `CORRUPTION.DATASET` has to be passed: ```bash #### python test_time.py --cfg cfgs/imagenet_others/[source/norm_test/norm_alpha/tent/memo/rpl/eta/eata/rdumb/sar/cotta/rotta/adacontrast/lame/gtta/rmt/roid/tpt].yaml CORRUPTION.DATASET [imagenet_a/imagenet_r/imagenet_k/imagenet_v2/imagenet_d109] E.g., to run ROID for the ImageNet-to-ImageNet-R benchmark, run the following command. ```bash #### python test_time.py --cfg cfgs/imagenet_others/roid.yaml CORRUPTION.DATASET imagenet_r Alternatively, you can reproduce our experiments by running the `run.sh` in the subdirectory `classification/scripts`. For the different settings, modify `setting` within `run.sh`. To run the different continual DomainNet-126 sequences, you have to pass the `MODEL.CKPT_PATH` argument. When not specifying a `CKPT_PATH`, the sequence using the *real* domain as the source domain will be used. The checkpoints are provided by [AdaContrast](https://github.com/DianCh/AdaContrast) and can be downloaded [here](https://drive.google.com/drive/folders/1OOSzrl6kzxIlEhNAK168dPXJcHwJ1A2X). Structurally, it is best to download them into the directory `./ckpt/domainnet126`. ```bash #### python test_time.py --cfg cfgs/domainnet126/rmt.yaml MODEL.CKPT_PATH ./ckpt/domainnet126/best_clipart_2020.pth For GTTA, we provide checkpoint files for the style transfer network. The checkpoints are provided on Google-Drive ([download](https://drive.google.com/file/d/1IpkUwyw8i9HEEjjD6pbbe_MCxM7yqKBq/view?usp=sharing)); extract the zip-file within the `classification` subdirectory.

Changing Configurations

Changing the evaluation configuration is extremely easy. For example, to run TENT on ImageNet-to-ImageNet-C in the `reset_each_shift` setting with a ResNet-50 and the `IMAGENET1K_V1` initialization, the arguments below have to be passed. Further models and initializations can be found [here (torchvision)](https://pytorch.org/vision/stable/models.html) or [here (timm)](https://github.com/huggingface/pytorch-image-models?tab=readme-ov-file). ```bash #### python test_time.py --cfg cfgs/imagenet_c/tent.yaml MODEL.ARCH resnet50 MODEL.WEIGHTS IMAGENET1K_V1 SETTING reset_each_shift For ImageNet-C, the default image list provided by robustbench considers 5000 samples per domain (see [here](robustbench/data/imagenet_test_image_ids.txt)). If you are interested in running experiments on the full 50,000 test samples, simply set `CORRUPTION.NUM_EX 50000`, i.e. ```bash #### python test_time.py --cfg cfgs/imagenet_c/roid.yaml CORRUPTION.NUM_EX 50000

#### Segmentation For running the experiments based on CarlaTTA, you first have to download the dataset splits as provided below. + clear [download](https://drive.google.com/file/d/19HUmZkL5wo4gY7w5cfztgNVga_uNSVUp/view?usp=sharing) + day2night [download](https://drive.google.com/file/d/1R3br738UCPGryhWhJE-Uy4sCJW3FaVTr/view?usp=sharing) + clear2fog [download](https://drive.google.com/file/d/1LeNF9PpdJ7lbpsvNwGy9xpC-AYlPiwMI/view?usp=sharing) + clear2rain [download](https://drive.google.com/file/d/1TJfQ4CjIOJtrOpUCQ7VyqKBVYQndGNa_/view?usp=sharing) + dynamic [download](https://drive.google.com/file/d/1jb1qJMhOSJ48XUQ7eRqT7agnDK9OBwox/view?usp=sharing) + dynamic-slow [download](https://drive.google.com/file/d/1RTciKaw2LhlQ4ecKMlarSKyOzsDgaurT/view?usp=sharing) + clear-highway [download](https://drive.google.com/file/d/1lZlxwBVBSBAguONX9K6gI2NlWqAxECvB/view?usp=sharing) + highway [download](https://drive.google.com/file/d/1Q_3iOuDK4t-W3lvsHwRddDqHTE8GEAIj/view?usp=sharing) Again, you probably have to change the data directory `_C.DATA_DIR = "./data"` in `conf.py`. Further, you have to download the pre-trained source checkpoints ([download](https://drive.google.com/file/d/1PoeW-GnFr374j-J76H8udblSwrae74LQ/view?usp=sharing)) and extract the zip-file within the `segmentation` subdirectory. #### 例如，要运行 GTTA，请使用 `cfgs` 目录中提供的 config 文件并运行： #### python test_time.py --cfg cfgs/gtta.yaml You can also change the test sequences by setting `LIST_NAME_TEST` to: + day2night: `day_night_1200.txt` + clear2fog: `clear_fog_1200.txt` + clear2rain: `clear_rain_1200.txt` + dynamic: `dynamic_1200.txt` + highway: `town04_dynamic_1200.txt` If you choose highway as the test sequence, you have to change the source list and the corresponding checkpoint paths. ```bash #### python test_time.py --cfg cfgs/gtta.yaml LIST_NAME_SRC clear_highway_train.txt LIST_NAME_TEST town04_dynamic_1200.txt CKPT_PATH_SEG ./ckpt/clear_highway/ckpt_seg.pth CKPT_PATH_ADAIN_DEC = ./ckpt/clear_highway/ckpt_adain.pth ## 📊 结果 GOLD achieves strong performance across both continual test-time classification and segmentation benchmarks. On image classification, GOLD consistently outperforms prior CTTA methods under continuously evolving corruptions, achieving the best average error on CIFAR10-C, CIFAR100-C, and ImageNet-C. These results show that restricting adaptation to the golden subspace is not only highly parameter-efficient, but also remarkably effective in maintaining robustness under long-term distribution shifts. GOLD also generalizes well to dense prediction tasks. On continual test-time semantic segmentation, it delivers superior or highly competitive performance across multiple dynamic scenarios, including challenging transitions such as day-to-night, clear-to-fog, and highway environments. Together, these results demonstrate that GOLD provides a unified and efficient solution for continual adaptation across diverse vision tasks. ## 👨‍🏫 致谢 We thank the following repos/projects for helpful components: + Robustbench [official](https://github.com/RobustBench/robustbench) + CoTTA [official](https://github.com/qinenergy/cotta) + TENT [official](https://github.com/DequanWang/tent) + AdaContrast [official](https://github.com/DianCh/AdaContrast) + EATA [official](https://github.com/mr-eggplant/EATA) + RoTTA [official](https://github.com/BIT-DA/RoTTA) + SAR [official](https://github.com/mr-eggplant/SAR) + RDumb [official](https://github.com/oripress/CCC) + CMF [official](https://openreview.net/forum?id=BllUWdpIOA¬eId=FbQwbITFM0) + DeYO [official](https://github.com/Jhyun17/DeYO) + TPT [official](https://github.com/azshue/TPT) + TCA [official](https://github.com/Successybbdwm/TCA) + CarlaTTA [official](https://github.com/Jo-wang/TTA) ## 🤗 联系方式 For questions and feedback, please open an issue or contact: ## - Guannan Lai ([laign@lamda.nju.edu.cn](mailto:laign@lamda.nju.edu.cn)) [](https://star-history.com/#AIGNLAI/GOLD&Date) ```

标签：Apex, PyTorch, 人工智能, 低秩子空间, 凭据扫描, 持续测试时适应, 机器学习, 模型泛化, 用户模式Hook绕过, 逆向工具