# model-merging

**Repository Path**: lsreback/model-merging

## Basic Information

- **Project Name**: model-merging
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-15
- **Last Updated**: 2026-07-15

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Code of Model Merging

> Official repository for **CSP**, **LOT Merging**, **TATR**, and **CAT Merging**.

## 🚀 News

- **CSP** — Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core Subspace — Accepted at **ICML 2026** 🎉
- **LOT Merging** — [*Towards minimizing feature drift in model merging: Layer-wise task vector fusion for adaptive knowledge integration*](https://arxiv.org/pdf/2505.23859) — Accepted at **NeurIPS 2025** 🎉
- **TATR** — [*Task arithmetic in trust region: A training-free model merging approach to navigate knowledge conflicts*](https://arxiv.org/pdf/2501.15065) — Accepted at **ACM MM 2025** 🎉
- **CAT Merging** — [*CAT Merging: A training-free approach for resolving conflicts in model merging*](https://arxiv.org/abs/2505.06977) — Accepted at **ICML 2025** 🎉

## ⚙️ Setup

Run the corresponding entry point for each method:

- `./main_csp_ta.py` 、`./main_csp_lot.py` 、`./main_csp_iso.py` 、`./main_csp_tsv.py` → CSP
- `./main_lot.py` → LOT Merging
- `./main_cat.py` → CAT Merging
- `./main_tatr.py` → TATR
- `./main_tatr_zeroshot.py` → TATR (zero-shot version)

## 📂Datasets

Supported datasets:

- SUN397
- Cars
- RESISC45
- EuroSAT
- SVHN
- GTSRB
- MNIST
- DTD

👉 Refer to dataset processing in [task_vectors](https://github.com/mlfoundations/task_vectors).
👉 Or directly download processed datasets from:

- [Baidu Cloud](https://pan.baidu.com/s/1w0Z2UVv3NVmqDhjH8WTOJQ?pwd=kvg6)
- [HuggingFace](https://huggingface.co/collections/tanganke/image-classification-datasets-662abda7d75efe6b0e6b43da)

## 🔑 Checkpoints

Fine-tuned checkpoints are available at:

- [task_vectors#checkpoints](https://github.com/mlfoundations/task_vectors#checkpoints)
- [Google Drive: task_vectors_checkpoints](https://drive.google.com/drive/folders/1u_Tva6x0p6oxu5Eo0ZZsf-520Cc_3MKw)

⚠️ Note: If `torch.load(xxx_checkpoint).state_dict()` fails, try:

```
pickle.load(open(xxx_checkpoint, "rb")).state_dict()
```

------

## 📖 Citation

If you find this repository useful, please cite our work:

```
@article{csp,
  title={Revisiting the Role of Pretrained Weights in Model Merging: On Near-Optimality within the Core Subspace},
  author={Sun, Wenju and Li, Qingyong and Li, Tiancheng and Geng, Yangli-ao and Li, Boyang},
  journal={International Conference on Machine Learning},
  year={2026}
}

@inproceedings{lotmerging,
  title={Towards minimizing feature drift in model merging: Layer-wise task vector fusion for adaptive knowledge integration},
  author={Sun, Wenju and Li, Qingyong and Wang, Wen and Liu, Yang and Geng, Yangli-ao and Li, Boyang},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}

@inproceedings{tatr,
  title={Task arithmetic in trust region: A training-free model merging approach to navigate knowledge conflicts}, 
  author={Wenju Sun and Qingyong Li and Wen Wang and Yangli-ao Geng and Boyang Li},
  booktitle={ACM MultiMedia},
  year={2025}
}

@article{catmerging,
  title={CAT merging: A training-free approach for resolving conflicts in model merging},
  author={Sun, Wenju and Li, Qingyong and Geng, Yangli-ao and Li, Boyang},
  journal={International Conference on Machine Learning},
  year={2025}
}
```