# torch_diffgm

**Repository Path**: frontxiang/torch_diffgm

## Basic Information

- **Project Name**: torch_diffgm
- **Description**: PyTorch Implementation of Expert Systems with Applications 2026 Paper - Dynamic Learning Rate Adaptation via Momentum-Guided Gradient Discrepancy
- **Primary Language**: Python
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-08-14
- **Last Updated**: 2026-08-10

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# DiffGM: Dynamic Learning Rate Adaptation via Momentum-Guided Gradient Discrepancy

**Official PyTorch implementation of the paper accepted by *Expert Systems with Applications (ESWA)*, 2027.**

> Dynamic learning rate adaptation via momentum-guided gradient discrepancy  
> Qian Xiang\*, Wenmi Chai\*, Lei Lei, Yafei Song, Can Li  
> *Expert Systems with Applications*, Volume 332, 2027, Article 133676  
> DOI: [10.1016/j.eswa.2026.133676](https://doi.org/10.1016/j.eswa.2026.133676)

---

## Overview
DiffGM (Adam with **Diff**erence between **G**radient and the first-order **M**omentum) is a novel adaptive optimizer that redefines gradient discrepancy measurement by integrating momentum-guided dynamics. It modulates per-parameter learning rates through the difference between current gradients and their exponential moving average (first-moment estimate), establishing intrinsic alignment with the true optimization trajectory.

DiffGM fundamentally resolves three inherent limitations of the DiffGrad optimizer:
1.  Misalignment between gradient discrepancy and actual parameter update directions
2.  High sensitivity to stochastic mini-batch gradient noise
3.  Overly conservative learning rates at saddle points

---

## Key Features
- **Rigorous theoretical guarantees**: Proven variance reduction, asymptotic convergence under noisy gradients, efficient saddle-point escape, implicit regularization, and implicit Fisher preconditioning for non-convex optimization.
- **Strong empirical performance**: Consistently outperforms its direct baseline DiffGrad across all tested tasks, and achieves state-of-the-art or competitive performance versus Adam, RAdam, and CAME on computer vision, industrial signal processing, 6G communications, and deep reinforcement learning.
- **Zero extra memory overhead**: Reuses the first-moment estimate $m_t$ from the standard Adam framework, requiring no additional memory storage (unlike DiffGrad, which must store the previous gradient $g_{t-1}$).
- **Drop-in replacement**: Fully compatible with the standard PyTorch optimizer API, replaceable with Adam/DiffGrad via a single line of code.
- **Ecosystem compatible**: Natively works with decoupled weight decay, gradient clipping, learning rate schedulers, and large-batch training pipelines.

---

## Installation

### Requirements
- Python >= 3.11
- PyTorch >= 2.1
- CUDA >= 12.1 (for GPU training)

### From source
```bash
git clone https://gitee.com/frontxiang/torch_diffgm.git
cd torch_diffgm
```

The optimizer is a pure-Python module with no extra compilation step. Just make sure the repository root is on your `PYTHONPATH` (running from the project root works out of the box).

---

## Quick Start
DiffGM follows the standard PyTorch optimizer interface. You can use it as a direct replacement for Adam in your training pipeline:

```python
import torch
import torch.nn as nn
from xqoptimizers import DiffGM

# Define your model
model = nn.Sequential(
    nn.Linear(10, 64),
    nn.ReLU(),
    nn.Linear(64, 2)
)

# Initialize DiffGM optimizer
optimizer = DiffGM(
    model.parameters(),
    lr=1e-3,            # Initial learning rate
    betas=(0.9, 0.999), # Exponential decay rates (beta1, beta2)
    eps=1e-8,           # Numerical stability term
    weight_decay=0.0,   # Weight decay factor
    scale=1.0           # Sensitivity of the AbsSig mapping (1.0 = paper default)
)

# Standard training loop
criterion = nn.CrossEntropyLoss()
for batch_x, batch_y in train_dataloader:
    optimizer.zero_grad()
    outputs = model(batch_x)
    loss = criterion(outputs, batch_y)
    loss.backward()
    optimizer.step()
```

### Core Update Rule
The core innovation of DiffGM is the momentum-guided friction coefficient (DFC). Instead of DiffGrad's consecutive gradient difference $\Delta g_t = g_t - g_{t-1}$, DiffGM measures the deviation of the current gradient from the EMA-smoothed optimization trajectory:

$$
\Delta g_t = g_t - m_t
$$

$$
\xi_t = \text{AbsSig}(\Delta g_t) = \frac{1}{1 + e^{-s \cdot |g_t - m_t|}}
$$

where $g_t$ is the current stochastic gradient, $m_t$ is the first-order EMA momentum estimate, and $s$ is the `scale` parameter (default $1.0$, matching the paper). The full parameter update rule:

$$
\theta_{t+1} = \theta_t - \alpha \cdot \xi_t \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}
$$

The DFC $\xi_t \in [0.5, 1]$ applies *less friction* (larger learning rate) when the current gradient diverges from the momentum trajectory, and *more friction* (learning rate scaled toward 0.5) when the gradient is consistent with the established trend.

---

## Algorithm

```
Algorithm 1: DiffGM (all computations are element-wise)

Require: initial learning rate α0
Require: decay rates β1, β2 ∈ (0, 1]
Require: loss function f(θ) with parameters θ

m0 ← 0 ; v0 ← 0 ; t ← 0
while θt not converged do
    gt ← ∇θ ft(θt-1)
    mt ← β1·mt-1 + (1-β1)·gt          # first moment (EMA)
    vt ← β2·vt-1 + (1-β2)·gt²         # second moment
    m̂t ← mt / (1 - β1^t)              # bias-corrected first moment
    v̂t ← vt / (1 - β2^t)              # bias-corrected second moment
    ξt  ← AbsSig(gt - mt)             # momentum-guided DFC
    θt ← θt-1 - α0 · ξt · m̂t / (√v̂t + ε)
    t ← t + 1
end while
return θt
```

**Key difference from DiffGrad.** DiffGrad computes $\xi_t = \text{AbsSig}(g_t - g_{t-1})$, which performs *instantaneous fluctuation detection* between two adjacent gradients and is sensitive to mini-batch noise. DiffGM computes $\xi_t = \text{AbsSig}(g_t - m_t)$, which performs *trend deviation measurement* against the EMA-smoothed descent direction, inheriting a low-pass filtering effect that attenuates high-frequency noise while preserving the persistent optimization trajectory.

---

## Theoretical Properties
Under standard assumptions (L-smoothness, bounded gradient noise, Robbins-Monro learning rate schedule), the paper rigorously establishes five core properties of DiffGM for non-convex optimization:

| # | Theorem | Statement (informal) |
|---|---------|----------------------|
| 1 | **Variance Reduction via EMA Smoothing** | $\mathrm{Var}(\Delta g_t^{\text{DiffGM}}) = \frac{\beta_1^2}{1+\beta_1}\sigma^2$, strictly smaller than $\mathrm{Var}(\Delta g_t^{\text{DiffGrad}}) = 2\sigma^2$ for $\beta_1 \in (0,1)$. |
| 2 | **Convergence Guarantee under Noisy Gradients** | With a Robbins-Monro schedule, $\lim_{t\to\infty} \mathbb{E}[\|\nabla f(\theta_t)\|^2] = 0$, even for non-convex objectives. |
| 3 | **Efficient Saddle-Point Escape** | At strict saddle points, the momentum $m_t$ retains a non-zero discrepancy $g_t - m_t$, maintaining an effective learning rate for escape—unlike DiffGrad, whose consecutive-gradient difference vanishes. |
| 4 | **Implicit Regularization** | The EMA in $m_t$ introduces a time-decaying penalty on gradient fluctuations, equivalent to gradient noise suppression, which improves generalization. |
| 5 | **Implicit Fisher Preconditioning** | The gradient-discrepancy signal is connected to the Fisher information matrix, aligning updates with the natural Riemannian geometry of parameter space. |

A convergence constraint derived from the analysis is $\frac{(1-\beta_1)^2}{\sqrt{1-\beta_2}} < 1$, which is satisfied by the default $\beta_1=0.9, \beta_2=0.999$.

---

## Main Results
### Key Performance on Benchmark Tasks (Test Set)
All results are reported as mean ± std over 5 runs (10 runs for RL). Bold = best, underline = second best (per the paper). Higher is better for accuracy/reward/SGCS.

| Task Category | Dataset | Model | Metric | DiffGM | DiffGrad | Adam | RAdam | CAME |
|---|---|---|---|---|---|---|---|---|
| Image Classification | CIFAR-10 | ResNet18 | Test Acc (%) | **93.20±0.25** | 92.50±0.44 | 89.02±1.75 | 91.57±1.43 | 92.79±0.13 |
| Image Classification | CIFAR-100 | ResNet18 | Test Acc (%) | **72.66±1.61** | 70.37±0.15 | 62.55±1.60 | 65.12±0.97 | 69.09±0.57 |
| Image Classification | PathMNIST | ResNet18 | Test Acc (%) | **92.40±0.17** | 90.56±0.61 | 91.19±0.24 | 91.88±0.13 | 87.79±1.15 |
| Image Classification | SVHN | ResNet18 | Test Acc (%) | **95.90±0.13** | 95.57±0.07 | 95.11±0.07 | 95.32±0.12 | 95.02±0.43 |
| Fault Diagnosis | HIT | 1D-CNN | Test Acc (%) | **90.93±0.48** | 90.62±0.16 | 89.47±0.31 | 90.61±0.47 | 85.97±0.46 |
| Radar Recognition | HRRP | 1D-CNN | Test Acc (%) | **96.66±0.05** | 96.56±0.14 | 96.52±0.12 | 96.59±0.10 | 96.06±0.14 |
| CSI Reconstruction | 6G CSI | Transformer | Test SGCS | **0.67±0.01** | 0.66±0.02 | 0.55±0.07 | 0.67±0.03 | 0.43±0.03 |
| Sentiment Analysis | SST-2 | XLM-RoBERTa | Test Acc (%) | 81.25±0.52 | 79.47±0.11 | **81.82±0.86** | 81.36±0.52 | 53.78±2.87 |
| Reinforcement Learning | CartPole-v1 | DQN | Avg Reward | 479.50±31.77 | 459.90±80.29 | 475.00±15.00 | 464.30±46.36 | **482.40±50.16** |

> Full results (training metrics, precision/recall/F1/AUC, convergence curves, and ablation studies) are available in the paper.

### Applicable Boundaries
DiffGM achieves the most significant gains in tasks requiring a balance between convergence speed and generalization (image classification, industrial signal processing, 6G CSI regression, DRL). On SST-2 sentiment analysis it is competitive with—but does not surpass—Adam, and on CartPole-v1 its final reward is slightly below CAME while offering markedly lower variance. In all tasks, DiffGM consistently and clearly outperforms its direct predecessor DiffGrad.

---

## Computational Efficiency
DiffGM reuses the first-moment estimate $m_t$ already maintained by Adam-type optimizers, avoiding the extra memory access required by DiffGrad to store and load $g_{t-1}$. This yields faster iterations despite identical asymptotic complexity $O(n)$ per step.

Per-task training-time reduction of DiffGM versus DiffGrad (reported in the paper):

| Task | Dataset | DiffGM Time (s) | DiffGrad Time (s) | Reduction |
|---|---|---|---|---|
| Image Classification | CIFAR-100 | 14,222 | 54,862 | ~74% |
| Sentiment Analysis | SST-2 | 50,443 | 75,745 | ~33% |
| Fault Diagnosis | HIT | 226 | 158 | — |
| Radar Recognition | HRRP | 1,408 | 1,508 | ~7% |
| Reinforcement Learning | CartPole-v1 | 937 | 1,089 | ~14% |

> Note: Absolute times vary with server load. DiffGM shows a 7–74% training-time reduction versus DiffGrad on most tasks; on a minority of scenarios (e.g., CIFAR-10, HIT) the wall-clock time is comparable or slightly higher. Versus classic optimizers such as Adam, DiffGM does not claim a per-iteration speed advantage—its benefit lies in stabilized trajectories and stronger generalization.

---

## Hyperparameter Guidelines

| Parameter | Default Value | Usage Guidance |
|---|---|---|
| `lr` | `1e-3` | Task-specific; use the same learning rate and schedule as you would for Adam (`1e-5` for NLP fine-tuning) |
| `betas[0]` ($\beta_1$) | `0.9` | Optimal balance of convergence speed and stability; tune in [0.8, 0.95]. Sensitivity study confirms 0.9 is best. |
| `betas[1]` ($\beta_2$) | `0.999` | Keep consistent with standard Adam defaults |
| `eps` | `1e-8` | Term for numerical stability |
| `weight_decay` | `0.0` | Recommended `1e-3` for image classification tasks; `0.0` for NLP/RL |
| `scale` | `1.0` | Sensitivity of the AbsSig DFC mapping (`1.0` matches the paper). Larger values make the sigmoid steeper, sharpening the response to gradient-momentum discrepancy. Leave at `1.0` unless tuning. |

---

## Project Structure
The DiffGM optimizer ships inside the `xqoptimizers` package, which also bundles several reference optimizers used as baselines in the paper:

```
torch_diffgm/
├── pdf/                        # Paper PDF
├── xqoptimizers/
│   ├── __init__.py             # Package exports
│   ├── diffgm.py               # DiffGM  (proposed method)
│   ├── diffgrad.py             # DiffGrad (direct baseline)
│   ├── diffmod.py              # DiffMod variant
│   ├── radam.py / radamw.py    # RAdam / RAdamW
│   ├── came.py                 # CAME
│   ├── adabound.py / adabob.py # AdaBound / AdaBoB
│   ├── adabelief.py            # AdaBelief (+ _fast, _derivative)
│   ├── adan.py                 # Adan
│   ├── apollo.py               # Apollo
│   ├── lion.py                 # Lion
│   ├── ranger.py               # Ranger
│   └── sophia.py               # Sophia
├── LICENSE
└── README.md
```

Import any optimizer from the package:
```python
from xqoptimizers import DiffGM, DiffGrad, RAdam, CAME
```

---

## Citation
**IF YOU USE THIS CODE, PLEASE CITE**: Qian Xiang, Wenmi Chai, Lei Lei, Yafei Song, and Can Li. 2026. 'Dynamic Learning Rate Adaptation via Momentum-Guided Gradient Discrepancy', Expert Systems with Applications: 133676.


```bibtex
@article{xiang2026diffgm,
  title={Dynamic learning rate adaptation via momentum-guided gradient discrepancy},
  author={Xiang, Qian and Chai, Wenmi and Lei, Lei and Song, Yafei and Li, Can},
  journal={Expert Systems with Applications},
  volume={332},
  pages={133676},
  year={2027},
  publisher={Elsevier},
  doi={10.1016/j.eswa.2026.133676}
}
```

---

## License
This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for full details.

## Contact
- Corresponding author: Qian Xiang (qianxljp@126.com)
- Code repository: [https://gitee.com/frontxiang/torch_diffgm.git](https://gitee.com/frontxiang/torch_diffgm.git)

## Acknowledgments
This work was supported by the National Key Research and Development Program of China (grant 2024YFB3311204), the National Natural Science Foundation of China (grants 62573424, 62402521, 62403487, 62203461, 62203365, 62227814), the Key Projects of the Shaanxi Province Natural Science Foundation (grant 2025JC-QYXQ-038), the Young Talent Fund of University Association for Science and Technology in Shaanxi (grant 20220106), the Young Talent Promotion Program of Shaanxi Association for Science and Technology (grants 20220121, 20230125), the Shaanxi Provincial Natural Science Foundation Youth Project (grant 2024JC-YBQN-0674), and the Open Foundation of the State Key Laboratory of Fluid Power and Mechatronic Systems (grant GZKF-202430).


## Other related papers:
- **Qian Xiang**, Xiaodan Wang, Lei Lei, and Yafei Song. 2025. 'Dynamic bound adaptive gradient methods with belief in observed gradients', Pattern Recognition: 111819. https://doi.org/10.1016/j.patcog.2025.111819 Code: https://gitee.com/frontxiang/torch_adabob.git
-  **Qian Xiang**, Xiaodan Wang, Yafei Song, and Lei Lei. 2025. ISONet: Reforming 1DCNN for aero-engine system inter-shaft bearing fault diagnosis via input spatial over-parameterization. Expert Systems with Applications, 277, 127248. https://doi.org/10.1016/j.eswa.2025.127248 Code: https://gitee.com/frontxiang/torch_isonet
-  **Qian Xiang**, Xiaodan Wang, Jie Lai, Lei Lei, Yafei Song, Jiaxing He, and Rui Li. 2024. 'Quadruplet depth-wise separable fusion convolution neural network for ballistic target recognition with limited samples', Expert Systems with Applications, 235: 121182. https://doi.org/10.1016/j.eswa.2023.121182
-  **Qian Xiang**, Xiaodan Wang, Yafei Song, Lei Lei, Rui Li, and Jie Lai. 2021. 'One-dimensional convolutional neural networks for high-resolution range profile recognition via adaptively feature recalibrating and automatically channel pruning', International Journal of Intelligent Systems, 36: 332-61. https://onlinelibrary.wiley.com/doi/abs/10.1002/int.22302
-  **Qian Xiang**, Xiaodan Wang, Jie Lai, Yafei Song, Rui Li, and Lei Lei. 2023. 'Group-Fusion One-Dimensional Convolutional Neural Network for Ballistic Target High-Resolution Range Profile Recognition with Layer-Wise Auxiliary Classifiers', International Journal of Computational Intelligence Systems, 16: 190. https://doi.org/10.1007/s44196-023-00372-w
-  **Qian Xiang**, Xiaodan Wang, Jie Lai, Yafei Song, Rui Li, and Lei Lei. 2022. 'Multi-scale group-fusion convolutional neural network for high-resolution range profile target recognition', Iet Radar Sonar and Navigation, 16: 1997-2016. https://doi.org/10.1049/rsn2.12312
-  **Qian Xiang**, Xiaodan Wang, Xuan Wu, Jie Lai, Jiaxing He, and Yafei Song. 2023. "CsiTransformer: A Limited-sample 6G Channel State Information Feedback Model." In 2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI 2023), 1160-66.   https://doi.org/10.1109/PRAI59366.2023.10331944
-  **Qian Xiang**, Xiaodan Wang, Jie Lai, Yafei Song, Jiaxing He, and Lei Lei. 2022. "5G Network Reference Signal Receiving Power Prediction Based on Multilayer Perceptron." In 2022 China Automation Congress (CAC 2022), 19-24.   https://doi.org/10.1109/CAC57257.2022.10055904.