Auto-regressive transformation for image alignment

Authors: Kanggeon Lee, Soochahn Lee, Kyoung Mu Lee

TL;DR: An auto-regressive approach to image alignment that sequentially predicts transformation parameters, enabling accurate registration of images with complex, non-rigid deformations.

What this paper is about

Image alignment (registration) typically estimates a single global or local transformation in one shot, which can struggle with large, complex deformations. Real-world image pairs – especially in medical or remote sensing domains – often require handling intricate spatial relationships that are hard to capture all at once.

Alignment Results in Challenging Scenarios. For image pairs with sparse features, scale differences, deformations degradations, and domain shifts, our method performs coarse-to-fine auto-regressive transformation refinement, achieving accurate alignment even in challenging scenarios where state-of-the-art methods struggle. The zoomed-in boxes show the local alignment results, and the highlighted vessel image below illustrates the intersection (yellow) between the two images (red and green).

Key idea

The paper reformulates image alignment as an auto-regressive process, where transformation parameters are predicted sequentially, each step conditioned on the previous ones. This decomposition allows the model to handle complex deformations by building up the transformation incrementally rather than predicting everything in a single pass.

Auto-Regressive Transformation (ART) iteratively refines the transformation D for image pairs I in a coarse-to-fine manner. Its sampling strategy enables effective operation across diverse domains and datasets.

ART first extracts multi-scale features Fs and Fd from the input image pair Is and Id. At each sampling step k, the corresponding features, F k s and F k d , are passed through the Cross-Attention Layer (CAL) to identify the correlated features that guide the network’s focus on regions requiring refinement. The attentive feature map˜F k s→d is then used to refine the transform field parameters Dk M and Dk A to Dk+1 M and Dk+1 A through multiple convolutional neural networks. This auto-regressive process continues until the initialized transform field parameters D0 M and D0 A reach the full resolution of the input image pair Is and Id.

Why it matters

Presented at ICCV 2025, this approach provides a principled way to tackle difficult alignment problems and could generalize across many domains where accurate image registration is critical.