// the find
robbyant-research/CoDeF
[CVPR'24 Highlight] Official PyTorch implementation of CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
CoDeF is the official implementation behind a CVPR 2024 paper that represents a video as a canonical 2D image plus a per-frame deformation field, so you can run an image editing/translation algorithm once on the canonical frame and propagate it to the whole video with temporal consistency. It's aimed at video editing/translation researchers and ML engineers prototyping consistent video-to-image-model pipelines, not at anyone wanting a drop-in video tool.
The core idea is genuinely clever: decoupling a static canonical content field from a temporal deformation field lets you lift any single-image algorithm (ControlNet, keypoint detectors, etc.) to video without retraining anything video-specific. The repo ships the whole chain, not just the model: SAM-Track mask prep, RAFT optical flow extraction, training/test scripts, and pretrained checkpoints for five example sequences so you can reproduce results without optimizing from scratch. Config-per-sequence (configs/*/base.yaml) is a sane way to manage the many per-video hyperparameters this kind of per-scene NeRF-like optimization needs.
There's no inference-only mode — every new video requires a fresh per-scene optimization (train_multi.sh), so this is closer to a research pipeline than a usable tool, and nothing in the README gives a sense of how long that optimization takes or how sensitive it is to canonical_wh tuning beyond a one-line warning. The dependency stack is fragile: tiny-cuda-nn as a hand-built CUDA extension, PyTorch Lightning 2.0.2, CUDA 11.7 pinned, plus external RAFT and SAM-Track repos you have to clone and wire up separately — expect a day of environment debugging before you get a single result. No last-mile tooling: no CI, no automated tests, no Colab-maintained-by-the-authors (the linked Colab is community-maintained by a third party), and the project has had no commits since April 2024, so any breakage against newer PyTorch/CUDA versions is on you to fix.