Energy-Guided Flow Matching

arXiv:2608.058112026-08-06

Making an image generator paint the blurry big picture first and sharp details later lets it reach better results with far less training

Pixel-space generative models learn to travel from noise straight to a finished image, forcing them to learn overall structure and fine texture at the same time. This paper instead makes the destination itself move, starting from a blurred, low-frequency version of the image and sharpening smoothly toward the finished image, with an image-specific 'energy' schedule that controls how fast each picture's details are released. In experiments this coarse-to-fine trajectory reached lower FID (a lower-is-better image-quality gap score) on ImageNet class-conditional generation using fewer training epochs than the standard fixed-endpoint approach.

METAL LAB explanatory visual

EG-FM: a generation path with a moving endpoint

Evidence statusMeasured results reported

1. Fixed endpoint (standard method)Standard flow matching always moves straight from noise toward the same fully-detailed image, forcing joint learning of structure and detail.
2. Build a low-frequency starting endpointA heat-kernel filter blurs the image into a low-frequency version, which becomes the endpoint at time t=0.
3. Measure each image's energy gapThe missing high-frequency energy between the blurred and clean image is measured per image to set a personalized release speed (heat-time).
4. Synchronize with a shared clockA shared smootherstep-based release clock ensures every image recovers the same fraction of detail at the same path time t.
5. Train with a moving endpoint and adjusted velocityBecause the endpoint keeps sharpening, the training target velocity adds a term for the endpoint's own motion to the usual noise-to-endpoint velocity.

An explanatory diagram made by METAL LAB, not a figure supplied by the paper's authors.

What they did

Standard flow matching (a way of generating images by learning a straight path from noise to the final image) always targets the same fully-detailed image at every step, so the model must learn coarse structure and fine detail jointly, which the authors argue makes training harder.
EG-FM replaces that fixed target with a heat-kernel-filtered target: a smoothed, low-frequency version of the image at the start of the path that gradually sharpens into the full image by the end.
Because different images contain different amounts of high-frequency detail, the method measures each image's own missing high-frequency energy and derives a per-image 'heat-time' schedule, synchronized by a shared release clock so every image recovers the same fraction of detail at the same path time.
Since the endpoint keeps moving, the target training velocity is redefined as the sum of the usual noise-to-endpoint velocity plus a new term capturing how fast the endpoint itself is moving.
The method was tested on three backbones (PixelDiT, DeCo, HyperDiT) across ImageNet 256x256 and 512x512 class-conditional generation and text-to-image generation.

Figure 1: Comparison of baselines and Energy-Guided Flow Matching (EG-FM). Marker size increases as checkpoints approach the desirable lower-left region, representing fewer training epochs and lower FID. Based on backbones of PixelDiT and DeCo, the EG-FM variants achieve improved FID performance with significant fewer training epochs.

Table 1: Class-conditional generation on ImageNet ×256. Each Energy-Guided FM row uses the same backbone and ADM evaluation protocol. PixelDiT additionally expose convergence at multiple budgets. NFE denotes the number of function evaluations, with ×2 accounting for conditional and unconditional CFG evaluations.
Method	Epochs	#Params	NFE	FID ↓	sFID ↓	IS ↑	Precision ↑	Recall ↑
REPA (35)	800	675M	250×2	1.42	4.70	305.7	0.80	0.65
DDT-XL (34)	400	675M	–	1.26	–	310.6	0.79	0.65
RAE-XL (38)	800	839M	–	1.13	–	262.6	0.78	0.67
PixelFlow-XL (4)	320	677M	120×2	1.98	5.83	282.1	0.81	0.60
PixNerd-XL (33)	320	700M	100×2	1.93	–	298.0	0.80	0.60
JiT-G (20)	600	2.0B	100×2	1.82	–	292.6	0.79	0.62
PixelU-H/16 (10)	600	1.17B	100×2	1.63	5.04	305.9	0.79	0.64
DiP-XL/16 (5)	600	631M	100×2	1.79	4.59	281.9	0.80	0.63
FREPix-XL (21)	320	674M	100×2	1.91	4.59	295.6	0.79	0.62
DeCo-XL/16 (26)	600	682M	100×2	1.69	4.59	304.0	0.79	0.63
+ Energy-Guided FM	440	682M	100×2	1.63	4.78	300.1	0.79	0.62
HyperDiT-H (12)	600	952M	100×2	1.56	4.73	306.5	0.80	0.64
+ Energy-Guided FM	220	952M	100×2	1.51	4.31	293.4	0.78	0.64
PixelDiT-XL (36)	80	797M	100×2	2.36	5.11	282.3	0.80	0.57
PixelDiT-XL (36)	320	797M	100×2	1.61	4.68	292.7	0.78	0.64
PixelDiT-XL (36)	800	797M	100×2	1.54	4.49	297.0	0.78	0.65
+ Energy-Guided FM	80	797M	100×2	1.99	5.09	280.8	0.81	0.61
+ Energy-Guided FM	200	797M	100×2	1.55	4.60	296.2	0.79	0.65
+ Energy-Guided FM	600	797M	100×2	1.45	4.41	299.6	0.78	0.65

Figure 2: Overview of Energy-Guided Flow Matching. Starting from a clean image, we construct a low-frequency endpoint and measure the residual spectral gap 𝒢~x. Based on the global release clock q⁡(t), we derive a sample-adaptive heat time h⁡(x,t) that controls the heat-kernel response R⁡(h,ρ), enabling the progressive recovery of frequency components from low to high. For the training trajectory, the moving spectral endpoints yt(x) evolve from a easily predicted low-pass manifold toward the full-image manifold, forming a coarse-to-fine curved path zt. The velocity yt(x0)−ϵ toward the current endpoint and the endpoint-motion term t∂tyt(x0) together form the target velocity vt.

Table 2: Quantitative comparison for class-conditional generation on ImageNet ×512. † denotes continued training from a checkpoint on ImageNet ×256.
Method	Epochs	Params	FID ↓	IS ↑
DiT-XL/2	600	675M	3.04	240.8
SiT-XL/2	600	675M	2.62	252.2
REPA	200	675M	2.08	274.6
PixNerd-XL†	320	700M	2.84	245.6
JiT-H	600	956M	1.94	309.1
PixelU-H/32	600	1.2B	1.92	322.1
DiP-XL/32	-	631M	2.31	291.7
DeCo-XL/16†	340	682M	2.22	290.0
PixelDiT-XL†	850	797M	1.81	278.6
+ EG-FM†	240	797M	1.68	295.5
HyperDiT-H + EG-FM†	260	952M	1.58	285.0

Figure 3: Visualization of sample-adaptive heat-time. Under the same release clock, texture-rich Sample A requires a faster decrease in heat time than Sample B. Lower heat time indicates weaker attenuation and greater frequency release; the dashed curve denotes mean heat-time over ImageNet.

Table 7: Detailed comparison for class-conditional ImageNet generation at ×512. Metrics are computed on 50K generated samples with the ADM evaluation suite. NFE includes conditional and unconditional classifier-free-guidance evaluations; † denotes continued training from a ×256 checkpoint. A dash indicates an unreported item.
Method	Epochs	#Params	NFE	FID ↓	sFID ↓	IS ↑	Precision ↑	Recall ↑
DiT-XL/2	600	675M	250×2	3.04	5.02	240.8	0.84	0.54
SiT-XL/2	600	675M	250×2	2.62	4.18	252.2	0.84	0.57
REPA	200	675M	250×2	2.08	4.19	274.6	0.83	0.58
PixNerd-XL†	320	700M	100×2	2.84	5.95	245.6	0.80	0.59
JiT-H	600	956M	100×2	1.94	–	309.1	–	–
PixelU-H/32	600	1.2B	100×2	1.92	5.98	322.1	0.80	0.58
DiP-XL/32	–	631M	100×2	2.31	4.48	291.7	0.84	0.58
DeCo-XL/16†	340	682M	100×2	2.22	4.67	290.0	0.80	0.60
PixelDiT-XL†	850	797M	100×2	1.81	5.61	278.6	0.78	0.67
+ EG-FM†	240	797M	100×2	1.68	4.77	295.5	0.79	0.63
HyperDiT-H + EG-FM†	260	952M	100×2	1.58	4.90	285.0	0.79	0.64

Figure 4: Qualitative results produced by PixelDiT-XL with EG-FM. From left to right, the panels show ImageNet class-conditional samples at ×512 and ×256, followed by text-to-image samples at ×512.

Table 8: Category-wise GenEval results for text-to-image generation at ×512. Overall is the unweighted mean of the six task scores. Bold and underlined values denote the best and second-best results in each column. A dash indicates an unreported item.
Method	#Params	Single obj.	Two obj.	Counting	Colors	Position	Color attr.	Overall ↑
PixArt-α	0.6B	0.98	0.50	0.44	0.80	0.08	0.07	0.48
SD3	8B	0.98	0.84	0.66	0.74	0.40	0.43	0.68
FLUX.1-dev	12B	0.99	0.81	0.79	0.74	0.20	0.47	0.67
DALL-E 3	–	0.96	0.87	0.47	0.83	0.43	0.45	0.67
BLIP3o	4B	–	–	–	–	–	–	0.81
OmniGen2	4B	1.00	0.95	0.64	0.88	0.55	0.76	0.80
PixelFlow	0.9B	–	–	–	–	–	–	0.60
PixNerd	1.2B	0.97	0.86	0.44	0.83	0.71	0.53	0.73
DeCo-XXL/16	1.1B	1.00	0.92	0.72	0.91	0.80	0.79	0.86
PixelDiT-T2I	1.3B	1.00	0.94	0.70	0.90	0.53	0.65	0.78
EG-FM-T2I	1.3B	1.00	0.95	0.74	0.92	0.72	0.77	0.85

Figure 9: FID across training checkpoints computed with torch-fidelity and the ADM evaluation suite. The two implementations show closely aligned convergence trends from 80 to 600 epochs.

Table 9: Category-wise DPG-Bench results for text-to-image generation at ×512. Bold and underlined values denote the best and second-best results in each column. A dash indicates an unreported item.
Method	#Params	Global	Entity	Attribute	Relation	Other	Overall ↑
PixArt-α	0.6B	81.7	80.1	80.4	81.7	76.5	71.6
PixArt-Σ	0.6B	87.5	87.1	86.5	84.0	86.1	79.5
PixelFlow	0.9B	–	–	–	–	–	77.9
PixNerd	1.2B	80.5	87.9	87.2	91.3	72.8	80.9
DeCo-XXL/16	1.1B	–	–	–	–	–	81.4
PixelDiT-T2I	1.3B	88.0	90.9	87.6	89.8	88.5	83.7
EG-FM-T2I	1.3B	89.3	89.2	90.2	90.9	89.7	83.9

Table 10: Matched training compute and wall time. GFLOPs and ΔGFLOPs are reported per sample, with ΔGFLOPs measured relative to Standard FM. FLOPs increase and time increase are percentage changes relative to the matched Standard FM configuration; negative time values indicate faster steps. Per-step wall time is in seconds, and epoch time is in hours. All experiments were conducted on 8 × B200 GPU.
Backbone	Path	Res.	Batch	GFLOPs	ΔGFLOPs	FLOPs inc.	Time/step	Time inc.	Epoch time
DeCo-XL/16	Standard FM	2562	256	734.81	0.00	0.00	0.10	0.00	0.14
DeCo-XL/16	EG-FM	2562	256	734.86	0.06	+0.01	0.10	+0.41	0.14
PixelDiT-B/16	Standard FM	2562	256	226.29	0.00	0.00	0.04	0.00	0.06
PixelDiT-B/16	EG-FM	2562	256	226.34	0.06	+0.03	0.04	+4.81	0.06
PixelDiT-L/16	Standard FM	2562	256	683.30	0.00	0.00	0.09	0.00	0.12
PixelDiT-L/16	EG-FM	2562	256	683.36	0.06	+0.01	0.09	+0.51	0.12
PixelDiT-XL/16	Standard FM	2562	256	933.58	0.00	0.00	0.11	0.00	0.15
PixelDiT-XL/16	EG-FM	2562	256	933.64	0.06	+0.01	0.11	+0.99	0.15
PixelDiT-XL/16	Standard FM	5122	64	4056.72	0.00	0.00	0.12	0.00	0.66
PixelDiT-XL/16	EG-FM	5122	64	4056.98	0.26	+0.01	0.12	+0.92	0.66

Figure 10: Additional text-to-image samples generated by EG-FM-T2I at ×512, spanning architectural scenes, weather, interiors, animals, and human-centered compositions.

Findings

On ImageNet 256x256, PixelDiT-XL with EG-FM cut FID from 2.36 to 1.99 at 80 epochs, reached 1.55 at 200 epochs (already beating the baseline's 320-epoch result of 1.61), and improved further to 1.45 at 600 epochs.
In the same setting, DeCo-XL/16 reached FID 1.63 at 440 epochs versus the baseline's 1.69 at 600 epochs, and HyperDiT-H reached FID 1.51 at 220 epochs versus the baseline's 1.56 at 600 epochs.
At ImageNet 512x512, fine-tuning from 256x256 checkpoints for only 40 epochs gave PixelDiT with EG-FM an FID of 1.68 and Inception Score of 295.5 (versus 1.81 and 278.6 for the standard model trained 530 more epochs), while HyperDiT with EG-FM reached the best FID of 1.58 among compared methods.
On text-to-image generation, EG-FM-T2I raised GenEval from 0.78 (PixelDiT-T2I baseline) to 0.85 and improved DPG-Bench by 0.2 points to 83.9, achieving the best DPG-Bench score and second-best GenEval score (0.01 behind DeCo-XXL/16) among compared methods.
Measured training/inference overhead was negligible: per-sample compute increase was at most 0.0256%, per-step wall-clock time increase stayed mostly within 1% (except 4.81% for PixelDiT-B/16), and inference speed/FLOPs were unchanged since no extra computation is needed at sampling time.

Where it can be used

Improving training efficiency and final quality of pixel-space image generators that avoid lossy latent compression
Extending an already-trained low-resolution (256x256) generation model to high resolution (512x512) with only a small amount of additional fine-tuning
Text-to-image generation systems where both object composition accuracy and dense-prompt compliance need improvement

Limits and open work

Not yet evaluated on heterogeneous or temporally extended signals such as joint text-image modeling, video generation, or embodied decision-making with actions.
Untested on the largest recent foundation-model backbones such as Flux- or Qwen-Image-scale models.
The benefit is smaller when applied to x-prediction (predicting the clean image directly, as in JiT): FID improved only from 2.37 to 2.33, likely because early, noisy clean-image estimates make the per-image schedule less reliable.
Reported FID values use the ADM evaluation suite; other evaluators like torch-fidelity give slightly different absolute values (about 0.02-0.06 lower), though trends match.
Experiments are limited to ImageNet class-conditional generation and a specific text-to-image dataset (BLIP3o), so generalization to other data domains is not yet verified.

Why it matters

Because it changes only the generative trajectory and target, not the model architecture or training data, this approach could let pixel-space generators (which skip lossy latent compression to keep fine detail) train faster and reach better quality with negligible extra cost. That matters for anyone building or fine-tuning high-resolution image generators who wants efficiency gains without redesigning the backbone.

Terms in this paper

Flow Matching · A generative modeling approach that learns a path from random noise to a target image
FID · A score measuring how different generated images are from real ones; lower is better
heat-kernel · A smoothing filter derived from the heat-diffusion equation, used here to blur out high-frequency detail
low-frequency / high-frequency components · Low-frequency parts of an image capture overall structure; high-frequency parts capture fine texture and detail
CFG (classifier-free guidance) scale · A setting controlling how strongly a generation is steered by its class or text condition

Original abstract (English)

Pixel-space generative models bypass lossy latent compression, yet necessitate joint learning of global structure and fine-grained details in a high-dimensional space. Standard flow matching interpolates noise toward a fixed clean-image endpoint, leaving the spectral evolution to be learned implicitly. In this paper, we introduce Energy-Guided Flow Matching(EG-FM) that explicitly models a coarse-to-fine generative trajectory by moving endpoint. Specifically, EG-FM replaces the fixed endpoint with a heat-kernel-filtered endpoint that evolves smoothly from low-frequency image to clean image. The fraction of high-frequency signal in moving endpoint is released by an image-specific energy-guided scheduling, leading to the re-targeting of velocity in flow matching. Our framework requires no adaptation of the backbone and training data, bringing negligible cost on the training and inference stages. In our experiment, EG-FM consistently achieves lower FID on the ImageNet class-conditional image generation task at 256 times 256 with fewer epochs, reaching an FID of 1.55 at 200 epochs and 1.45 at 600 epochs. We continue training the generation task on the setting of 512 times 512 resolution, yielding a FID of 1.58 after only 40 high-resolution adaptation epochs. Furthermore, we transfer EG-FM on text-to-image generation and achieve 0.85 on GenEval score and 83.9 on DPG-Bench. Code is available at https://github.com/ysng123/EG-FM.

Authors · Haoyang Tong

Read on arXiv

Latest papers

All papers →

Latest from METAL LAB

Figures: Haoyang Tong et al., arXiv:2608.05811, arxiv-nonexclusive