1 Peking University 2 JD Explore Academy
Rewards for robot reinforcement learning, learned from how progress
stalls, regresses, and recovers in real executions.
Event2Reward learns task progress from successful demonstrations and the events in real robot executions. Temporal labels anchor the progress scale; local relations teach the model how progress changes through stalls, regressions, and recoveries.
Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.
Learning task progress from robot demonstrations provides a practical way to construct dense shaping rewards for robotic reinforcement learning. A common approach uses normalized demonstration time as progress labels, implicitly assuming monotonically increasing task progress. During policy exploration, however, actual task progress can stall or even regress. Such models may mistake regressions for progress, assigning positive shaping rewards to actions that cause progress loss. Real suboptimal and failed executions offer corrective experience, but assigning absolute progress values to their states is difficult. Our key observation is that stalls, regressions, and recoveries establish local relative progress relations even when absolute progress is unknown. We introduce Event2Reward, which uses normalized temporal labels to anchor the progress scale and these within-trajectory relations to constrain local changes, combining absolute and relative progress supervision. Its progress predictions serve as potentials for reward shaping. We curate an event-annotated dataset from approximately 100K raw real-robot execution records, covering successful, suboptimal, and failed executions, without requiring framewise absolute progress labels. Under the same training budgets, Event2Reward improves average policy success by 18.16 percentage points over a baseline using only sparse task rewards across 40 tasks in four LIBERO suites. On real robots, average success after 30 minutes of wall-clock training reaches 60.0% on three precise insertion tasks and 90.0% on three object manipulation tasks, exceeding the sparse-reward baseline under the same budget by 36.7 and 23.3 percentage points, respectively. Ablation studies further show that absolute progress supervision and relative progress relations are complementary, with joint supervision yielding better downstream policy learning.
Real-world reinforcement learning
The first 10 trials are a human-guided warm-up. From trial 11 on, the robot learns on its own, rewarded by Event2Reward’s progress estimate. After 15 min 16 s it inserts the plug 10 times in a row; with sparse rewards alone this takes about 45 min.
Method
Event2Reward does not need to know what fraction of a failed attempt is done. It only needs to know, for two moments, whether progress is higher, lower, or about the same.
In a successful execution without events, each frame is labelled with its normalized time, y_t = t / T: 0 at the start, 1 at completion. These labels set the progress scale.
In an execution with an event, the annotation only marks where the event starts, a, and where it ends, r (a failed execution has no r; the interval runs to the end), and states how progress inside compares with a: about the same, lower, or not higher. No frame gets a number.
Pick a kind and an episode. Signs between stills: the relation the annotation requires. Numbers: Event2Reward’s prediction, which may differ. Click a still or drag the curve to move the video.
Blue curve: Event2Reward’s prediction. Black points a, r and the shading between them: the annotation. Hollow dots: frames sampled for comparison; red line or band: what the relation compares against. Held-out LIBERO test episodes, not used in training; videos play at 2×.
Experiments
Matched training budgets, evaluated in simulation and on real robots.
Average policy success across four LIBERO suites over sparse rewards.
Three precise insertion tasks after 30 minutes. Sparse: 23.3%; Robometer (finetune): 37.8%.
Three object manipulation tasks after 30 minutes. Sparse: 66.7%; Robometer (finetune): 81.7%.
| Suite | Sparse | Robo-Dopamine (one-shot) | Robometer (finetune) | Event2Reward |
|---|---|---|---|---|
| Spatial | 36.80 ± 6.80 | 36.72 ± 4.77 | 34.04 ± 5.03 | 46.92 ± 6.94 |
| Object | 47.40 ± 3.13 | 57.56 ± 6.09 | 60.56 ± 6.53 | 72.16 ± 2.97 |
| Goal | 66.36 ± 3.39 | 64.00 ± 3.88 | 56.44 ± 5.67 | 76.32 ± 3.57 |
| LIBERO-10 | 26.84 ± 8.80 | 31.72 ± 4.92 | 40.88 ± 19.00 | 54.64 ± 8.70 |
200K environment steps for Spatial, Object, and Goal; 1.2M for LIBERO-10.
Success rate (%) during training. Shaded bands: ±1.96 SEM over five seeds.
Real-robot episodes
Each task shows one success and one failure, with Event2Reward's progress for both.
Place the two outer bowls into the central bowl to form a stack of three.
Success rate after 30 min of training; mean over 3 runs × 20 episodes
Event2Reward 100% Robometer (finetune) 100% Robo-Dopamine (one-shot) 90% Sparse 70%
Pick up the cabbage, the corn and the pepper and place all three on the plate.
Success rate after 30 min of training; mean over 3 runs × 20 episodes
Event2Reward 75% Robometer (finetune) 60% Robo-Dopamine (one-shot) 60% Sparse 50%
Press the blue, pink and yellow buttons in that order.
Success rate after 30 min of training; mean over 3 runs × 20 episodes
Event2Reward 95% Robometer (finetune) 85% Robo-Dopamine (one-shot) 93.3% Sparse 80%
Use the released models, or fine-tune Event2Reward on your own robot.
The event-annotated dataset will be released later.
@misc{nie2026event2reward,
title = {Event2Reward: Learning Robotic Rewards from Progress Relations},
author = {Nie, Junnan and Fan, Zitian and Zhang, Jiachen and Lao, Junyi and Zhang, Likui and Liu, Chenghao and Lu, Xiaoyang and Wang, Yunbo and Zhang, Yixue and Zhai, Linbo and Li, Jiayi and Wang, Mengzhe and Wu, Xichang and Wang, Bin and Liu, Chang and Zhang, Tianle and Huang, Songfang},
year = {2026},
eprint = {2610.39212},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.39212}
}