Procedure planning takes a visual initial state \(v_s\) and a visual goal state \(v_g\), encoded into
latents \(x_s\) and \(x_g\), and asks for the sequence of actions \(a_{1:T}\) connecting them. CEFITO
splits this into two parts: learning an action-conditioned energy field, and searching over it at
inference time.
We train a predictor \(P_\theta\) that maps the initial latent \(x_s\) and a candidate action sequence
\(a_{1:T}\) (encoded in language form with a frozen CLIP text encoder) to a predicted goal latent
\(\tilde{x}_g = P_\theta(x_s, a_{1:T})\). The distance \(d(P_\theta(x_s, a_{1:T}), x_g)\) then acts as an
energy: low for correct sequences, high for incorrect ones. Notably, this needs no
supervision of intermediate visual states.
Contrastive training. A margin-based triplet loss pulls the prediction for the ground-truth
sequence \(a^+_{1:T}\) towards \(x_g\), while pushing predictions for \(N\) wrong sequences
\(a^-_{1:T,i}\) away from it.
Mixed-negative sampling. Negatives are drawn as a mixture of hard negatives
(same task, wrong ordering or composition) and easy negatives (sequences from a different task),
controlled by a ratio \(r\). This imposes both low-level structure (action order) and high-level structure
(task separation) on the energy field.
Adaptive margins. A single fixed margin is suboptimal, since negatives differ in how
similar they are to the ground truth. We instead set the margin \(\tau_i\) per negative from its action
overlap with the positive sequence, so a reordered sequence is separated less aggressively than a sequence
of unrelated actions.
Auxiliary action reconstruction. Since intermediate visual states are unavailable, we feed
a learnable mask token per action step and reconstruct the action sequence from the resulting intermediate
latents through a decoding head \(D_\phi\), which encourages informative intermediate representations.
At inference, planning becomes a discrete search over the action space. Searching the full space is
expensive, so a lightweight MLP task classifier first predicts the high-level task \(\hat{c}\) from
\((x_s, x_g)\). This restricts the search to the task-specific action subset \(\mathcal{A}(\hat{c})\)
derived from the training data, and the plan is the candidate with minimal energy:
\[ \tilde{a}_{1:T} = \arg\min_{\tilde{a}_{1:T} \in \mathcal{A}(\hat{c})} \big\| P_\theta(x_s, \tilde{a}_{1:T}) - x_g \big\|_2 . \]
Unlike feed-forward or diffusion-based planners, this makes the reasoning over the action space explicit:
actions irrelevant to the predicted task are never considered.
