Let a curriculum contain $N$ tasks trained sequentially, and let $b_i$ be the global step at
which training of task $i$ ends. Let $R_i(s)$ be the expert-normalised return of task $i$ at
step $s$, where $f = 0$ is the failure floor and $100$ is a privileged, state-based scripted
oracle policy reference. Scores are not capped above, so a model that outperforms the
reference policy takes values above 100. Define $C_i = R_i(b_i)$ as the task's performance
when its own training ends. Evaluation is run at a fixed checkpoint interval, so each integral
is computed by the trapezoidal rule over those checkpoints, and all curves are averaged over
three random seeds before a metric is applied.
Backward transfer. For every non-final task, we take the shaded region
between $C_i$ and the task's evaluation curve, and express it as a fraction of the rectangle
enclosing it:
$$ D_i = \frac{\displaystyle\int_{b_i}^{b_N} \max\!\left(0,\, C_i - R_i(s)\right) \mathrm{d}s}
{\left(b_N - b_i\right)\left(C_i - f\right)} \times 100. $$
The clipping at 0 means later gains do not offset earlier losses, so $D_i$ lies in
$[0, 100]$: 0 if the curve never drops after training, 100 if it collapses to the floor
immediately and stays there. The normalisation requires $C_i > f$, which means a task must end
with non-zero performance. If a task is never learnt, it holds nothing that could be forgotten
and backward transfer calculation is excluded. Averaging over the tasks that were learnt
yields the backward transfer score for this task suite, lower is better:
$$ \mathcal{I} = \left\{\, i < N : C_i > f \,\right\}, \qquad
\mathrm{BWT} = \frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} D_i. $$
Note that our BWT is a forgetting score in $[0, 100]$, and so is not the same quantity as the
backward transfer introduced in Lopez-Paz and Ranzato (2017), which subtracts a task's
original performance from its final performance and is higher-is-better. Theirs compares two
data points; ours integrates over the whole remaining curriculum.
Forward transfer. Forward transfer is measured only on the final composition
task to examine the reusing ability of the models. Let $R_N^{\mathrm{seq}}(t)$ be its learning
curve during its position in the curriculum, rebased to $t = 0$, and let
$R_N^{\mathrm{scr}}(t)$ be the curve obtained when learning the same task from scratch with no
preceding primitives. With a shared training budget of $T$ steps, FWT is the shaded region between them
divided by the learning length:
$$ \mathrm{FWT} = \frac{1}{T} \int_{0}^{T}
\left( R_N^{\mathrm{seq}}(t) - R_N^{\mathrm{scr}}(t) \right) \mathrm{d}t. $$
This is higher-is-better. Because the curves are compared over the whole budget rather than at
a fixed threshold, FWT rewards learning the composition task faster and learning it better.
