Decision Transformer: Reinforcement Learning via Sequence Modeling
arXiv (Cornell University), 2021 · paper · AI and machine learning
Recasts reinforcement learning as sequence modeling, using a causally masked Transformer conditioned on desired return, past states and actions to output actions. The resulting Decision Transformer matches or exceeds leading model-free offline reinforcement learning baselines on Atari, OpenAI Gym and Key-to-Door tasks.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
10 findings
Substance In Table 2 the two BC averages appear swapped: "Average (Without Reacher)" prints 46.4 and "Average (All Settings)" prints 47.7, but the nine non-Reacher BC entries average 47.7 and all twelve average 46.4.
- p. 7 Table 2, BC column, average rows
46.4
- p. 7 Table 2, BC column, Average (All Settings)
47.7
- p. 7 Table 2 Average rows, BC column
46.4 ... 47.7
Judge's note: Recomputed from the BC column in the p.7 image. The DT and CQL averages reproduce from their columns with the same row definitions, so only the BC pair is out of place.
Second check: another model agreed
Report a problem with this finding
Substance BC on Medium Reacher is 48.9 in Table 2 but 58.4 for 100%BC in Table 3, although Section 5.1 defines 100%BC as standard BC trained on the entire dataset.
- p. 7 Table 2, Medium Reacher, BC
48.9
- p. 8 Table 3, Medium Reacher, 100%BC
58.4
Only the authors can say what is right here.
Judge's note: Checked p.7 and p.8. The other seven 100%BC entries in Table 3 equal the Table 2 BC values exactly. 48.9 appears in Table 3 as the 25%BC value. The Table 3 100%BC average (printed 39.5) is closer to the mean computed with 58.4 (39.6) than with 48.9 (38.4). Only the authors can say which value is correct.
Second check: another model agreed
Report a problem with this finding
Substance Table 5 gives DT (Ours) on Qbert as "15.1 ± 11.4", but Tables 1 and 4 give the same configuration as "15.4 ± 11.4".
- p. 9 Table 5, Qbert, DT (Ours)
15.1 ± 11.4
- p. 7 Table 1, Qbert, DT (Ours)
15.4 ± 11.4
- p. 8 Table 4, Qbert
15.4 ± 11.4
- p. 9
Qbert 15.1 ± 11.4
Judge's note: The Table 5 caption says K = 30 for Qbert, which is the main configuration (Table 8). The other three DT rows of Table 5 match Table 1 exactly, and so does the Qbert SD. The Table 12 raw score 2215.8 normalizes to 15.4, so 15.4 is the consistent value.
Second check: another model agreed
Report a problem with this finding
Substance Table 7 gives the Agnostic BC score for Medium-Expert Hopper as "59.9", but Table 2 gives BC on Medium-Expert Hopper as "79.6". 59.9 is Table 2's Medium-Expert HalfCheetah BC value.
- p. 10 Table 7, Medium-Expert Hopper, BC (Agnostic)
59.9
- p. 7 Table 2, Medium-Expert Hopper, BC
79.6
Judge's note: The text calls BC reward-agnostic, so its score should not change under delayed reward. The other Table 7 Hopper BC values match Table 2 (63.9 and 27.6), as do the DT dense values (107.6, 67.6, 82.7) and the CQL dense values (111.0, 58.0, 48.6). That makes a copy from the wrong row the evident explanation.
Second check: another model agreed
Report a problem with this finding
Substance Table 3 prints the DT (Ours) 'Average' as "56.1", but the eight DT entries in the table average 54.9.
- p. 8 Table 3, Average row, DT column
56.1
Judge's note: On the page 8 image, the DT column reads 42.6, 67.6, 74.0, 51.2, 36.6, 82.7, 66.6 and 18.0. These sum to 439.3, and 439.3/8 = 54.9. The 10%BC average (56.7) and the CQL average (43.5) both reproduce from their columns. Rounding of the inputs moves a mean by at most 0.05, so it cannot explain a gap of 1.2. Nothing in the paper defines a different averaging set. Side observation, not part of this cluster: the 100%BC average is printed as 39.5, while its column averages 39.6.
Second check: another model agreed
Report a problem with this finding
Substance Table 4 gives Pong 10%BC as "2.5 ± 0.2", but the raw score in Table 13 ("−20.3 ± 0.1") normalizes to about 1.9 using the Table 11 baselines.
- p. 8 Table 4, Pong, 10%BC
2.5 ± 0.2
- p. 21 Table 13, Pong, 10%BC
−20.3 ± 0.1
Only the authors can say what is right here.
Judge's note: Appendix B (p.20) says Table 13 holds the raw scores behind Table 4, and Table 11 gives Pong random = −21 and gamer = 15. Normalizing −20.3 gives 1.9. The other Pong cells convert correctly: −16.2 gives 13.3 and 5.2 gives 72.8 (printed 72.7). The value 2.5 ± 0.2 is the same as the DT K=1 Pong entry in Table 5. Only the authors can say whether the raw or the normalized value is the wrong one.
Second check: another model agreed
Report a problem with this finding
Substance Table 4 gives Qbert 40%BC as "11.8 ± 5.8", but the raw score in Table 13 ("1674.1 ± 776.0") normalizes to 11.4 using the Table 11 baselines.
- p. 8 Table 4, Qbert, 40%BC
11.8 ± 5.8
- p. 21 Table 13, Qbert, 40%BC
1674.1 ± 776.0
Only the authors can say what is right here.
Judge's note: Table 11 gives Qbert random = 164 and gamer = 13455. Normalizing 1674.1 gives 11.36. The spread is consistent: 776/13291 gives 5.8. The neighbouring Qbert cells convert correctly: 10%BC 1045 gives 6.6 and 25%BC 2302.5 gives 16.1 (printed 16.0). A gap of 0.4 is too large to be a rounding effect.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 3 (Training) says predicting states or returns-to-go "is easily permissible within our framework (as shown in Section 5.4)". The return-token prediction experiment is in Section 5.5, not 5.4.
- p. 5 Section 3, Training paragraph
(as shown in Section 5.4)
Judge's note: I read Section 5.4 (p.9: Key-to-Door credit assignment, actions only, Table 6) and Section 5.5 (p.10: 'We modify Decision Transformer to output return tokens in addition to action tokens'). The pointer should be to Section 5.5.
Second check: another model agreed
Report a problem with this finding
Reference or label Section 4.1 says "we report the mean and standard deviation of 3 seeds", but the captions of Tables 1, 2, 4, 12 and 13 say "mean and variance" for the same ± values.
- p. 6 Section 4.1
report the mean and standard deviation of 3 seeds
- p. 7 Table 1 caption
We report the mean and variance across 3 seeds
- p. 7 Table 2 caption
We report the mean and variance for three seeds
- p. 8 Table 4 caption
We report the mean and variance across 3 seeds.
Judge's note: Checked the text on p.6 and the captions on pp.7, 8 and 21. The ± values in Table 1 are a linear rescaling of those in Table 12 (27.3/28*100 = 97.5; 2.9/36*100 = 8.1), which is how a standard deviation behaves; a variance would scale with the square of the factor. The 'variance' label in the captions therefore appears to be wrong.
Second check: another model agreed
Report a problem with this finding
Wording A.3 states "Since we require a return prompt to generate actions and we do assume knowledge of the optimal path length upfront, we use a simple prior", but p.20 says the approach "avoids the need for any external or oracle information like the optimal path length". The first sentence presumably should read "do not assume".
- p. 19 A.3, second paragraph
we do assume knowl- edge of the optimal path length upfront
- p. 20 A.3 continued
avoids the need for any external or oracle information like the optimal path length
- p. 19 A.3
we do assume knowl-
Judge's note: Checked pp.19-20. The reasoning only works if the optimal path length is not known, which is why a prior over returns is introduced. The continuation on p.20 explicitly says no oracle path-length information is needed. As printed, the sentence reverses the intended meaning.
Second check: another model agreed
Report a problem with this finding
24 minor (typos and the like; most checked by the judge only)
Reference or label The Figure 3 caption says "across Atari, OpenAI Gym, and Minigrid", but the third bar group is labelled "Key-To-Door" and the paper never mentions Minigrid elsewhere.
- p. 6 Figure 3 caption and x-axis
across Atari, OpenAI Gym, and Minigrid
- p. 6 Figure 3 caption / x-axis
Atari, OpenAI Gym, and Minigrid
Judge's note: Checked the p.6 image: the x-axis labels are Atari, OpenAI Gym and Key-To-Door. Searching the text finds 'Minigrid' only in this caption; everywhere else the paper calls the environment Key-to-Door (Mesnard et al. [12]). Key-to-Door may be built on MiniGrid, but the paper does not say so, so the caption and axis label disagree.
Second check: another model agreed
Report a problem with this finding
Reference or label The same work is listed twice: [18] "Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019." and [30] "Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019." The two entries also have different author lists.
- p. 15 Reference [18]
Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949
- p. 15 Reference [30]
Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, 2019.
- p. 15 References [18]
Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949, 2019.
Judge's note: Checked p.15. Both entries have the same title and year, and [18] omits Matthew Soh while [30] includes him. [18] is cited as BEAR in Sections 4 and 5, and [30] in the related work section on p.? ("[29, 30, 31]"). Because the two author lists differ, at least one of them must be wrong, so this goes beyond a cosmetic duplicate. Which author list is correct rests on outside knowledge.
Second check: another model agreed
Report a problem with this finding
Wording Section 2.1 indexes the trajectory from 0, "τ = (s0, a0, r0, ..., sT, aT, rT)", but writes the expected return as E[Σ_{t=1}^T r_t], which leaves out r0. Eq. (2) also starts at R̂1, s1.
- p. 4 Section 2.1
τ = (s0, a0, r0, s1, a1, r1, . . . , sT , aT , rT )
- p. 4 Section 2.1 expected return
hPT t=1 rt i
Judge's note: Checked on the p.4 image. The trajectory starts at index 0, while the objective sum and Eq. (2) start at t=1. This is a notational inconsistency, and either convention would work if applied consistently.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.1 says DT "outperforms or matches REM, QR-DQN, and BC on all 4 games", but in Table 1 BC's Qbert mean (17.3 ± 14.7) is above DT's (15.4 ± 11.4).
- p. 6 Section 4.1, last sentence
outperforms or matches REM, QR-DQN, and BC on all 4 games
- p. 7 Table 1, Qbert row
17.3 ± 14.7
- p. 6 Section 4.1
outperforms or matches REM, QR-DQN, and BC on
- p. 7 Table 1, Qbert
15.4 ± 11.4 104.2 0.0 0.0 17.3 ± 14.7
Judge's note: Checked Table 1 (p.7) and Table 4 (p.8). Table 4 bolds 100%BC's 17.3 as the best Qbert score, and the Table 1 caption says only 'in most games'. 'Matches' could be argued within the large spread, so the wording overstates rather than misstates.
Second check: another model agreed
Report a problem with this finding
Wording Section 4.1 says "we report numbers directly from the CQL and REM papers", while Appendix B says "we report raw data provided to us by REM authors" for REM and QR-DQN.
- p. 6 Section 4.1
we report numbers directly from the CQL and REM papers
- p. 20 Appendix B
we report raw data provided to us by REM authors
Judge's note: Checked p.6 and p.20. Appendix B also notes a discrepancy between Agarwal et al. and Kumar et al. The two statements describe the baseline source differently. The main-text wording is loose rather than clearly wrong.
Second check: another model agreed
Report a problem with this finding
Wording Table 1 gives DT Pong as "106.1 ± 8.1", but normalizing the Table 12 raw score 17.1 with Table 11 (random −21, gamer 15) gives 105.8.
- p. 7 Table 1, Pong, DT
106.1 ± 8.1
- p. 21 Table 12, Pong, DT
17.1 ± 2.9
Only the authors can say what is right here.
Judge's note: Checked Tables 1, 11 and 12 on pp.7, 20 and 21. The other Pong entries (CQL 19.3 to 111.9; the SD 2.9 to 8.1) and the Breakout and Qbert DT entries convert as expected. Reaching 106.1 needs a raw mean of about 17.2. The gap is small, and the authors would need to say which figure is right.
Second check: another model agreed
Report a problem with this finding
Wording The Table 2 caption says "Decision Transformer (DT) outperforms conventional RL algorithms on almost all tasks", but a TD baseline has a higher mean on 5 of the 12 rows.
- p. 7 Table 2 caption
outperforms conventional RL algorithms on almost all tasks
- p. 7 Table 2 caption
Decision Transformer (DT) outperforms conventional RL algorithms on almost all tasks.
Judge's note: Counted from the p.7 image. DT is beaten on ME Hopper (CQL 111.0), M HalfCheetah (BRAC 46.3), M Walker (BRAC 81.1), MR HalfCheetah (BRAC 47.7) and MR Reacher (CQL 19.0). The body text's 'a majority of the tasks' fits 7 of 12; 'almost all' overstates.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.2 says "On every task, the desired target returns and the true observed returns are highly correlated", but the Qbert panel of Figure 4 is nearly flat.
- p. 8 Section 5.2
On every task, the desired target returns and the true observed returns are highly correlated.
- p. 9 Figure 4, Qbert panel
Qbert
Only the authors can say what is right here.
Judge's note: In the Figure 4 crop, the Qbert DT curve stays at roughly 10–25 while the target return runs from 0 to 100, far below the oracle line. Reacher shows only a weak rise. So 'every task … highly correlated' is stronger than the figure shows. The claim for the other panels is not in question.
Second check: another model agreed
Report a problem with this finding
Wording Section 5.3 says "performance of Decision Transformer is significantly worse when K = 1", but for Qbert Table 5 shows 15.1 ± 11.4 vs "13.6 ± 11.3".
- p. 9 Sec 5.3
performance of Decision Transformer is significantly worse when K = 1
- p. 9 Table 5, Qbert
13.6 ± 11.3
Judge's note: Table 5 on page 9 shows large gaps for Breakout, Pong and Seaquest. For Qbert the difference is 1.5, against spreads of about 11. The paper reports no significance test, so the blanket 'significantly worse' wording overstates the Qbert result.
Second check: another model agreed
Report a problem with this finding
Wording Table 9 gives the DT Gym learning-rate decay as "Linear warmup for first 10^5 training steps", but A.2.1 says "Models were trained for 10^5 gradient steps". The warmup therefore spans the whole of training.
- p. 18 A.2.1
Models were trained for 105 gradient steps
- p. 19 Table 9, Learning rate decay
Linear warmup for first 105 training steps
Only the authors can say what is right here.
Judge's note: Checked p.18 (A.2.1) and p.19 (Table 9). The superscript reads 10^5 in both places. A warmup lasting the entire run is not impossible, but it does not fit the phrase "first ... steps". Only the authors can say which value was intended.
Second check: another model agreed
Report a problem with this finding
Wording A.2.2 says BC was trained "for 2.5 × 10^4 gradient steps", yet Table 10 gives the BC learning-rate decay as "Linear warmup for first 10^5 training steps". As stated, the warmup is longer than the whole training run.
- p. 19 A.2.2 text
2.5 × 104 gradient steps
- p. 19 Table 10, Learning rate decay
Linear warmup for first 105 training steps
- p. 18 A.2.1
Models were trained for 105 gradient steps
- p. 19 Table 10 / A.2.2
We train for 2.5 × 104 gradient steps
Only the authors can say what is right here.
Judge's note: Checked p.19: the A.2.2 text and Table 10 are as quoted. The warmup could never finish. Table 10's wording is identical to Table 9's, which suggests it was copied. Only the authors know the value actually used.
Second check: another model agreed
Report a problem with this finding
Wording A.3 defines "P_prior(R̂ = k) ∝ T + 1 − k" as "a simple prior over returns that favors shorter paths", but the same section says returns-to-go "correspond to negative path lengths". If k is the return value (k ≤ 0), the weight T+1−k grows as paths get longer.
- p. 19 A.3
Pprior( ˆR = k) ∝T + 1 −k
Only the authors can say what is right here.
Judge's note: Checked p.19. Rewards are -1 per step and returns-to-go are negative path lengths, so R̂ = k ≤ 0. Then T+1−k = T+1+|k|, which is larger for longer paths: k=-1 gives T+2, k=-5 gives T+6. The prior favours shorter paths only if k means path length (−R̂), which the notation does not say. Please ask the authors to clarify the definition of k or the sign.
Second check: another model agreed
Report a problem with this finding
Typo Subject–verb disagreement on p.3: "empirical evidence suggest that a transformer modeling approach..." should read "suggests".
- p. 3 Introduction, third paragraph
empirical evidence suggest
Judge's note: The crop of p.3 shows the text as quoted. 'Evidence' is singular, so the verb should be 'suggests'.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.2 reads "Score are normalized so that 100 represents an expert policy"; this should be "Scores are".
- p. 7 Section 4.2
Score are normalized
- p. 7 Sec 4.2, last paragraph
Score are normalized so that 100 represents an expert policy
Judge's note: Seen on the p.7 page image.
Second check: none (judge only)
Report a problem with this finding
Typo There are copy-editing slips: "highsight return information" (p.9, for 'hindsight'), "but did this does not make" (p.18, A.1) and "for the model it generate" (p.19, A.3, for 'to generate').
- p. 9 Section 5.4
highsight return information
- p. 9 Section 5.4
fixed content window
- p. 11 Section 5.8
model and generative a diverse
- p. 18 A.1
did this does not make
Judge's note: All three appear on the page images and crops. The Table 6 caption spells the word 'hindsight'. 'content window', 'generative' and 'disciminators' are judged under C022, C027 and C029.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.4 reads "rather than having a fixed content window" where 'context window' is meant.
- p. 9 Section 5.4
fixed content window
- p. 9 Sec 5.4
rather than having a fixed content window
Judge's note: The text is visible on the page 9 image. The sentence contrasts this with using the entire episode as the context, and the paper uses 'context length' elsewhere.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.6 contains the ungrammatical sentence "Delayed returns minimally affect Decision Transformer; and due to the nature of the training process, while imitation learning methods are reward agnostic."
- p. 10 Section 5.6
and due to the nature of the training process, while imitation learning methods are reward agnostic.
- p. 10 Sec 5.6
Delayed returns minimally affect Decision Transformer; and due to the nature of the training process, while imitation learning methods are reward agnostic.
- p. 10 Sec 5.6
and due to the nature of the training process, while imitation learning methods are reward agnostic
Judge's note: The sentence is visible on the page 10 image. The clause beginning 'and due to' has no main verb, and the clause beginning 'while' is left dangling. The intended meaning can be partly recovered from context.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.8 reads "simultaneously model and generative a diverse set of behaviors"; the verb should be 'generate'.
- p. 11 Section 5.8
and generative a diverse set of behaviors
- p. 11 Sec 5.8
simultaneously model and generative a diverse set of behaviors
- p. 11
model and generative a diverse set of behaviors
Judge's note: The text is visible in the crop of page 11.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.2 has the misspelling "goal-conditoned online RL settings" for 'goal-conditioned'.
- p. 11 Section 6.2
goal-conditoned online RL settings
Judge's note: The misspelling is visible in the crop of page 11.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.4 has the misspelling "learn disciminators to guide generation" for 'discriminators'.
- p. 12 Section 6.4
disciminators
- p. 12 Sec 6.4
learn disciminators to guide
Judge's note: The misspelling is visible in the crop of page 12.
Second check: none (judge only)
Report a problem with this finding
Typo Section 6.4 reads "it is more natural to prompt the model desired target return", with a word missing ('with the').
- p. 12 Sec 6.4
prompt the model desired target return
Judge's note: The text is visible in the crop of page 12.
Second check: none (judge only)
Report a problem with this finding
Typo A.1 contains a stray word: "but did this does not make a significant difference in performance". It should read "but this does not make".
- p. 18 Appendix A.1
but did this does not make
Judge's note: Seen on the p.18 page image, A.1 second paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo A.3 reads "This makes it possible for the model it generate its own (realizable) returns-to-go". It should read "for the model to generate".
- p. 19 Appendix A.3
possible for the model it generate its own
- p. 19 A.3
for the model it generate its own
Judge's note: Seen on the p.19 page image, A.3 second paragraph.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B says "Table 11 shows the normalized scores used for normalization used in Hafner et al. [21]". Table 11 actually lists raw Random/Gamer baseline scores (its caption reads "Atari baseline scores used for normalization"), and the sentence repeats "used".
- p. 20 Appendix B
Table 11 shows the normalized scores used for normalization used in Hafner et al. [21]
Judge's note: Checked p.20: the Table 11 values (e.g. Pong −21/15) are raw reference scores, not normalized scores. This is a wording slip; the data are unaffected.
Second check: none (judge only)