Conservative Q-Learning for Offline Reinforcement Learning

• p. 3 Theorem 3.1

  Thus, if α is sufficiently large

• p. 2 Theorem 3.1 support assumption

  For any $\mu(\ba|\bs)$ with $\supp \mu \subset \supp \hatbehavior$

• p. 17 Proof of Theorem 3.1, Equation 11

  − α μ(a|s)/π̂β(a|s)

Only the authors can say what is right here.

Judge's note: Read Theorem 3.1 on p3 and its proof on p17, plus tex block 2. The proof says "since μ(a|s) > 0, α > 0, π̂β(a|s) > 0" and picks α via "α · min_{s,a}[μ(a|s)/π̂β(a|s)] ≥ max ...", which needs min μ/π̂β > 0. The theorem only requires support inclusion, and since the paper assumes π̂β(a|s) > 0 for all a, that condition restricts nothing. Reworked the counterexample: one state, two actions, self-loop, γ=1/2, π̂β=(1/2,1/2), μ=(1,0), π=(0,1). Eq. 11 gives Q̂(s,a2) = r̂ + γQ̂(s,a2) − α·0, so Q̂ = r̂/(1−γ) = 2 when r̂=+1, while Qπ(s,a2)=0 for a zero-mean reward. The penalty term [(I−γPπ)^{-1} μ/π̂β](s,a2) is 0 because π never leaves a2. Overestimation therefore persists for every α. The displayed inequality still holds; the "sufficiently large α" corollary fails without a positivity condition on μ, which appears only inside the proof.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 21 Proof of Theorem D.1, final display

  f(π*) = 0

• p. 21 Opening paragraph

  any α > 0 is sufficient to obtain a lower bound

• p. 7 Claimed nonnegativity of the projected penalty

  \policy(\ba|\bs)^T P_\Qfeat \left[\frac{\policy(\ba|\bs) - \behavior(\ba|\bs)}{\behavior(\ba|\bs)} \right] \geq 0

• p. 21 Appendix D.1, proof of Theorem D.1, last display

  ∀π(a|s), f(π) = π(a|s)T PF

Only the authors can say what is right here.

Judge's note: I read p.21 and tex block 7 and reworked the step. The stationary-point argument gives a minimum only if f is convex, and that is not shown. Counterexample: one state, two actions, π_β=(0.5,0.5), a single feature F=(1,0)^T. Then D=diag(0.5,0.5), F^TDF=0.5 (invertible), P_F=[[1,0],[0,0]]. For π=(0.25,0.75), h=(−0.5,0.5) and f = 0.25·(−0.5) = −0.125 < 0. A second check with F=(1,2)^T and π=(0.9,0.1) gives P_F=[[0.2,0.4],[0.4,0.8]] and f=−0.176. A negative penalty means the fitted value can exceed the LSTD-Q value for every α>0, so the step 'V̂^{k+1} ≤ V̂^{k+1}_LSTD-Q' (p.22) and the remark on p.21 fail. The same claim is reused for term (b) in Theorem D.2 (p.23). In the tabular case F=I the quantity is Σ(π−π_β)²/π_β ≥ 0, so the claim fails only under function approximation. A stated theorem is left without a valid proof.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 23 Equation 17 and following paragraph

  we can show that (b) ≥ 0

• p. 7 Equation 17, penalty

  \eta \alpha_k \policy(\ba|\bs)^T \bM^k D \left[ \frac{\policy(\ba|\bs) - \behavior(\ba|\bs)}{\behavior(\ba|\bs)} \right]

Only the authors can say what is right here.

Judge's note: Page 23 image (Eq 17 and following paragraph). Counterexample reworked: one state, πβ=(0.5,0.5), π=(0.9,0.1), D=diag(0.5,0.5), gradients (1,2) so M=[[1,2],[2,4]] and ∇Q^T D ∇Q = 2.5 (invertible). D h = (0.4,−0.4); M D h = (0.4−0.8, 0.8−1.6) = (−0.4,−0.8); π^T M D h = 0.9·(−0.4)+0.1·(−0.8) = −0.44 < 0. So (b) ≥ 0 is false in general; the claim rests on the Theorem D.1 analysis that fails in the same way (C059). Distinct location and theorem, so not a duplicate. Additionally the Theorem D.2 statement on page 22 ends 'lower-bounds the corresponding tabular function iterate if:' with no condition printed. The stated result is left without a valid proof.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 26 Lemma D.4.1 statement

  |D(s)|

• p. 27 End of proof of Lemma D.4.1

  Combining these arguments together, we obtain the following upper bound

• p. 26 Lemma D.4.1

  √|A| / |D(s)|

• p. 8 Lemma D.4.1 statement

  \frac{\sqrt{|\mathcal{A}|}}{{|\mathcal{D}(\bs)|}}

Only the authors can say what is right here.

Judge's note: I compared the p.26 and p.27 images; tex block 8 has {|\mathcal{D}(\bs)|} without \sqrt in the lemma statement. Both steps of the proof bound terms that scale as 1/√|D(s)|, and the closing display has √|D(s)|. Since 1/|D(s)| ≤ 1/√|D(s)|, the lemma as printed is a stronger bound than the proof delivers (e.g. 0.01 vs 0.1 at |D(s)|=100). Theorem D.4 and Theorem 3.6 use the √ form and are unaffected. The stated lemma is wrong as printed.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 27 Final paragraph completing Theorem D.4

  which justifies the factor of 2

• p. 8 Final paragraph

  upper bounding the sampling error for $\hatbehavior$ by the corresponding sampling error for $\policy^*$

• p. 26 Theorem D.4, Equation 27

  Theorem D.4.

Only the authors can say what is right here.

Judge's note: Viewed p.26 (Eq. 27, Lemma D.4.1) and p.27 (final paragraph). The lemma's bound for a policy π is an expectation under d^π_M̂. For π̂_β it is under d^{π̂_β}_M̂ with integrand √(|A|/|D(s)|); for π* it is under d^{π*}_M̂. The inequality √(D_CQL(π*,π̂_β)(s)+1) ≥ 1 compares integrands only, and the paper nowhere assumes a common state distribution. I rechecked the reviewer's counterexample numerically (γ=0.9, |A|=2, counts 1000/8100/900, D_CQL(s0)=1/9): B(π*)/K ≈ 0.1·0.04472·1.0541 + 0.9·0.01571 ≈ 0.01886, and B(β)/K ≈ 0.1·0.04472 + 0.81·0.01571 + 0.09·0.04714 ≈ 0.02144, which is larger. So the step that yields the factor 2 in Eq. 27 fails as argued, and what the lemma does give is B(π*)+B(π̂_β). Theorem D.4 / 3.6 with the factor 2 is therefore left without a valid proof. Whether the inequality itself is false is not shown.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 31 Table 6, row hopper-mixed, column CQL(H)

  1563.2

• p. 31 Table 7, row hopper-mixed, column CQL(H) (Lagrange)

  1866.1

• p. 8 Table 1, row hopper-mixed, CQL(H)

  48.6

• p. 31 Table 7, row hopper-mixed, Lagrange column

  hopper-mixed 1866.1

Judge's note: Viewed page 31 (Tables 6 and 7), page 30 (Table 5) and the Table 1 crop. Table 6 CQL(H): hopper-medium-expert 3628.4, hopper-mixed 1563.2, hopper-medium 1866.1. Table 7 Lagrange column: hopper-medium-expert 3628.4, hopper-mixed 1866.1 (bold), walker-mixed 1227.2, hopper-medium 1866.1. The two columns agree exactly on hopper-medium-expert and hopper-medium, and Table 7's 1227.2 matches Table 5's walker2d-mixed, so they report the same configuration; Appendix F states that the Section 6 experiments use the Lagrange version. I found no statement of a different run or seed set for Table 7. Table 1 gives CQL(H) normalized scores of 48.6 for hopper-mixed and 58.0 for hopper-medium. Both are the same hopper environment, so hopper-mixed's raw return must be lower than hopper-medium's 1866.1, which fits 1563.2 and not 1866.1. The Table 7 cell therefore looks like a copy of the hopper-medium row. The text's "clear benefits, for instance ... hopper-mixed dataset" rests on this cell: the gap over the fixed-α column falls from 863.3 to 560.4 with the Table 6 value, so the direction is unchanged. I did not use the reviewers' conversion of normalized score to raw return, since it relies on D4RL reference scores that are not in the paper.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 27 last paragraph of Appendix D.4

  which means the sampling error term for π∗pointwise upper bounds the

• p. 26 Theorem D.4, Equation 27

  J(π∗, M) ≥J(ˆπβ, M)−2

Only the authors can say what is right here.

Judge's note: Read Lemma D.4.1 and Theorem D.4 on page 26 and the closing paragraph on page 27. In the lemma, the expectation for a policy π is taken under d^π_{M̂}. So the term for π̂β is E over d^{π̂β}_{M̂} and the term for π* is E over d^{π*}_{M̂}. The argument "D_CQL(π̂β,π̂β)=0, hence √(D_CQL(π*,π̂β)+1) ≥ √(D_CQL(π̂β,π̂β)+1)" only compares the integrands state by state. I checked the reviewer's counterexample: start state s0 with |D(s0)|=100 and π̂β(a1)=0.99; a1 leads to absorbing s1 with |D|=99, a2 leads to absorbing s2 with |D|=1; γ=0.99. Then err(π*)∝0.01·√1.0101/10+0.99/√99≈0.1005 and err(π̂β)∝0.01/10+0.99(0.99/√99+0.01)≈0.1094, so err(π̂β) > err(π*). The step bounding err(π̂β) by err(π*) is therefore false in general, and Eq. 27 is left without a valid proof as written (it would need to keep the two terms separate). I have not shown that the theorem itself is false.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 6 Section 4, Implementation details

  standard implementations of soft actor-critic (SAC) [21]

• p. 11 Reference [21]

  T. Haarnoja, H. Tang, P. Abbeel, and S. Levine. Reinforcement learning with deep energy-based policies.

• p. 6 Section 4, Implementation details

  soft actor-critic (SAC) [21]

• p. 7 Section 6, first paragraph

  SAC [21], an off-policy actor-critic method

Only the authors can say what is right here.

Judge's note: Rests on outside knowledge: that the 2017 'Reinforcement learning with deep energy-based policies' paper introduces soft Q-learning and is not the soft actor-critic paper, which is a separate work. From the files: pages 6 and 7 cite SAC as [21], and the page 11 reference list gives [21] as quoted. A search of the text finds no other Haarnoja entry in the reference list, and [21] is also used for SAC in Section 4's algorithm description, Appendix F, and for Q-ensembles ('[21, 18]') on page 9.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 9 Table 3 caption

  Table 3: CQL, REM and QR-DQN in setting (1) with 1% data (top), and 10% data (bottom).

• p. 9 Section 6, Atari paragraph

  When only using only 1% or 10% of the data, in setting (2) (Table 3)

• p. 9 Table 3 caption

  CQL, REM and QR-DQN in setting (1) with 1% data (top), and 10% data (bottom)

• p. 9 Section 6, Atari paragraph, definition of settings

  (2) datasets consisting of only 1% and 10% of all samples observed by an online DQN agent

Judge's note: Viewed the p9 image. The body text defines "(1) a dataset consisting of the first 20% of the samples" and "(2) datasets consisting of only 1% and 10% of all samples". The Figure 1 caption uses setting (1) for the 20% data, and the body cites Table 3 for setting (2). The table caption's "setting (1)" is therefore a label slip and should read (2). No alternative numbering of the settings appears in the paper. The duplicated "only" is a separate cosmetic slip in the same sentence.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 16 Figure 2 caption

  that CQL (left) generally has negative values of ∆, whereas BEAR (right) generally has positive ∆

• p. 16 Figure 2 caption

  Note that CQL (left) generally has negative values of ∆, whereas BEAR (right) generally has positive ∆

• p. 16 Figure 2 caption

  CQL (left) generally has negative values

• p. 16 Figure 2 caption

  CQL (left) generally has negative values of ∆, whereas BEAR (right) generally has positive ∆

Judge's note: Page 16 image: sub-figures (a) hopper-expert-v0 and (b) hopper-medium-v0 each have a left panel with y-axis Delta^k and a right panel with y-axis Average Return, and each panel's legend shows both BEAR (blue) and CQL (orange). No reading of the layout makes CQL "left" and BEAR "right". The body text itself uses "Figure 2(a) (right)" for the return plot, which contradicts the caption's usage. The caption's claim that BEAR's values "also increase during training" holds for hopper-expert (rising to above 30) but not visibly for hopper-medium, where the BEAR curve stays roughly flat near 4; the body text only claims the rise for Figure 2(a).

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Appendix C, first sentence of page

  Next, we prove Theorem 3.3 that shows that the additional term that maximizes the expected Q-value

• p. 18 Appendix C

  Proof of Theorem 3.2.

• p. 18 first paragraph, top of page

  Next, we prove Theorem 3.3 that shows that the additional term

Judge's note: Page 18 image: the sentence names Theorem 3.3, and the next paragraph is 'Proof of Theorem 3.2'. Page 19 has a separate 'Proof of Theorem 3.3' (the slow-policy-change result, tex lines 295-301). In the tex (line 767) the sentence uses \ref{thm:cql_underestimation}, the label of Theorem 3.3, instead of thm:cql_underestimates (Theorem 3.2). Wrong pointer; fix is 'Theorem 3.2'.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 18 Appendix C, proof of Theorem 3.2

  tighter than the expression for policy value in Proposition 3.2

• p. 18 Appendix C, proof of Theorem 3.2, after the fixed-point expression

  this expression is tighter than the expression for policy value in Proposition 3.2

• p. 18 after the fixed-point derivation of V^pi

  the expression for policy value in Proposition 3.2

Judge's note: Viewed p18. A text search of the whole article finds the word "Proposition" only in this sentence; all results are numbered as Theorems. The sentence is inside the proof of Theorem 3.2 and compares against "the previous proof", which is the proof of Theorem 3.1 (Equation 1 bound). As printed it points to a non-existent item, and read as 'Theorem 3.2' it would compare the result with itself. Intended target is Theorem 3.1.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 2 Section 2, policy improvement equation

  Es∼D,a∼πk(a|s)

• p. 2 Preliminaries

  \hat{\policy}^{k+1} \leftarrow& \arg\max_{\policy} \E_{\bs \sim \mathcal{D}, \ba \sim \policy^k(\ba|\bs)}\left[\hat{Q}^{k+1}(\bs, \ba)\right]

Judge's note: I checked the crop of p.2 and tex block 2. Both show a∼π^k(a|s) under arg max_π. The prose just above ('improving the policy π(a|s) by updating it towards actions that maximize the expected Q-value') needs a∼π(a|s). The intent is clear and the fix is evident: replace π^k with π in the subscript.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 3 Section 3.1, Theoretical analysis

  corresponding entry is a very large but finite value δ ≥2Rmax

• p. 3 Section 3.1, Theoretical analysis

  and δ ∈(0, 1)

Judge's note: Viewed page 3. Both uses appear in the 'Theoretical analysis' paragraph, a few lines apart. The two definitions are incompatible (δ in (0,1) versus δ at least 2R_max/(1−γ)), so the symbol is reused for a different quantity. A different symbol is needed for the fill-in value.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 4 Section 3.2, Variants of CQL

  R(µ) = −DKL(µ, ρ)

• p. 14 Appendix A, CQL(ρ) optimization problem

  Ex∼µ(x)[f(x)] + DKL(µ||ρ) s.t.

• p. 14 Appendix A, CQL(ρ) optimization problem

  Ex∼µ(x)[f(x)] + DKL(µ||ρ)

• p. 5 Appendix A

  \E_{\bx \sim \mu(\bx)}[f(\bx)] + D_{\mathrm{KL}}(\mu || \rho)

Only the authors can say what is right here.

Judge's note: The p.14 image and tex block 5 show +D_KL. Page 4 shows R(µ) = −D_KL(µ, ρ). I reworked the step. E_μ[f] − KL(μ||ρ) is concave in μ, and setting its derivative to zero under the simplex constraint gives μ ∝ ρ·exp(f). E_μ[f] + KL(μ||ρ) is convex in μ, so its maximum lies at a vertex of the simplex. Example: ρ=(0.5,0.5), f=0. The printed objective is 0 at μ=ρ (its minimum) and log 2 at μ=(1,0). The appendix display has the wrong sign; the stated solution and Eq. 4 stand once it is corrected.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 22 Theorem D.2

  lower-bounds the corresponding tabular function iterate if:

• p. 4 Section 3.1, after Theorem 3.2

  we present these results in Theorem D.1 and Theorem D.2 in Appendix D.1

Only the authors can say what is right here.

Judge's note: Viewed p22: after "if:" there is only white space and then "Proof."; no displayed condition on α_k was lost in extraction. The proof (p22-23) does not state an explicit condition either, saying only that α_k can be chosen "as in the proof of Theorem D.1". The main text (p4) relies on this theorem for the non-linear case. Only the authors can supply the missing condition.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 CQL(ρ), unnumbered optimization

  Eₓ∼μ(x)[f(x)] + D_KL(μ||ρ)

• p. 5 CQL(ρ) derivation

  \E_{\bx \sim \mu(\bx)}[f(\bx)] + D_{\mathrm{KL}}(\mu || \rho)

Only the authors can say what is right here.

Judge's note: Read the display on the p14 image and in tex block 5 ("+ D_{\mathrm{KL}}(\mu || \rho)"). Reworked: maximising E_μ[f] − KL(μ||ρ) over the simplex gives μ* ∝ ρ exp(f). With +KL the objective is convex in μ and its maximum sits at a vertex. Counterexample checked: two outcomes, f=(0,0), ρ=(1/2,1/2). The claimed μ*=ρ gives objective 0, while μ=(1,0) gives KL = log 2 > 0. So the printed problem does not have the stated solution. The main text on p4 uses −D_KL, so this is a sign slip in the appendix display. The stated solution stands for the intended objective, hence S3.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 14 CQL(ρ), Equation 7

  modulo some normalization terms

• p. 5 Equation 7 first expectation

  \E_{\ba \sim \rho(\ba|\bs)} \left[Q(\bs, \ba) \frac{\exp(Q(\bs, \ba))}{Z}\right]

• p. 14 Appendix A, Equation 7 and the sentence before it

  (modulo some normalization terms)

Only the authors can say what is right here.

Judge's note: I read p.14 and tex block 5. Working it through with R = −KL (the main-text sign): KL(μ*||ρ) = E_{μ*}[log(exp Q/Z)] = E_{μ*}[Q] − log Z. So the inner maximum is E_{μ*}[Q] − KL = log Z = log E_ρ[exp Q]. Eq. 7 keeps E_{μ*}[Q], which differs from this by a Q-dependent amount, so it is not a normalisation term. Example: ρ uniform over 2 actions, Q=(0, ln 3): Z=2, log Z≈0.693, Eq. 7 term = 0.75·ln 3 ≈ 0.824. The main text describes CQL(ρ) as an 'exponential weighted average', so the authors may intend Eq. 7 as a definition (it matches log Z in gradient only if μ* is held fixed). As a derivation from Eq. 3, though, the step is not exact. Query to the authors.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 17 Equation 12

  ˆQk+1(s, a) = Bπ ˆQk(s, a) ≤Bπ ˆQk(s, a) −α µ(a|s)

• p. 17 Equation 12

  Q̂ᵏ⁺¹(s, a) = B^πQ̂ᵏ(s, a)

• p. 5 Equation 12

  \hat{Q}^{k+1}(\bs, \ba) = \bellman^\policy \hat{Q}^k(\bs, \ba) \leq

• p. 17 Appendix C, Equation 12

  ˆQk+1(s, a) = Bπ ˆQk(s, a) ≤Bπ ˆQk(s, a) −α

Only the authors can say what is right here.

Judge's note: The p.17 image and tex block 5 both show an un-hatted B^π and no penalty in the middle expression, so this is not an extraction artefact. Eq. 11 on the same page gives Q̂^{k+1} = B̂^πQ̂^k − αµ/π̂β. Written as X ≤ X − p + c, the printed chain holds iff p ≤ c, but the proof later picks α so that p ≥ c. The intended line, B̂^πQ̂^k − αµ/π̂β ≤ B^πQ̂^k − αµ/π̂β + C…, is correct, so Theorem 3.1 stands.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 6 Section 4, Implementation details

  the learning rate for the policy is chosen to be 3e-5 (vs 3e-4 or 1e-4 for the Q-function)

• p. 29 Appendix F, Hyperparameters

  chosen to be 3e-5 or 1e-4 for continuous control tasks

• p. 29 Appendix F, Policy learning rate bullet

  We chose 3e −5 as the default across all environments.

• p. 29 Hyperparameters, policy learning rate

  [3e −5, 1e−4, 3e−4.

Only the authors can say what is right here.

Judge's note: Viewed the p6 crop and the p29 page image. The three statements do not agree on which rate (policy or Q-function) took two values, or on whether the policy rate was 3e-5 everywhere or 1e-4 for some tasks. Appendix F notes that 1e-4 "seemed to be better" on some tasks, which may explain "3e-5 or 1e-4", but it then says 3e-5 was the default across all environments. Only the authors can say which values produced the reported results. The unclosed bracket in the policy-learning-rate bullet is visible on p29 and is cosmetic.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 6 Algorithm 1, line 4

  φt := φt−1 + ηπEs∼D,a∼πφ(·|s)[Qθ(s, a)−log πφ(a|s)]

Judge's note: Viewed page 6. The image shows ∇_θ in line 3 and no ∇_φ in line 4, so as printed the update adds a scalar objective value to the parameters. The intended meaning is clear from the surrounding text.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 19 Proof of Theorem 3.4, last displayed equation

  −Eµk(a|s)[Bπk ˆQk(s, a)]−αk ˆ∆k.

• p. 20 Proof of Theorem 3.4, first display and α_k condition

  (πβ(a|s) −µk(a|s))T h

• p. 20 Appendix C, proof of Theorem 3.4, display 'for, αk satisfying'

  (πβ(a|s) −µk(a|s))T

• p. 6 proof of Theorem gap_amplify

  \alpha_k > \max \left(\frac{(\behavior(\ba|\bs) - \mu_k(\ba|\bs))^T \left[ \bellman^{\policy^k} \left( \hat{Q}^k - Q^k \right)(\bs, \cdot) \right]}{\hat{\Delta}^k}, 0\right)

Only the authors can say what is right here.

Judge's note: I checked pp.19–20 and tex block 6. From E_µ[Q̂^{k+1}] = E_µ[BQ̂^k] − α_kΔ̂^k and E_β[Q̂^{k+1}] = E_β[BQ̂^k] − 0, subtraction gives +α_kΔ̂^k. Δ̂^k is defined as ≥0, so its sign cannot absorb the difference. Combining with the tabular iterate gives E_βQ̂ − E_βQ = E_µQ̂ − E_µQ + (π_β−µ_k)^T(a) + α_kΔ̂^k, so the strict inequality needs α_k > (µ_k−π_β)^T(a)/Δ̂^k. Counterexample to the printed condition: (a)=(2,0), π_β=(0.5,0.5), µ=(1,0). Then Δ̂=1 and the printed threshold is max(−1,0)=0, but α=0.5 gives −1+0.5 < 0. Theorem 3.4 in the main text is qualitative ('for large enough α_k') and still holds; the explicit condition in the proof does not.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 20 Appendix C, final display of the proof of Theorem 3.4

  + DTV( ˆπβ, µk) · 2 · Cr,T,δRmax

• p. 6 proof of Theorem gap_amplify

  {\hat{\Delta}^k} + D_{\mathrm{TV}}(\hat{\behavior}, \mu_k) \cdot \frac{2 \cdot C_{r, T, \delta} R_{\max}}{1 - \gamma}, 0\right)

Only the authors can say what is right here.

Judge's note: I checked p.20 and tex block 6. The requirement has the form α_kΔ̂^k > b + e, where e is the sampling contribution: an action-distribution difference applied to a vector bounded by 2C R_max/(1−γ). Solving for α_k gives (b+e)/Δ̂^k. The printed b/Δ̂^k + e is smaller whenever Δ̂^k < 1, so as printed it is not a sufficient condition. Example: Δ̂=0.1, b=0, e=1 requires α_k > 10, but the printed condition gives α_k > 1. The qualitative theorem is unaffected.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 22 Proof of Theorem D.1, displayed chain, second line

  choose αk to make this negative

• p. 7 Second line of the inequality chain

  - \underbrace{\E_{d^\behavior(\bs)}\left[\hat{V}^{k+1}_{\text{LSTD-Q}}(\bs) - V^{k+1}(\bs) \right] - \alpha_k

• p. 7 proof of Theorem policy_eval_func_approx

  \leq \E_{d^\behavior(\bs)}\left[V^{k+1}(\bs)\right] - \underbrace{\E_{d^\behavior(\bs)}\left[\hat{V}^{k+1}_{\text{LSTD-Q}}(\bs) - V^{k+1}(\bs) \right] - \alpha_k \E_{d^\behavior(\bs)}[f(\policy(\ba|\bs))]}

Only the authors can say what is right here.

Judge's note: The p.22 image and tex block 7 both show the minus sign. Write A=E[V̂_LSTD], B=E[V], c=α_kE[f]. Then line 1 is A−c and the correct rewriting is B + (A−B−c). As printed, line 2 is B−(A−B)−c, which differs from line 1 by 2(A−B) and is not an upper bound when A > B. The α_k formula that follows, (A−B)/E[f], matches the plus version, so this is a repairable sign slip in the display.

Second check: another model agreed · checked by hand

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 23 Paragraph after the proof of Theorem D.2

  Qᵏ⁺¹ = Q̂ᵏ − η(B^πQ̂ᵏ − Q̂ᵏ)

• p. 7 Tabular counterpart

  Q^{k+1} = \hat{Q}^k - \eta \left(\bellman^\policy \hat{Q}^k - \hat{Q}^k \right)

• p. 23 Appendix D.1, paragraph after the proof of Theorem D.2

  Qk+1 = ˆQk −η

• p. 7 after proof of corr:nonlinear_ntk

  $Q^{k+1} = \hat{Q}^k - \eta \left(\bellman^\policy \hat{Q}^k - \hat{Q}^k \right)$

Only the authors can say what is right here.

Judge's note: I checked p.23 and tex block 7. The vectorised update and Eq. 17 term (a) on the same page both have +η M^k D(B^πQ̂^k − Q̂^k). The derivative of ½(Q−y)² is Q−y, so descent gives Q + η(y−Q). The printed expression moves away from the target. The sign slip is local and no later formula depends on it.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 8 Table 2 caption

  On the harder mazes, CQL is the only method that attains non-zero returns

• p. 8 Table 2, row antmaze-medium-diverse, column BEAR

  8.0

• p. 8 Table 2, antmaze-medium-diverse row, BEAR column

  antmaze-medium-diverse

• p. 8 Table 2, antmaze-medium-diverse row

  0.0 0.0 8.0 0.0 0.0 53.7 5.1

Judge's note: Viewed page 8 image. In the antmaze-medium-diverse row the BEAR column reads 8.0 (BC 0.0, SAC 0.0, BRAC-p 0.0, BRAC-v 0.0, CQL(H) 53.7, CQL(rho) 5.1). The body text calls the 'medium and large mazes' the harder ones, so the medium maze is covered by the caption's claim. The body's softer 'meaningful progress' is compatible; the caption's literal 'non-zero' with italic 'only' is not. No explanation elsewhere.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 26 Proof of Theorem D.4, first display

  [DCQL(π∗, πβ)(s)] ≥J(ˆπβ, ˆ

• p. 26 Equation 26

  1 −γ Es∼dπ

• p. 26 proof of Theorem D.4, first display

  J(π∗, ˆ M) −αEs∼dπ

• p. 26 Equation 26

  −α 1 1 −γ

Only the authors can say what is right here.

Judge's note: I checked p.26 and tex block 8. Eq. 26 has α·1/(1−γ)·E[D_CQL(π,π̂β)]. Comparing π* with π̂_β (zero penalty) gives J(π*,M̂) − α/(1−γ)·E[D_CQL(π*,π̂β)] ≥ J(π̂β,M̂). The printed inequality lacks the factor, yet Eq. 27 then reinstates α/(1−γ). Restoring the factor repairs the step, and the theorem stands.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send

• p. 27 Proof of Lemma D.4.1, first line of the norm-bound chain

  ||Δd^π_M||₁

• p. 8 First inequality in the norm-bound chain

  \leq \sum_{\bs, \bs'} \left\vert \Delta(\bs'|\bs) \right\vert d^\policy_{\hat{M}}

• p. 27 proof of Lemma D.4.1, first line of the ||Δd||_1 display

  ||∆dπ M||1

• p. 8 proof of the lemma

  \vert\vert \Delta d^\policy_M||_{1} &= \sum_{\bs'} \left\vert \sum_{\bs} \Delta(\bs'|\bs) d^\policy_{M}(\bs) \right\vert \leq \sum_{\bs, \bs'} \left\vert \Delta(\bs'|\bs) \right\vert d^\policy_{\hat{M}}

Only the authors can say what is right here.

Judge's note: Checked the page 27 image and tex block 8. The triangle inequality gives Σ_{s,s'}|Δ(s'|s)| d_M(s), with the same distribution on both sides, not d_{M̂}. I reworked the reviewer's counterexample with column-stochastic P_M=[[1/2,0],[1/2,1]], P_M̂=[[0,0],[1,1]], γ=1/2 and ρ=(1,0). This gives d_M=(2/3,1/3), d_M̂=(1/2,1/2) and Δ=[[1/2,0],[-1/2,0]], so Δd_M=(1/3,-1/3). The left side is 2/3 and the right side is 1/2, so the printed inequality fails. The preceding display, "d_M − d_M̂ = (1−γ)γGΔḠμ = γḠΔd_M̂", also mixes terms: (1−γ)γGΔḠρ equals γGΔd_M̂, and the alternative identity is γḠΔG, which gives γḠΔd_M. If the chain starts from ||Δd_M̂||_1 (using γGΔd_M̂), the rest of the chain holds and gives the lemma as stated, with expectation under d_M̂. The step is faulty but the stated result can stand, so S3.

Second check: another model agreed

Report a problem with this finding

This finding is

wrong right unclear or overstated already corrected (erratum, later version)

I am an author of this paper Send