Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
arXiv (Cornell University), 2018 · paper · AI and machine learning
Proposes soft actor-critic, an off-policy actor-critic algorithm based on maximum entropy reinforcement learning, where the agent maximizes reward while acting as randomly as possible. It reports state-of-the-art results on continuous control benchmarks and stable performance across random seeds.
Checked: PDF from arxiv.org, fetched 4 Oct 2026. Page numbers are those of that PDF.
3 findings
Substance Equation 14: the inner sum over l with expectation over s_l, a_l has the summand "[r(s_t, a_t) + αH(π( · |s_t))|s_t, a_t]", indexed by t rather than l.
- p. 11 Appendix A, Equation 14
γl−t Esl∼p,al∼π [r(st, at) + αH(π( · |st))|st, at]
Only the authors can say what is right here.
Judge's note: Read Equation 14 on the page 11 image: the subscripts inside the bracket are t, while the sum index and the sampled variables are l. Reworking it as printed: the summand is a function of the conditioning variables only, so the inner expectation equals c = r(s_t,a_t) + αH(π(·|s_t)) for every l, and the inner sum is c·Σ_{l≥t} γ^{l−t} = c/(1−γ). J(π) is then the undiscounted objective of Equation 1 scaled by 1/(1−γ), not the "discounted expected reward and entropy for future states" described in the next sentence. The intended summand is presumably r(s_l,a_l) + αH(π(·|s_l)). Nothing downstream uses the printed form.
Second check: another model agreed
Report a problem with this finding
Wording In the proof of Lemma 1, "r_π(s_t, a_t) ≜ r(s_t, a_t) + E_{s_{t+1}∼p}[H(π( · |s_{t+1}))]" has no factor γ on the entropy term, so Equation 15 is not the backup defined by Equations 2–3.
- p. 11 Appendix B.1, proof of Lemma 1 (definition before Equation 15)
rπ(st, at) ≜r(st, at) + Est+1∼p [H (π( · |st+1))]
- p. 4 Section 4.1, Equations 2 and 3
T πQ(st, at) ≜r(st, at) + γ Est+1∼p [V (st+1)]
Only the authors can say what is right here.
Judge's note: Read Equations 2 and 3 on page 4 and the proof on page 11. Substituting Eq. 3 into Eq. 2: T^πQ = r + γE_{s'}[E_{a'}[Q(s',a') − log π(a'|s')]] = r + γE_{s'}[H(π(·|s'))] + γE[Q(s',a')]. Eq. 15 with the printed r_π gives r + E_{s'}[H] + γE[Q]. The two differ by (1−γ)E_{s'}[H], which is non-zero for γ<1 and a policy with non-zero entropy. Defining r_π with γ on the entropy term makes Eq. 15 identical to Eq. 2 and the standard policy-evaluation argument goes through, so Lemma 1 itself stands.
Second check: another model agreed
Report a problem with this finding
Wording The proof of Theorem 1 asserts strict inequalities, "J_{π*}(π*( · |s_t)) < J_{π*}(π( · |s_t)) for all π ∈ Π, π ≠ π*" and "Q^{π*}(s_t, a_t) > Q^π(s_t, a_t) for all (s_t, a_t)", whereas the theorem states ≥ and only the non-strict forms follow.
- p. 12 Appendix B.3, Theorem 1 statement
Qπ∗(st, at) ≥Qπ(st, at) for all π ∈Π
- p. 12 Appendix B.3, proof
we get Qπ∗(st, at) > Qπ(st, at) for all (st, at) ∈S × A
- p. 12 Appendix B.3, proof of Theorem 1
Jπ∗(π∗( · |st)) < Jπ∗(π( · |st)) for all π ∈Π, π ̸= π∗
Only the authors can say what is right here.
Judge's note: Read on the page 12 image. At convergence π* minimises J_{π*} over Π at each state, which gives ≤ only. Counterexample checked against the stated assumptions: take two absorbing states and a policy π that equals π* at s1 and differs at s2. Then π ≠ π*, but J_{π*}(π(·|s1)) = J_{π*}(π*(·|s1)) and Q^π(s1,a) = Q^{π*}(s1,a), so both strict inequalities fail. The Lemma 2 argument (Eqs. 18–19) propagates ≥ only. The theorem as stated uses ≥ and stands once < and > are replaced by ≤ and ≥.
Second check: another model agreed
Report a problem with this finding
16 minor (typos and the like; most checked by the judge only)
Wording Section 5.1 says SAC "outperforms them on the harder tasks with a large margin, both in terms of learning speed and the final performance", but in Figure 1(e) (Humanoid-v1) the PPO curve reaches about the same level as SAC, or slightly above it, by 10 million steps.
- p. 7 Section 5.1, second paragraph
outperforms them on the harder tasks with a large margin, both in terms of learning speed and the final performance
- p. 6 Figure 1(e) Humanoid-v1 and caption
outperforming both on-policy and off-policy methods in the most challenging tasks
- p. 8 Section 6 Conclusion
outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy PPO algorithm
Judge's note: Viewed Figure 1 on page 6 and the Section 5.1 text on page 7. In panel (e) the brown PPO curve rises to roughly 6000-6500 by the end, overlapping or slightly exceeding the orange SAC curve; SAC is clearly faster. The large-margin final-performance claim holds for Ant-v1 and Humanoid (rllab) but not visibly for Humanoid-v1. The sentence is hedged with "overall" and the values are read by eye from a plot, so this is a mild overstatement only.
Second check: another model agreed
Report a problem with this finding
Wording Appendix E says the deterministic ablation "assumes a deterministic policy in the value update (Equation 6) and the policy update (Equation 13)", and Section 5.2 says "we updated the target value network weights with hard updates", but the Figure 4 caption says it "does not use a value function ... and uses hard target updates for the target Q-functions".
- p. 14 Appendix E, text
assumes a deterministic policy in the value update (Equation 6) and the policy update (Equation 13)
- p. 14 Figure 4 caption
does not use a value function, drops the entropy terms in the actor and critic function updates, and uses hard target updates for the target Q-functions
- p. 14 Appendix E text
a deterministic ablation which assumes a deterministic policy in the value update (Equation 6) and the policy update (Equation 13)
- p. 7 Section 5.2, Stochastic vs. deterministic policy
we updated the target value network weights with hard updates
Only the authors can say what is right here.
Judge's note: Read the Appendix E text and Figure 4 caption on page 14 and Section 5.2 on page 7. The body text refers to a value update (Equation 6 is the V_ψ gradient) and a target value network, while the caption says there is no value function and that target Q-functions are used. I found no statement in the paper that reconciles the two. With a deterministic policy and no entropy term V(s) reduces to Q(s, π(s)), so the descriptions may be functionally close, but only the authors can say which architecture was run.
Second check: another model agreed
Report a problem with this finding
Typo Section 2 prints "they use it as an regularizer"; the article should be "a regularizer".
- p. 2 Section 2, first paragraph
they use it as an regularizer
Judge's note: Read on the page 2 image, right column, first paragraph of the actor-critic discussion. The wording is as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.2 (continuation on page 6) prints "for all hyperparameter)"; it should read "for all hyperparameters".
- p. 6 Section 4.2, continuation text below Figure 1
for all hyperparameter)
- p. 2 Section 2, first paragraph
they use it as an regularizer
- p. 7 Section 5.1, first paragraph
The solid curves corresponds to the mean
Judge's note: Read on the page 6 image, first line of text below Figure 1. The other two slips grouped into this cluster ("an regularizer", "curves corresponds") are real but are judged under C001 and C014; this entry covers the missing plural only.
Second check: none (judge only)
Report a problem with this finding
Typo Theorem 1 reads "soft policy improvement from any π ∈ Π" in Section 4.1 but "soft policy improvement to any π ∈ Π" in the Appendix B.3 restatement.
- p. 4 Section 4.1, Theorem 1
soft policy improvement from any
- p. 12 Appendix B.3, Theorem 1
soft policy improvement to any π ∈Π
Judge's note: Both crops viewed. The two statements of the same numbered theorem differ by one preposition ("from any" versus "to any"). The mathematical content is the same, so this is a wording inconsistency only.
Second check: none (judge only)
Report a problem with this finding
Typo The Duan et al. reference reads "Duan, Y., Chen, X. Houthooft, R., Schulman, J., and Abbeel, P." with no comma after "Chen, X."; separately, Section 4.1 has "In the other words, in the policy improvement step".
- p. 9 References, Duan et al. 2016
Duan, Y., Chen, X. Houthooft, R., Schulman, J., and Abbeel
- p. 9 Duan reference
Duan, Y., Chen, X. Houthooft, R.
- p. 4 Section 4.1
In the other words, in the policy improvement step
Judge's note: The page 9 image shows the missing author separator. The page 4 image shows "In the other words" in Section 4.1, and the same phrase recurs in Appendix C on page 12. Both are cosmetic.
Second check: none (judge only)
Report a problem with this finding
Typo Equation 9 prints the gradient factor as "∇θQθ(at, st)", with arguments reversed relative to Qθ(st, at) used later in the same equation and everywhere else.
- p. 5 Section 4.2, Equation 9
∇θQθ(at, st)
Judge's note: Read on the crop and the page 5 image. Equation 9 is printed as ∇θQθ(a_t, s_t)(Qθ(s_t, a_t) − r(s_t, a_t) − γV(s_t+1)). The argument order is inconsistent within the equation; this is notational and the meaning is not affected.
Second check: none (judge only)
Report a problem with this finding
Typo Section 4.2 prints "which is represented by a neural network an can be differentiated"; "an" should be "and".
- p. 5 Section 4.2, paragraph after Equation 10
network an can be differentiated
- p. 5 Section 4.2
represented by a neural network an can be differentiated
- p. 5
which is represented by a neural network an can be differentiated
Judge's note: Read on the page 5 image, left column, paragraph after Equation 10. The wording is as quoted.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 1 caption says "Soft actor-critic (yellow) performs consistently across all tasks", but the SAC curve in the legend and panels is drawn in orange.
- p. 6 Figure 1 caption
Soft actor-critic (yellow) performs consistently across all tasks
Judge's note: Page 6 image: the legend lists SAC first and its curve is orange in all six panels; no curve is yellow. Colour naming is partly subjective and the legend identifies the curve unambiguously, so this is a cosmetic point.
Second check: none (judge only)
Report a problem with this finding
Typo The Figure 1 caption prints "performs consistently across all tasks and outperforming both on-policy and off-policy methods"; the verb forms do not match ("performs ... and outperforming").
- p. 6 Figure 1 caption
performs consistently across all tasks and outperforming both on-policy
Judge's note: Read on the page 6 image and the caption crop. The non-parallel verb form is as printed; it should be "outperforms" or "..., outperforming".
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.2, "Reward scale", prints "Larger reward magnitudes correspond to lower entries."; "entries" is evidently a slip for "entropies".
- p. 7 Section 5.2, paragraph 'Reward scale'
Larger reward magnitudes correspond to lower entries.
Judge's note: Read on the crop and the page 7 image. The surrounding sentences say reward scale acts as the temperature and controls the policy's stochasticity (near-uniform for small rewards, near-deterministic for large). Nothing in the paragraph could be called "entries", so the intended word is "entropies". It is a wrong-word slip that the context makes recoverable, hence S4.
Second check: none (judge only)
Report a problem with this finding
Typo Section 5.1 prints "The solid curves corresponds to the mean"; it should read "curves correspond".
- p. 7 Section 5.1, first paragraph
The solid curves corresponds to the
Judge's note: Read on the page 7 image, first paragraph of Section 5.1. The subject-verb disagreement is as printed.
Second check: none (judge only)
Report a problem with this finding
Typo Figure 3 caption: "the mean action does not, in general, correspond the optimal action for the maximum return objective" is missing "to" after "correspond".
- p. 8 Figure 3 caption, part (a)
correspond the optimal action for the maximum return objective
Judge's note: Read the Figure 3 caption on the page 8 image; the word "to" is absent in the PDF itself.
Second check: none (judge only)
Report a problem with this finding
Typo The Kingma & Ba reference gives the venue as "International Conference for Learning Presentations (ICLR), 2015".
- p. 9 References, Kingma & Ba
International Conference for Learning Presentations (ICLR), 2015
- p. 9 Kingma & Ba reference
International Conference for Learning Presentations (ICLR)
Only the authors can say what is right here.
Judge's note: Rests on outside knowledge: that ICLR stands for International Conference on Learning Representations. The page 9 image prints "International Conference for Learning Presentations (ICLR)" exactly as quoted, so it is not an extraction artefact.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix B.3 has "it must be case that" (missing "the") and Appendix C has "the actions needs to be bounded" (subject-verb disagreement).
- p. 12 Appendix B.3, proof
it must be case that
- p. 12 Appendix C, first paragraph
the actions needs to be bounded
Judge's note: Both are visible on the page 12 image. Copy-edit slips only.
Second check: none (judge only)
Report a problem with this finding
Typo Appendix E opens "Figure 4 compares SAC to Trust-PCL (Figure 4. Trust-PC fails to solve most of the task within the given number of environment steps": an unclosed, self-referential parenthesis, "Trust-PC" for "Trust-PCL", and "task" for "tasks".
- p. 14 Appendix E, first two sentences
Figure 4 compares SAC to Trust-PCL (Figure 4. Trust-PC fails to solve most of the task
- p. 14 Appendix E first sentence
Trust-PCL (Figure 4. Trust-PC fails to solve most of the task
- p. 14
Trust-PC fails to solve most of the task
Judge's note: The page 14 image shows the unclosed "(Figure 4.", the truncated name "Trust-PC" and the singular "task"; "if ran longer" follows in the same sentence. All are in the PDF, not extraction artefacts.
Second check: none (judge only)