21. f st ,at( )= f st ,at( )+ ∇a f st ,a( )a=at
at − at( )
∇θ J = Eρ,π ∇θ logπθ at st( ) Q st ,at( )− f st ,at( )( )⎡
⎣
⎤
⎦ + Eρ,π ∇θ logπθ at st( ) f st ,at( )⎡
⎣
⎤
⎦
= Eρ,π ∇θ logπθ at st( ) Q st ,at( )− f st ,at( )( )⎡
⎣
⎤
⎦ + Eρ,π ∇a f st ,a( )a=at
∇θ µθ st( )⎡
⎣
⎤
⎦
22.
23. ∇θ J = Eρ,π ∇θ logπθ at st( ) Q st ,at( )−Qw st ,at( )( )⎡
⎣
⎤
⎦ + Eρ,π ∇aQw st ,a( )a=at
∇θ µθ st( )⎡
⎣
⎤
⎦
∇θ J = Eρ,π ∇θ logπθ at st( ) A st ,at( )− Aw st ,at( )( )⎡
⎣
⎤
⎦ + Eρ,π ∇aQw st ,a( )a=at
∇θ µθ st( )⎡
⎣
⎤
⎦
a
24. ∇θ J = Eρ,π ∇θ logπθ at st( ) A st ,at( )− Aw st ,at( )( )⎡
⎣
⎤
⎦ + Eρ,π ∇aQw st ,a( )a=at
∇θ µθ st( )⎡
⎣
⎤
⎦
Aw = Qw st ,at( )− Eπ Qw st ,at( )⎡⎣ ⎤⎦
= Qw st ,µθ st( )( )+ ∇aQw st ,a( )a=µθ st( )
at − µθ st( )( )− Eπ Qw st ,µθ st( )( )+ ∇aQw st ,a( )a=µθ st( )
at − µθ st( )( )⎡
⎣⎢
⎤
⎦⎥
= ∇aQw st ,a( )a=µθ st( )
at − µθ st( )( )
rt+1 +γV st+1( )−V st( )
Eπ at[ ]= µθ st( )
25.
26. m*
= m −η(t −τ )
E m*
⎡⎣ ⎤⎦ = E m[ ]
Var m*
⎡⎣ ⎤⎦ = Var m[ ]− 2ηCov m,t[ ]+η2
Var t[ ]
η*
=
Cov m,t[ ]
Var t[ ]
27. ∇θ J = Eρ,π ∇θ logπθ at st( ) A st ,at( )−η st( )Aw st ,at( )( )⎡
⎣
⎤
⎦ +
Eρ,π η st( )∇aQw st ,a( )a=at
∇θ µθ st( )⎡
⎣
⎤
⎦
Var A −ηAw⎡⎣ ⎤⎦ = Var A[ ]− 2ηCov A,Aw( )+η2
Var Aw( )
η*
=
Cov A,Aw( )
Var Aw( )