Action Prior in Backward Pass of Control as Inference in RL (1)

Remember

p(O_{t+1:T}|s_{t+1}) = \int p(O_{t+1:T}|s_{t+1}) p(a_{t+1}|s_{t+1}) da_{t+1}

We’ve defined

V_t(s_t) = \log \beta_t(s_t)

Q_t(s_t) = \log \beta_t(s_t,a_t)

We assumed action prior is uniform, but what if it is not?

V(s_t) = \log \int \exp(Q(s_t,a_t)+ \log p(a_t|s_t)) da_t

Q(s_t,a_t) = r(s_t,a_t) + \log \mathbb{E}[\exp(V(s_{t+1}))]

Now let

\tilde{Q}(s_t, a_t) = r(s_t, a_t) + \log p(a_t|s_t) + \log \mathbb{E}[\exp(V(s_{t+1}))]

V(s_t) = \log \int \exp(\tilde{Q}(s_t,a_t)) da_t = \log \int \exp(Q(s_t,a_t)+\log p(a_t|s_t))da_t

Oh! Now we’ve seen that with a modificatio to the reward function we can recover $V$ and $Q$ with a different action prior