Introduction

SARSA is a classic on-policy reinforcement learning algorithm. Its name comes from the update tuple State, Action, Reward, State, Action, which reflects exactly what the algorithm uses to update its Q-values. The central idea is that the agent learns from the action it actually plans to take next under its current policy.

The Update Rule

The tabular SARSA update is:

Q(s, a) = Q(s, a) + alpha * (r + gamma * Q(s_next, a_next) - Q(s, a))

The important difference from Q-learning is the use of a_next. SARSA does not update toward the greedy action in the next state unless the policy actually chooses that action.

That is why SARSA is called on-policy.

A Small Tabular Implementation

Here is a simple Python implementation with an epsilon-greedy policy and a toy environment.

1import random
2
3class SimpleEnv:
4    def __init__(self):
5        self.state = 0
6        self.terminal_state = 4
7
8    def reset(self):
9        self.state = 0
10        return self.state
11
12    def step(self, action):
13        # action 0 moves left, action 1 moves right
14        if action == 1:
15            self.state = min(self.state + 1, self.terminal_state)
16        else:
17            self.state = max(self.state - 1, 0)
18
19        reward = 1 if self.state == self.terminal_state else -0.01
20        done = self.state == self.terminal_state
21        return self.state, reward, done
22
23
24def epsilon_greedy(Q, state, epsilon, n_actions):
25    if random.random() < epsilon:
26        return random.randrange(n_actions)
27    return max(range(n_actions), key=lambda a: Q[state][a])
28
29
30def sarsa(env, episodes=200, alpha=0.1, gamma=0.99, epsilon=0.1):
31    n_states = 5
32    n_actions = 2
33    Q = [[0.0 for _ in range(n_actions)] for _ in range(n_states)]
34
35    for _ in range(episodes):
36        state = env.reset()
37        action = epsilon_greedy(Q, state, epsilon, n_actions)
38        done = False
39
40        while not done:
41            next_state, reward, done = env.step(action)
42
43            if done:
44                td_target = reward
45                Q[state][action] += alpha * (td_target - Q[state][action])
46                break
47
48            next_action = epsilon_greedy(Q, next_state, epsilon, n_actions)
49            td_target = reward + gamma * Q[next_state][next_action]
50            Q[state][action] += alpha * (td_target - Q[state][action])
51
52            state, action = next_state, next_action
53
54    return Q
55
56env = SimpleEnv()
57q_table = sarsa(env)
58for i, row in enumerate(q_table):
59    print(i, row)

This is not a production RL framework. It is the clearest way to see the algorithm itself.

Why the Next Action Matters

Suppose your policy is still exploratory. SARSA updates with the value of the exploratory next action, not with the value of the best-looking action in the table.

That makes SARSA more conservative in some environments because it learns the value of behaving according to its actual policy, including exploration.

Tuning the Main Parameters

The three core hyperparameters are:

• 'alpha: learning rate'
• 'gamma: discount factor'
• 'epsilon: exploration rate'

If epsilon is too high for too long, learning can stay noisy. If it drops too fast, the agent may stop exploring before it has discovered a good policy.

Common Pitfalls

A common mistake is implementing the Q-learning update rule by accident and then calling it SARSA. If you use max_a Q(s_next, a) instead of Q(s_next, a_next), you changed the algorithm.

Another mistake is not handling terminal states separately. In terminal states there is no next action value to bootstrap from.

A third issue is expecting tabular SARSA to scale to huge or continuous state spaces without function approximation. The tabular form is best for small discrete problems.

Summary

• SARSA is an on-policy temporal-difference control algorithm
• Its update uses the next action actually chosen by the current policy
• The core implementation needs a Q-table, an epsilon-greedy policy, and the SARSA update rule
• Terminal states should be handled without bootstrapping from a next action value
• SARSA is easiest to understand and debug in small discrete environments first