Introduction

Feedback alignment is a learning rule that replaces the exact backpropagated transpose with fixed random feedback matrices. It is mostly useful for research and experimentation, not as a drop-in replacement for ordinary model.fit. In TensorFlow, the clean way to implement it is a custom training loop where you compute the hidden-layer update explicitly.

What Changes Compared With Backpropagation

In standard backpropagation, the error signal for a hidden layer uses the transpose of the next forward weight matrix. In feedback alignment, that backward path is replaced with a fixed random matrix B that is never trained.

For a two-layer network:

• the forward pass still uses trainable W1 and W2
• the output layer gradient is computed normally
• the hidden layer receives its error signal through B, not W2 transposed

That means automatic differentiation does not “become” feedback alignment automatically. You have to write the custom update rule yourself.

Set Up the Variables Explicitly

A simple TensorFlow implementation starts with raw tf.Variable objects.

1import tensorflow as tf
2
3tf.random.set_seed(7)
4
5input_dim = 2
6hidden_dim = 8
7output_dim = 2
8learning_rate = 0.05
9
10W1 = tf.Variable(tf.random.normal((input_dim, hidden_dim), stddev=0.2))
11b1 = tf.Variable(tf.zeros((hidden_dim,)))
12W2 = tf.Variable(tf.random.normal((hidden_dim, output_dim), stddev=0.2))
13b2 = tf.Variable(tf.zeros((output_dim,)))
14
15B = tf.constant(tf.random.normal((output_dim, hidden_dim), stddev=0.2))

The matrix B is the fixed feedback matrix. Its shape matters: it maps output-layer error back into hidden-layer units.

Write a Custom Training Loop

The core of the method is the manual hidden-layer update.

1import tensorflow as tf
2
3x = tf.constant([
4    [0.0, 0.0],
5    [0.0, 1.0],
6    [1.0, 0.0],
7    [1.0, 1.0],
8], dtype=tf.float32)
9
10y = tf.constant([0, 1, 1, 0], dtype=tf.int32)
11y_one_hot = tf.one_hot(y, depth=2)
12
13for step in range(1000):
14    hidden_pre = tf.matmul(x, W1) + b1
15    hidden = tf.nn.relu(hidden_pre)
16    logits = tf.matmul(hidden, W2) + b2
17
18    loss = tf.reduce_mean(
19        tf.nn.softmax_cross_entropy_with_logits(labels=y_one_hot, logits=logits)
20    )
21
22    probs = tf.nn.softmax(logits)
23    batch_size = tf.cast(tf.shape(x)[0], tf.float32)
24    d_logits = (probs - y_one_hot) / batch_size
25
26    dW2 = tf.matmul(hidden, d_logits, transpose_a=True)
27    db2 = tf.reduce_sum(d_logits, axis=0)
28
29    relu_grad = tf.cast(hidden_pre > 0.0, tf.float32)
30    hidden_signal = tf.matmul(d_logits, B) * relu_grad
31    dW1 = tf.matmul(x, hidden_signal, transpose_a=True)
32    db1 = tf.reduce_sum(hidden_signal, axis=0)
33
34    W1.assign_sub(learning_rate * dW1)
35    b1.assign_sub(learning_rate * db1)
36    W2.assign_sub(learning_rate * dW2)
37    b2.assign_sub(learning_rate * db2)
38
39    if step % 200 == 0:
40        print(f"step={step} loss={loss.numpy():.4f}")

The key line is hidden_signal = tf.matmul(d_logits, B) * relu_grad. In ordinary backpropagation you would use the transpose of W2 there instead.

Evaluate the Learned Model

Once training is done, evaluate the network normally.

1hidden = tf.nn.relu(tf.matmul(x, W1) + b1)
2logits = tf.matmul(hidden, W2) + b2
3predictions = tf.argmax(logits, axis=1)
4
5print("Predictions:", predictions.numpy())
6print("Targets:", y.numpy())

For toy problems, feedback alignment can often learn a useful mapping. But it is usually less stable and less efficient than standard backpropagation.

Why a Custom Loop Matters

TensorFlow’s high-level training APIs assume standard gradients. Feedback alignment is a different credit-assignment rule, so you need a training loop where you control the update directly. That is the real implementation boundary.

This is also why feedback alignment is mostly used for experiments and papers rather than mainstream training pipelines. The implementation is not hard, but it is a deliberate departure from the default optimization machinery.

Common Pitfalls

• Expecting automatic differentiation to produce feedback alignment automatically.
• Giving the fixed matrix B the wrong shape.
• Forgetting activation derivatives, such as the ReLU mask, in the hidden update.
• Comparing one noisy feedback-alignment run against a polished backprop baseline and drawing strong conclusions too quickly.
• Treating feedback alignment as a production default rather than a research-oriented learning rule.

Summary

• Feedback alignment replaces exact transposed feedback with a fixed random matrix.
• In TensorFlow, the clean implementation is a custom training loop with manual updates.
• The output layer still uses an ordinary gradient; the hidden-layer signal is the approximation.
• Matrix shapes and activation derivatives still matter just as much as in standard training.
• Use feedback alignment as an experiment or teaching tool, not as a default optimizer path.