Introduction

A Restricted Boltzmann Machine, or RBM, is a two-layer energy-based model with visible units and hidden units but no connections within the same layer. RBMs are not a standard modern default for deep learning, but they are still useful for understanding contrastive divergence, generative modeling, and the historical foundations of deep belief networks.

The Core RBM Structure

An RBM has:

• a visible layer representing observed data
• a hidden layer representing latent features
• weights connecting every visible unit to every hidden unit

For a binary-binary RBM, both layers are usually modeled with Bernoulli activations. The model learns weights so that training examples have lower energy than unlikely configurations.

In practice, training usually uses contrastive divergence rather than exact likelihood gradients, because computing the exact partition function is too expensive.

A Minimal TensorFlow 2 RBM

The following example shows a small binary RBM trained with one-step contrastive divergence, often written as CD-1.

1import tensorflow as tf
2
3
4class RBM(tf.Module):
5    def __init__(self, n_visible, n_hidden):
6        self.W = tf.Variable(tf.random.normal([n_visible, n_hidden], stddev=0.01))
7        self.v_bias = tf.Variable(tf.zeros([n_visible]))
8        self.h_bias = tf.Variable(tf.zeros([n_hidden]))
9
10    def sample_prob(self, probs):
11        return tf.cast(tf.random.uniform(tf.shape(probs)) < probs, tf.float32)
12
13    def hidden_probs(self, v):
14        return tf.sigmoid(tf.matmul(v, self.W) + self.h_bias)
15
16    def visible_probs(self, h):
17        return tf.sigmoid(tf.matmul(h, tf.transpose(self.W)) + self.v_bias)
18
19    def contrastive_divergence(self, v0, learning_rate=0.1):
20        h0_prob = self.hidden_probs(v0)
21        h0_sample = self.sample_prob(h0_prob)
22
23        v1_prob = self.visible_probs(h0_sample)
24        v1_sample = self.sample_prob(v1_prob)
25
26        h1_prob = self.hidden_probs(v1_sample)
27
28        positive_grad = tf.matmul(tf.transpose(v0), h0_prob)
29        negative_grad = tf.matmul(tf.transpose(v1_sample), h1_prob)
30
31        batch_size = tf.cast(tf.shape(v0)[0], tf.float32)
32
33        self.W.assign_add(learning_rate * (positive_grad - negative_grad) / batch_size)
34        self.v_bias.assign_add(learning_rate * tf.reduce_mean(v0 - v1_sample, axis=0))
35        self.h_bias.assign_add(learning_rate * tf.reduce_mean(h0_prob - h1_prob, axis=0))
36
37
38data = tf.constant([
39    [1., 1., 1., 0., 0., 0.],
40    [1., 0., 1., 0., 0., 0.],
41    [0., 0., 0., 1., 1., 1.],
42    [0., 0., 0., 1., 0., 1.],
43])
44
45rbm = RBM(n_visible=6, n_hidden=2)
46
47for epoch in range(500):
48    rbm.contrastive_divergence(data, learning_rate=0.1)
49
50print(tf.sigmoid(tf.matmul(data, rbm.W) + rbm.h_bias).numpy())

This is intentionally compact. It is enough to demonstrate the learning mechanics without burying the core idea under framework scaffolding.

What Contrastive Divergence Is Doing

CD-1 approximates the gradient by comparing two phases:

• positive phase: the model sees real data
• negative phase: the model reconstructs from sampled hidden activations

The update moves the weights toward patterns that explain the real data and away from patterns generated by the reconstruction sample.

It is only an approximation, but it is the standard practical training rule in introductory RBM implementations.

Data Requirements

The simple code above assumes binary inputs. If your data is not naturally binary, you typically:

• binarize it
• scale it into probabilities and sample
• or switch to a variant such as a Gaussian-Bernoulli RBM

That assumption matters. A binary RBM trained directly on arbitrary continuous features without the right modeling choices can behave poorly even if the code runs.

Why TensorFlow Is Useful Here

TensorFlow helps mostly with vectorized matrix operations. An RBM is mathematically simple but very linear-algebra-heavy, so expressing the positive and negative phases as batch matrix multiplies keeps the implementation compact and fast enough for experimentation.

Unlike a typical feed-forward network, you do not usually train an RBM with a standard Keras fit loop and a predefined loss function. The learning rule is more custom, which is why a lower-level TensorFlow style is often clearer.

Common Pitfalls

The biggest pitfall is treating an RBM like a standard supervised neural network. RBMs are energy-based generative models with a very different training procedure.

Another common mistake is forgetting the binary assumption in simple Bernoulli RBM examples. If the data type does not match the model family, the training signal can become misleading.

Developers also often expect reconstructions to look perfect after a few updates. RBMs are approximate models, and CD-1 is a rough training method, so intuition should focus on learned structure rather than on exact reconstruction quality alone.

Summary

• An RBM has visible and hidden units with no intra-layer connections.
• TensorFlow is a good fit for RBMs because the training rule is mostly batch linear algebra.
• A common training method is one-step contrastive divergence, or CD-1.
• Simple introductory RBMs usually assume binary visible and hidden units.
• RBMs are best understood as custom energy-based models, not as ordinary feed-forward classifiers.