Introduction

Yes, you can estimate variable importance for a Keras model, but there is no single built-in score equivalent to the feature importance you get from a tree model. With neural networks, importance is usually measured indirectly through permutation tests, gradients, ablation, or explanation libraries such as SHAP.

Why Keras Does Not Expose One Importance Number

Neural networks learn distributed representations. A feature may matter because of its interaction with several other features rather than because it owns one clear split or coefficient.

That means "importance" depends on what you mean:

• importance for global model accuracy
• importance for one specific prediction
• importance under correlation between features
• importance measured as sensitivity or performance drop

Because those are different questions, Keras leaves interpretation to separate techniques instead of exposing one universal API.

Permutation Importance Is the Best Simple Baseline

Permutation importance is often the easiest place to start. The idea is:

1. measure baseline model performance
2. shuffle one feature column
3. measure how much performance drops

If shuffling a feature hurts the model a lot, that feature was important to the model.

1import numpy as np
2import tensorflow as tf
3
4
5def permutation_importance(model, x, y):
6    baseline = model.evaluate(x, y, verbose=0)[1]
7    scores = []
8
9    for column in range(x.shape[1]):
10        x_shuffled = x.copy()
11        np.random.shuffle(x_shuffled[:, column])
12        shuffled_score = model.evaluate(x_shuffled, y, verbose=0)[1]
13        scores.append(baseline - shuffled_score)
14
15    return scores
16
17
18x = np.random.rand(200, 3).astype("float32")
19y = ((x[:, 0] + 2 * x[:, 1]) > 1.2).astype("float32")
20
21model = tf.keras.Sequential([
22    tf.keras.layers.Input(shape=(3,)),
23    tf.keras.layers.Dense(8, activation="relu"),
24    tf.keras.layers.Dense(1, activation="sigmoid"),
25])
26
27model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
28model.fit(x, y, epochs=10, verbose=0)
29
30print(permutation_importance(model, x, y))

This method is model-agnostic and easy to explain to others.

Gradient-Based Importance

Another option is to look at gradients of the output with respect to the inputs. Large gradients mean the prediction is sensitive to changes in that feature near the current sample.

1import tensorflow as tf
2import numpy as np
3
4sample = tf.convert_to_tensor([[0.3, 0.8, 0.1]], dtype=tf.float32)
5
6with tf.GradientTape() as tape:
7    tape.watch(sample)
8    prediction = model(sample)
9
10grads = tape.gradient(prediction, sample)
11print(np.abs(grads.numpy()))

This is useful for local explanations, but it measures sensitivity around a point, not a global importance score for the whole dataset.

Ablation and Zero-Out Tests

Ablation is a simpler cousin of permutation importance. Instead of shuffling a feature, you remove or neutralize it and observe the effect.

For example:

• set one feature column to zero
• replace it with the mean value
• mask it in the preprocessing layer

This is easy to implement and often helpful when the features have a natural "missing" value.

SHAP and Other Explanation Libraries

If you need richer explanations, SHAP is a common choice. It can produce both global and per-sample importance estimates. The tradeoff is more complexity and compute cost.

For many practical workflows, the progression looks like this:

1. start with permutation importance
2. inspect a few local gradients or attribution maps
3. move to SHAP or integrated gradients only if you need deeper analysis

That keeps the interpretation work proportional to the problem.

Be Careful with Correlated Features

Feature importance in neural networks is especially tricky when inputs are correlated. If two columns carry almost the same information, permutation importance may underestimate each one individually because the model can still rely on the other.

That is not a bug in the method. It is a property of the data and the model. Always interpret importance scores together with domain knowledge.

Common Pitfalls

The biggest mistake is expecting a single canonical importance number from Keras itself. Neural networks usually require interpretation methods layered on top of the model.

Another mistake is reading gradient magnitude as a universal global importance score. Gradients are local and can change from one sample to another.

Developers also forget that permutation importance measures performance drop, so the metric you choose matters. Accuracy, AUC, and loss can produce different rankings.

Finally, correlated features can distort all importance methods. If two features substitute for one another, neither may look as important as it really is in isolation.

Summary

• Keras does not provide one built-in feature-importance number like tree models do.
• Permutation importance is often the simplest and most useful global method.
• Gradients help explain local sensitivity for individual predictions.
• Ablation tests and SHAP provide deeper interpretability options.
• Always interpret importance scores carefully when features are correlated.