How to asynchronously run Matplolib server-side with a timeout? The process hangs randomly

Python

Matplotlib

Asynchronous Programming

Server-side Rendering

Timeout Handling

How to asynchronously run Matplolib server-side with a timeout? The process hangs randomly

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

Matplotlib hangs on servers because its default backend (TkAgg, Qt5Agg) tries to connect to a display server that does not exist. The fix is to use the non-interactive Agg backend (matplotlib.use('Agg')) before importing pyplot, and run plot generation in a separate process with a timeout using concurrent.futures.ProcessPoolExecutor. Thread-based approaches fail because matplotlib is not thread-safe — the Agg backend uses global state that causes deadlocks under concurrent access.

The Problem

python

1# This hangs on a headless server
2import matplotlib.pyplot as plt
3
4plt.figure()
5plt.plot([1, 2, 3])
6plt.savefig("plot.png")  # Hangs or segfaults

The default backend attempts to create a GUI window. On a server without a display ($DISPLAY not set), this blocks indefinitely or crashes.

Fix 1: Use the Agg Backend

python

1# MUST be called before importing pyplot
2import matplotlib
3matplotlib.use('Agg')  # Non-interactive backend — no display required
4import matplotlib.pyplot as plt
5
6def create_plot(data, output_path):
7    fig, ax = plt.subplots()
8    ax.plot(data)
9    ax.set_title("Server-side Plot")
10    fig.savefig(output_path, dpi=100, bbox_inches='tight')
11    plt.close(fig)  # Free memory
12    return output_path
13
14create_plot([1, 2, 3, 4, 5], "/tmp/plot.png")

matplotlib.use('Agg') must be called before any matplotlib.pyplot import. The Agg backend renders to a file or buffer without requiring a display.

Fix 2: Process-Based Execution with Timeout

python

1import matplotlib
2matplotlib.use('Agg')
3import matplotlib.pyplot as plt
4from concurrent.futures import ProcessPoolExecutor, TimeoutError
5import io
6
7def generate_plot(data):
8    """Run in a separate process to isolate matplotlib state."""
9    fig, ax = plt.subplots()
10    ax.plot(data['x'], data['y'])
11    ax.set_title(data.get('title', 'Plot'))
12
13    buf = io.BytesIO()
14    fig.savefig(buf, format='png', dpi=100)
15    plt.close(fig)
16    buf.seek(0)
17    return buf.getvalue()
18
19def create_plot_with_timeout(data, timeout=10):
20    """Generate a plot with a timeout to prevent hanging."""
21    with ProcessPoolExecutor(max_workers=1) as executor:
22        future = executor.submit(generate_plot, data)
23        try:
24            result = future.result(timeout=timeout)
25            return result
26        except TimeoutError:
27            future.cancel()
28            raise RuntimeError(f"Plot generation timed out after {timeout}s")
29
30# Usage
31data = {'x': [1, 2, 3, 4], 'y': [10, 20, 15, 25], 'title': 'Sales'}
32png_bytes = create_plot_with_timeout(data, timeout=5)
33
34with open("output.png", "wb") as f:
35    f.write(png_bytes)

ProcessPoolExecutor creates a separate process with its own matplotlib state. If the process hangs, future.result(timeout=N) raises TimeoutError instead of blocking forever.

Fix 3: Flask/Django Web Server Integration

python

1import matplotlib
2matplotlib.use('Agg')
3import matplotlib.pyplot as plt
4import io
5from flask import Flask, send_file
6
7app = Flask(__name__)
8
9@app.route('/plot')
10def plot():
11    fig, ax = plt.subplots(figsize=(8, 6))
12    ax.plot([1, 2, 3, 4], [10, 20, 25, 30])
13    ax.set_title("Dynamic Plot")
14
15    buf = io.BytesIO()
16    fig.savefig(buf, format='png', dpi=100)
17    plt.close(fig)  # Critical: prevents memory leaks
18    buf.seek(0)
19
20    return send_file(buf, mimetype='image/png')
21
22if __name__ == '__main__':
23    app.run()

Always call plt.close(fig) after saving to free memory. Without it, each request leaks a figure object, eventually exhausting server memory.

Fix 4: Thread Safety with Locks

python

1import matplotlib
2matplotlib.use('Agg')
3import matplotlib.pyplot as plt
4import threading
5import io
6
7# Matplotlib is NOT thread-safe — use a lock
8_plot_lock = threading.Lock()
9
10def thread_safe_plot(data, output_path):
11    with _plot_lock:
12        fig, ax = plt.subplots()
13        ax.bar(range(len(data)), data)
14        fig.savefig(output_path, dpi=100)
15        plt.close(fig)
16
17# Or better: use the object-oriented API without pyplot
18from matplotlib.figure import Figure
19from matplotlib.backends.backend_agg import FigureCanvasAgg
20
21def truly_thread_safe_plot(data):
22    """No global state — safe for concurrent threads."""
23    fig = Figure(figsize=(8, 6))
24    canvas = FigureCanvasAgg(fig)
25    ax = fig.add_subplot(111)
26    ax.plot(data)
27
28    buf = io.BytesIO()
29    fig.savefig(buf, format='png')
30    buf.seek(0)
31    return buf.getvalue()

The object-oriented API (Figure + FigureCanvasAgg) avoids pyplot's global state, making it safe for concurrent use without locks.

Fix 5: Environment Variable Approach

bash

1# Set the backend via environment variable (no code changes needed)
2export MPLBACKEND=Agg
3
4# Or in a Dockerfile
5ENV MPLBACKEND=Agg
6
7# Or in a systemd service file
8Environment=MPLBACKEND=Agg

python

# matplotlibrc file (project-level or user-level)
# ~/.config/matplotlib/matplotlibrc
backend: Agg

Setting MPLBACKEND=Agg as an environment variable applies the backend globally without modifying code.

Common Pitfalls

Importing pyplot before setting the backend: import matplotlib.pyplot as plt triggers backend initialization. matplotlib.use('Agg') must come before any pyplot import, otherwise it raises a warning and has no effect.
Not closing figures (plt.close()): Each plt.figure() or plt.subplots() allocates memory. Without plt.close(fig), figures accumulate in memory, causing OOM crashes on long-running servers.
Using threads with pyplot: pyplot maintains global state (current figure, current axes) that is not thread-safe. Concurrent threads corrupt each other's plots. Use the OO API (Figure + FigureCanvasAgg) or a process pool instead.
Font cache building on first run: The first matplotlib import on a new server triggers font cache building, which can take 30+ seconds and look like a hang. Pre-warm the cache during deployment with python -c "import matplotlib.pyplot".
Missing system fonts in Docker: Minimal Docker images may lack fonts, causing matplotlib to fall back to a slow font search. Install fontconfig and basic fonts (apt-get install fonts-dejavu-core).

Summary

Use matplotlib.use('Agg') before importing pyplot — or set MPLBACKEND=Agg environment variable
Run plot generation in ProcessPoolExecutor with a timeout to prevent indefinite hangs
Always call plt.close(fig) after saving to prevent memory leaks
Use the object-oriented API (Figure + FigureCanvasAgg) for thread-safe rendering
Pre-warm the font cache during deployment to avoid first-run delays
Install fonts-dejavu-core in Docker images for reliable font rendering