Introduction

Ceph debug output is controlled per subsystem, so changing the “dout level” really means changing one or more debug_* settings such as debug_osd, debug_mon, or debug_ms. The right setting depends on which daemon you are investigating and whether you want a temporary runtime change or a persistent configuration change.

Understand the debug_* Format

Many Ceph debug settings are written in a two-number form such as 1/5 or 20/20.

In practice, operators use that pair to control how much detail goes to normal logs versus the in-memory debug buffer. Higher numbers produce more detail, but they also increase log volume and can affect performance.

Examples of common subsystems include:

• 'debug_osd for OSD behavior'
• 'debug_mon for monitor behavior'
• 'debug_mgr for manager daemons'
• 'debug_ms for messenger and network-related issues'

If you are debugging a specific OSD problem, increasing debug_osd is usually more useful than turning everything up globally.

Temporary Runtime Changes

For short-lived troubleshooting, it is better to change the level on the running daemon and then turn it back down after collecting evidence.

A common pattern is:

ceph tell osd.3 injectargs '--debug_osd 10/10'
ceph tell mon.a injectargs '--debug_mon 5/5'

That adjusts the daemon without editing persistent config first.

If your release supports centralized config commands, you may also see or use:

ceph config set osd.3 debug_osd 10/10
ceph config get osd.3 debug_osd

The first style is often used for immediate runtime debugging, while the second is useful when you want Ceph’s config database to hold the value explicitly.

Persistent Configuration

If you need the setting to survive daemon restart, apply it persistently rather than only through a temporary runtime command.

On releases that use the config database heavily, a persistent change often looks like this:

ceph config set osd debug_osd 1/5
ceph config set mon debug_mon 1/5

That targets all OSDs or all monitors.

In older deployments, you may still see the values placed in ceph.conf:

1[osd]
2debug osd = 1/5
3
4[mon]
5debug mon = 1/5

The exact management path depends on the Ceph release and how the cluster is operated, but the subsystem idea stays the same.

Choose the Smallest Useful Scope

It is tempting to raise every debug level everywhere, especially during an outage. That usually creates too much noise.

A better workflow is:

1. Identify the failing daemon type.
2. Increase only the relevant debug_* key.
3. Reproduce the issue or wait for it to recur.
4. Capture the logs.
5. Restore the lower level.

Example for a single OSD:

ceph tell osd.12 injectargs '--debug_osd 20/20'
journalctl -u ceph-osd@12 -f
ceph tell osd.12 injectargs '--debug_osd 1/5'

That approach gives you focused data without drowning the rest of the cluster in verbose logs.

Check the Result

After changing the setting, verify both the config value and the resulting logs. Depending on the host setup, logs may be visible through journalctl, container logs, or Ceph log files.

A quick validation pattern is:

ceph config get osd.12 debug_osd
ceph -s

The first command confirms the value. The second makes sure the cluster is still healthy enough while you investigate.

Common Pitfalls

The biggest mistake is turning on extremely verbose logging cluster-wide and forgetting to turn it back down. Ceph can generate a large amount of output, which makes both performance and incident analysis worse.

Another common issue is changing the wrong subsystem. Raising debug_mon will not help much with an OSD data-path problem, and raising debug_osd will not explain a monitor election issue.

Operators also sometimes assume a single integer is always enough. In Ceph, many debug keys use a paired N/M style, so copying a one-number example blindly can cause confusion.

Finally, persistent changes and temporary runtime changes are not the same. If the value disappears after restart, it was probably only injected into the live daemon.

Summary

• Ceph debug output is controlled per subsystem through keys such as debug_osd and debug_mon.
• Higher values create more detail but also more overhead and more noise.
• Use runtime changes for short investigations and persistent config for repeatable settings.
• Scope the change to the smallest relevant daemon or daemon class.
• Always verify the value and reset verbose logging after the investigation ends.