Introduction

If SSH keeps asking for a password or passphrase after you have set up key-based authentication, the problem is almost always one of four things: the SSH agent does not have your key loaded, the file permissions on the remote server are too open, the server's sshd_config does not allow public key authentication, or you are connecting with the wrong key. The fix usually takes less than five minutes once you identify which of these is the cause.

Quick Diagnostic: Use Verbose Mode

The single most useful troubleshooting step is connecting with verbose output. This shows exactly what happens during the authentication handshake:

ssh -vvv [email protected]

Look for these lines in the output:

debug1: Offering public key: /home/user/.ssh/id_ed25519
debug1: Server accepts key: /home/user/.ssh/id_ed25519
debug1: Authentication succeeded (publickey).

If you instead see:

debug1: Trying private key: /home/user/.ssh/id_rsa
debug3: no such identity: /home/user/.ssh/id_rsa: No such file or directory
debug1: Next authentication method: password

That tells you the client could not find or offer the correct key and fell back to password authentication.

Fix 1: Add Your Key to the SSH Agent

The SSH agent holds your private keys in memory so you do not have to enter the passphrase every time. If the key is not loaded, SSH cannot offer it to the server.

1# Check which keys are currently loaded
2ssh-add -l
3
4# If it says "The agent has no identities", add your key
5ssh-add ~/.ssh/id_ed25519
6
7# On macOS, add to Keychain so it persists across reboots
8ssh-add --apple-use-keychain ~/.ssh/id_ed25519

If ssh-add -l returns "Could not open a connection to your authentication agent," the agent is not running:

1# Start the agent
2eval "$(ssh-agent -s)"
3
4# Then add the key
5ssh-add ~/.ssh/id_ed25519

To make the agent start automatically on login, add the eval line to your ~/.bashrc, ~/.zshrc, or equivalent shell config file.

Fix 2: Fix File Permissions

SSH is strict about file permissions. If any SSH-related file or directory has permissions that are too open, SSH silently refuses to use the key and falls back to password authentication. No error is shown on the client side, which makes this a common source of confusion.

On the remote server:

1# Fix the home directory (must not be writable by group/other)
2chmod 755 ~
3# or chmod 700 ~ for stricter
4
5# Fix the .ssh directory
6chmod 700 ~/.ssh
7
8# Fix authorized_keys
9chmod 600 ~/.ssh/authorized_keys

On the local machine:

1# Private key must be readable only by you
2chmod 600 ~/.ssh/id_ed25519
3
4# Public key can be more open (but 644 is typical)
5chmod 644 ~/.ssh/id_ed25519.pub
6
7# .ssh directory
8chmod 700 ~/.ssh

Here is the complete permissions table:

A common mistake: using chmod 777 on the home directory for some other reason. SSH sees that and refuses to trust authorized_keys in that directory.

Fix 3: Verify Server Configuration

The SSH server must be configured to accept public key authentication. Check /etc/ssh/sshd_config on the remote server:

# Check the relevant settings
grep -E "^(PubkeyAuthentication|AuthorizedKeysFile|PasswordAuthentication)" /etc/ssh/sshd_config

The expected values:

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

If PubkeyAuthentication is set to no, change it to yes and restart the SSH daemon:

1# Edit the config
2sudo nano /etc/ssh/sshd_config
3
4# Restart sshd (systemd)
5sudo systemctl restart sshd
6
7# Restart sshd (older init systems)
8sudo service ssh restart

Once key authentication is working, you can optionally disable password authentication entirely for better security:

PasswordAuthentication no

Fix 4: Ensure the Correct Key Is Being Used

If you have multiple SSH keys, SSH may offer the wrong one. Use ~/.ssh/config to map specific keys to specific hosts:

1Host github.com
2    HostName github.com
3    User git
4    IdentityFile ~/.ssh/github_ed25519
5    IdentitiesOnly yes
6
7Host production
8    HostName prod.example.com
9    User deploy
10    IdentityFile ~/.ssh/deploy_ed25519
11    IdentitiesOnly yes
12
13Host staging
14    HostName staging.example.com
15    User deploy
16    IdentityFile ~/.ssh/staging_ed25519
17    IdentitiesOnly yes

The IdentitiesOnly yes directive is important. Without it, SSH tries all keys loaded in the agent before using the specified IdentityFile. If the agent has many keys, the server may reject the connection after too many failed attempts (typically 3 or 6).

Fix 5: Verify authorized_keys on the Server

Make sure your public key is correctly added to the remote server's ~/.ssh/authorized_keys file:

# On your local machine, show your public key
cat ~/.ssh/id_ed25519.pub

The output looks like:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG... user@local-machine

This entire line (including the key type and comment) must be a single line in ~/.ssh/authorized_keys on the server. Common mistakes:

• The key was split across multiple lines during copy-paste.
• The key was added to the wrong user's authorized_keys.
• There are trailing whitespace or invisible characters.

The simplest way to copy a key to a server is:

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

This command handles creating the .ssh directory, setting permissions, and appending the key correctly.

Passphrase vs Password

These are two different things, and the prompts look similar but mean different things:

If you see the passphrase prompt, key authentication is working correctly. The passphrase decrypts your local private key. To avoid typing it every time, load the key into the SSH agent (see Fix 1).

If you see the password prompt, key authentication failed entirely. Check permissions, server config, and whether the correct key was offered (Fixes 2-5).

SELinux and Security Contexts

On systems running SELinux (RHEL, CentOS, Fedora), correct file permissions alone may not be enough. SELinux security contexts must also be correct:

1# Check SELinux context
2ls -Z ~/.ssh/authorized_keys
3
4# Restore correct context
5restorecon -Rv ~/.ssh

If the authorized_keys file was created or copied in an unusual way, its SELinux context may be wrong, causing SSH to silently reject it even though UNIX permissions are correct.

Common Pitfalls

Not starting the SSH agent. On many systems, the SSH agent does not start automatically. If ssh-add -l returns "Could not open a connection to your authentication agent," the agent is not running. Start it with eval "$(ssh-agent -s)".

Using chmod 777 or chmod 775 on the home directory. SSH requires that the home directory, .ssh directory, and authorized_keys file are not writable by group or other users. This is a security requirement, not a bug.

Offering too many keys from the agent. If you have 10+ keys loaded in the SSH agent and the server's MaxAuthTries is set to 6 (the default), SSH exhausts its attempts before reaching the correct key. Use IdentitiesOnly yes in your SSH config to send only the specified key.

Copying the private key instead of the public key to authorized_keys. The authorized_keys file should contain the .pub file content (starting with ssh-ed25519 or ssh-rsa), not the private key (which starts with -----BEGIN OPENSSH PRIVATE KEY-----).

Not restarting sshd after config changes. Changes to /etc/ssh/sshd_config do not take effect until the SSH daemon is restarted or reloaded. Use sudo systemctl restart sshd.

Using RSA keys that are too short. OpenSSH 8.8+ disables RSA-SHA1 signatures by default. If you are using an older RSA key (typically 1024-bit), generate a new Ed25519 key instead: ssh-keygen -t ed25519 -C "[email protected]".

Summary

• Use ssh -vvv to see exactly where authentication fails.
• Load your key into the SSH agent with ssh-add to avoid passphrase prompts on every connection.
• Fix permissions: 700 for ~/.ssh, 600 for authorized_keys and private keys, and 755 or stricter for the home directory.
• Verify PubkeyAuthentication yes in the server's /etc/ssh/sshd_config.
• Use ~/.ssh/config with IdentityFile and IdentitiesOnly yes to control which key is offered to which host.
• Distinguish between the passphrase prompt (local key decryption, working as intended) and the password prompt (key auth failed, needs debugging).
• Use ssh-copy-id to correctly install your public key on remote servers.