OpenSSH released version 10.6p2 yesterday (5 November) with what its own release notes call a critical security workaround for Linux systems that use the musl C library with PAM turned off. In plain terms: on the wrong combination of settings, an account that should never accept a password will accept any password longer than 256 characters. On some setups that gets you root.
If you run Ubuntu, Debian or Red Hat, you can relax. They use glibc, not musl, and they aren't affected. If anything you look after runs Alpine Linux, or a small embedded build, read on. It's a quick fix, and the way the bug happened is worth five minutes even if you never touch Alpine.
What actually goes wrong
The researcher's write-up at sshhole.com walks through it, and it's a lovely, horrible chain of small decisions that each made sense on their own.
- On Unix systems, a password field of * means "no password will ever match this". It isn't a hash, so nothing can hash to it. Alpine uses it for root, and it's common on key-only deploy users.
- When the C function crypt() fails, the POSIX standard says it returns a null pointer. Back in 2012, musl changed its version to return a failure string instead, which is usually *.
- musl's SHA-256 and SHA-512 crypt code gives up on passwords over 256 characters. Giving up means failing, and failing means returning *.
- Since 2016, sshd has hashed passwords for accounts it can't really check using another account's salt, so that attackers can't tell real usernames from fake ones by timing.
Put those together. Someone tries to log in as root with a 300-character password. sshd hashes it with a salt borrowed from a real account, musl's crypt fails because the password is too long and hands back *, and sshd compares that with the stored *. They match. You're in.
The OpenSSH team's view is blunt: "We consider the root cause of this bug to be outside sshd". They've fixed it in sshd anyway, because that's where it can be exploited.
Who needs to act
All of these have to be true for a machine to be exposed:
- It uses musl libc (Alpine, plus embedded systems such as OpenWrt, Buildroot and Yocto builds, according to the write-up).
- sshd was built and runs without PAM.
- Password authentication is switched on in sshd.
- At least one account has * as its password field, and at least one other account has a real password.
The write-up says the Alpine images on AWS and Azure give the default alpine user passwordless doas, which is why getting into the wrong account can quickly become root. Whether a given box is exposed depends on how it's configured, so check rather than assume.
On containers, a bit of perspective. Lots of apps ship in Alpine-based Docker images, but most of those images don't run sshd at all, so there's nothing to log into. The ones to worry about are images somebody built with SSH inside for convenience, and small Alpine VMs used as bastions, jump boxes or cheap utility servers. Those tend to be the forgotten ones.
The Register reports no sign of the bug being exploited so far. Now the details are public, I wouldn't count on that lasting.
What to do
On Alpine, update OpenSSH to 10.6p2 or later with apk upgrade openssh and restart sshd. If you can't update straight away, turn password logins off. The write-up suggests dropping PasswordAuthentication no into a file in /etc/ssh/sshd_config.d/ and restarting sshd.
Honestly, that second step is worth doing on every server whether you're affected or not. If everyone logs in with keys, password bugs like this one simply don't apply to you. It's the first thing I'd look at in a server health check, along with who holds which keys. If you're not sure who has keys to your servers at all, my guide on getting back into a server after a developer leaves covers sorting that out.
For containers, rebuild any image that runs sshd from a fixed base and redeploy it. Patching inside a running container doesn't survive the next deploy. Better still, ask whether it needs SSH in it at all.
The bit that matters for your own code
Here's why I think this is worth more than a patch notice. None of the pieces is obviously wrong. musl returning * on failure was a deliberate choice, arguably a safer one than returning null to programs that don't check. sshd's timing protection is good practice. The bug lives in the gap between them: an error value that happens to look exactly like a valid stored value.
Bespoke software is full of gaps like that. A lookup that returns 0 on failure, where 0 is also a real user ID. An API wrapper that returns an empty list on a timeout, so the code carries on as if there were simply no results. A permission check that treats "couldn't load the role" the same as "no restrictions". Each one works fine in testing, because in testing nothing fails.
The researcher found this one by building sshd several different ways and running around 17,600 logins against each build, then looking for the one that disagreed. That's a good habit to borrow. Make the failure paths happen on purpose and check your code fails closed.
We look for exactly this kind of thing in a code audit: £250 + VAT, covering the code, the server and the database, with a written report. It's especially worth it if the app was written quickly, or with a lot of AI help, and nobody has asked what happens when things go wrong. We wrote about a similar problem with older PHP apps after PHP2Shell earlier this week.
And if you'd rather someone just kept on top of patches like this for you, that's what Look after it is for. It's from £355 a month + VAT and covers security updates, hosting and backups for software we look after.
If you're not sure whether anything of yours runs Alpine or has password SSH switched on, drop me a line and we'll help you find out.
Cheers, Rob
