Windows SSH Server issues
In the process of automating various administrative tasks on the servers I’m working on, I decided to install the OpenSSH server on some of the Windows machines. The installation itself is rather simple and is described in the “Install OpenSSH” document on the Windows docs site. The issues started later 🙂
Running
By default, the OpenSSH server installed on Windows is installed as a service but will not start automatically. It is set to a Manual start. It is described in the installation document I linked above, but it is easy to overlook. So, in order to make it start automatically, you can go to Services and switch the Startup Type to “Automatic” or “Automatic (Delayed Start)”. You should also remember about firewall rules – you should allow connection to your new SSH server. You can also do this directly in PowerShell:
# Start the sshd service Start-Service sshd # OPTIONAL but recommended: Set-Service -Name sshd -StartupType 'Automatic' # Confirm the firewall rule is configured. It should be created automatically by setup. Get-NetFirewallRule -Name *ssh* # There should be a firewall rule named "OpenSSH-Server-In-TCP", which should be enabled # If the firewall does not exist, create one New-NetFirewallRule -Name sshd -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
Logging
By default, OpenSSH installed on Windows is not saving its own log files. It is using ETW logging capabilities. I found it rather hard to use, so I decided to switch from ETW to old, good file-based logs. In order to do this, I adjusted the SSH config file. In Windows it is stored in the %programdata%\ssh\sshd_config (typically: C:\ProgramData\ssh\sshd_config). The thing I changed was the line with the SyslogFacility setting. I had to uncomment it and set it to LOCAL0 (there is zero at the end):
SyslogFacility LOCAL0
If you require more detailed logs, you can also uncomment and adjust the next line to read:
LogLevel DEBUG
Why? Because I had some issues with logging and I wanted to check what is going on. The log files are stored in the %programdata%\ssh\logs (typically: C:\ProgramData\ssh\logs) and are following the standard SSH log format.
Signing in
I found two issues when signing in. The first was related to the number of keys stored in my .ssh directory. SSH was trying to use them one by one, which exceeded the limit of login attempts. Initially, I wanted to use a password, to simply test the connection, so I executed SSH with the following parameters to make it skip keys and go directly to the password:
ssh -o PasswordAuthentication=yes -o PreferredAuthentications=keyboard-interactive,password -o PubkeyAuthentication=no [email protected]
The second was related to the users present on the server. There are two types of users we are using: local Windows users (for some automation, for basic authentication purposes, and so on) and domain users (managed by the Active Directory users management). It is not obvious how to SSH as the domain user. Here is the method for the local user and for the domain-based user:
# for the local user, you can simply use: ssh [email protected] # for the domain-based user it is a little bit tricky: ssh user@[email protected] # if you are using older Windows, you may also try: ssh 'DOMAIN\user'@server.domain.com
As you can see, there are two @ signs in the command, which looks strange but works 🙂
PS: for older Windows I noticed that I have to use a different method – showed in the second example.
Using keys instead of passwords
One more place where things are getting tricky is when you try to use keys instead of passwords (Microsoft manual is here). I wanted to try, so I created the %userProfileDir%\.ssh\authorized_keys file and I placed my public key in there. Unfortunately, it was not working. This is why I decided to switch SSH logs to DEBUG level and to see what is happening there. Here is what I found in logs:
9776 2021-05-10 05:12:43.007 debug1: userauth-request for user userName service ssh-connection method none [preauth] 9776 2021-05-10 05:12:43.007 debug1: attempt 0 failures 0 [preauth] 9776 2021-05-10 05:12:43.007 debug1: user does not match group list administrators at line 84 9776 2021-05-10 05:12:43.132 debug1: userauth-request for user userName service ssh-connection method publickey [preauth] 9776 2021-05-10 05:12:43.132 debug1: attempt 1 failures 0 [preauth] 9776 2021-05-10 05:12:43.132 debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:wcCQGvc32432354dtDcjDs/QqJdMgvkMzasYnOZ5k [preauth] 9776 2021-05-10 05:12:43.132 debug1: trying public key file C:\\Users\\userName\\.ssh\\authorized_keys 9776 2021-05-10 05:12:43.148 Authentication refused.
Obviously, it was trying to get the public key from the proper file, but for some reason, the authentication was refused. The “Authentication refused” message was not really helpful – there is no information why it was refused. I was not sure if the key was copied improperly, or maybe there is some other kind of issue. What I remember from Linux is that the file has to have proper permissions. This led me to the conclusion that there can be something wrong with the ability of the SSH server to read the file.
I noticed that the public keys are stored in different places for regular users and different for members of Administrators group. This can also be important to you – the keys for Administrators are (by default) stored in %programData%\ssh\administrators_authorized_keys. This can be changed in the SSH config file, by commenting out the lines that define this particular behavior. In my case, however, it was not the wrong file location issue. I had my file in the proper location, filled with the proper data. So, permissions?
I found that for the Administrators related autorized_keys file, the documentation states one more step than for the local users. They are using additional command to set ACL (Access Control List) parameters to the file. It is allowing full access for the Administrators and System accounts to the file provided in the command. In my instance I had to run CMD as Administrator and execute the following command:
icacls.exe "C:\Users\userName\.ssh\authorized_keys" /inheritance:r /grant "Administrators:F" /grant "SYSTEM:F"
Once the extended permissions were granted, I was able to sign in to SSH using keys.