Diagnosis of GSI-OpenSSH Server Issues

  1. Host key permissions failure
  2. Unable to map implicit username
  3. Entry in grid-mapfile refers to non-existent username
  4. Client proxy uninitialized or non-GSI agent
  5. Account is locked

Verifying your GSI configuration

As a first step, please run the following command to verify your GSI configuration:

sh$ grid-proxy-init -debug -verify -cert /etc/grid-security/hostcert.pem -key /etc/grid-security/hostkey.pem

If you do not see 'Proxy Verify OK' in the output, there is a problem with your GSI configuration.

Running the GSI-OpenSSH SSH server in debugging mode

The following information can be used to diagnose some common problems that you may be experiencing with the GSI-OpenSSH server. Each snippet of the server's output was taken from running the server with the '-ddd' flag (which increases the debugging level to 3, the maximum):

sh# $GLOBUS_LOCATION/sbin/sshd -ddd -o 'UsePrivilegeSeparation no' -r
You can add the '-p <port number>' option to run the sshd on an alternate port for debugging without affecting your system sshd. Then, give the same '-p <port number>' option to gsissh to test the sshd.
The -r flag is a new option in OpenSSH 3.9 and gsi_openssh 3.5. It is not needed for previous versions.

The presence of a debugging flag also runs the server without detaching it from the console. The server will only handle one connection in this mode.

Host key permissions failure

The host key that the SSH server is using for GSI authentication must only be readable by the user which owns it. Any other permissions will cause the following debugging output to be generated.

7959: debug1: Local version string SSH-1.99-OpenSSH_3.4p1
7959: debug1: list_hostkey_types: ssh-rsa,ssh-dss
7959: debug1: Problem with local credentials
7959: debug1: bad file system permissions on private key
  key must only be readable by the user
    File=/etc/grid-security/hostkey.pem
  Function:proxy_init_cred
 7959: debug1: SSH2_MSG_KEXINIT sent
7959: debug1: SSH2_MSG_KEXINIT received

Fix: Make sure that the host key's UNIX permissions are mode 400 (that is, it should only have mode readable for the user that owns the file, and no other mode bits should be set).

Unable to map implicit username

If the server was passed an "implicit username" (i.e. requested to map the incoming connection to a username based on some contextual clues such as the certificate's subject), and no entry exists in the grid-mapfile for the incoming connection's certificate subject, the server should output a clue that states it is unable to set the username against which to authenticate.

7978: debug2: input_userauth_request: try method none
7978: Failed none for cphillip from 141.142.21.10 port 1240 ssh2
7978: debug1: gssapi received empty username
7978: debug1: failed to set username from gssapi context
7978: Failed external-keyx for cphillip from 141.142.21.10 port 1240 ssh2
7978: debug1: gssapi received empty username
7978: debug1: failed to set username from gssapi context
7978: Failed gssapi for cphillip from 141.142.21.10 port 1240 ssh2
 7978: debug1: userauth-request for user cphillip service ssh-connection method publickey
7978: debug1: attempt 0 failures 3

Fix: Add a line to your grid-mapfile.

Entry in grid-mapfile refers to non-existent username

If the subject name given in the system's grid-mapfile points to a non-existent user, the server will give an internal error which is best caught when it is running in debugging mode.

8046: debug2: input_userauth_request: setting up authctxt for cphillip
8046: debug2: input_userauth_request: try method none
8046: Failed none for cphillip from 141.142.21.10 port 1259 ssh2
8046: debug1: gssapi received empty username
8046: debug1: gssapi successfully set username to cphillip2
8046: debug1: userauth-request for user cphillip2 service ssh-connection method external-keyx
8046: debug1: attempt 0 failures 1
8046: input_userauth_request: illegal user cphillip2
8046: debug2: input_userauth_request: try method external-keyx
8046: GSI user /C=US/O=National Computational Science Alliance/CN=Chase Phillips is authorized as target user cphillip2
8046: INTERNAL ERROR: authenticated invalid user cphillip2
 8046: debug1: Calling cleanup 0x806bb38(0x0)

Fix: Add a new account to the system matching the username pointed at by the user's subject in the grid-mapfile.

Client proxy uninitialized or non-GSI agent

Should the user attempt to connect without first creating a proxy certificate, or if the user is connecting via a SSH client that does not support GSI authentication, the server will note that no GSSAPI data was sent to it. Verify that the client is able to connect through another GSI service (such as the gatekeeper) to make sure that the user's proxy has been created correctly.

9597: debug2: input_userauth_request: setting up authctxt for cphillip
9597: debug2: input_userauth_request: try method none
9597: Failed none for cphillip from 141.142.21.10 port 2554 ssh2
9597: debug1: gssapi received empty username
9597: debug1: No suitable client data
 9597: debug1: failed to set username from gssapi context
9597: Failed external-keyx for cphillip from 141.142.21.10 port 2554 ssh2
9597: debug1: gssapi received empty username
9597: debug1: No suitable client data
9597: debug1: failed to set username from gssapi context
9597: Failed gssapi for cphillip from 141.142.21.10 port 2554 ssh2
9597: debug1: userauth-request for user cphillip service ssh-connection method publickey
9597: debug1: attempt 0 failures 3
9597: debug2: input_userauth_request: try method publickey

Fix: Verify that you are using a GSI-enabled SSH client and that your GSI proxy has been properly initialized via 'grid-proxy-info'. If you need to initialize this proxy, use the command 'grid-proxy-init'.

Account is locked

OpenSSH now checks for locked accounts by default. On Linux systems, locked accounts are defined as those that have !! in the password field of /etc/shadow. This is the default entry for accounts created with the useradd command. Even if you are using GSI authentication and do not need local passwords, sshd won't let the user login with this message:

Too many authentication failures for username

In the sshd debugging info it will indicate that the account is locked:

User username not allowed because account is locked

Here is some additional information from the sshd Manual:

Regardless of the authentication type, the account is checked to ensure that it is accessible. An account is not accessible if it is locked, listed in DenyUsers or its group is listed in DenyGroups. The definition of a locked account is system dependant. Some platforms have their own account database (eg AIX) and some modify the passwd field ( "*LK*" on Solaris and UnixWare, "*" on HP-UX, containing "Nologin" on Tru64, a leading "*LOCKED*" on FreeBSD and a leading "!!" on Linux). If there is a requirement to disable password authentication for the account while allowing still public-key, then the passwd field should be set to something other than these values (eg "NP" or "*NP*" ).

Fix: Replace !! with (for example) NP in /etc/shadow.