Skip to content

Access using Kerberos

Accessing CERN Linux machines via Kerberos

Kerberos is shared-secret networked authentication system. Its use at CERN serves a dual purpose:

  • user convenience: once signed in, authentication is handled "automagically" on the users' behalf.
  • security: by reducing the need to enter passwords into potentially compromised machines, the impact of such a compromise can be greatly reduced.

In the past both Kerberos version IV and 5 were supported: Kerberos IV support has been stopped in 2010.

This document does not aim to explain Kerberos. For an overview and introduction into the concepts, please see man kerberos, MIT's site or the Kerberos FAQ. Several books exist on the subject as well.

NOTE: For MAC clients related support please check https://devices.docs.cern.ch/devices/mac/AboutKerberosAndSsh

Client-side configuration

In order for Kerberos to work, the user needs to be registered in a central database (KDC). In CERN parlance, this means a CERN account is required for the user.

Kerberos client binaries are part of the default install of many operating systems (such as Mac OS-X, BSD, Linux, Solaris, ..), or are at least available as add-ons. This document does not cover how to install these binaries on a particular operating system.

Linux clients must be registered with "LINUX" as the "Operating System" in the network database (go to https://network.cern.ch to check the registration). The precise "Version" of the "Operating System" does not matter for this purpose.

Please note that hosts with a dynamic IP (ie. within the .dyndns.cern.ch subdomain) can not get Kerberos tokens. Please see KB0004217 for more information.

Trying it out

To test whether a client is configured correctly, please use kinit and klist. These commands should live in your default PATH, otherwise look into /usr/kerberos/bin or /usr/heimdal/bin (please avoid using kinit from a Java JRE, these often have problems such as echoing the password).

$ kinit -f afsusername@CERN.CH
afsusername@CERN.CH's Password: <enter your password>
$
  • no message means no error
  • your password should not get displayed on the terminal
  • On a correctly configured machine inside the CERN domain, kinit afsusername would be sufficient.

In order to verify that everything went fine, use klist:

$ klist
Credentials cache: FILE:/tmp/krb5cc_<i>USERID</i>_<i>somenumber</i>
        Principal: <i>afsusername</i>@CERN.CH

  Issued           Expires          Principal
Sep 28 15:20:24  Sep 29 16:20:23  krbtgt/CERN.CH@CERN.CH
Sep 28 15:20:24  Sep 29 16:20:23  afs@CERN.CH

Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached

In this example, only Kerberos 5 credentials for the CERN realm were obtained as Kerberos 4 is no longer supported, and the expiry time is in the future (at the time of this writing). Looks good.

In order to get access to AFS after a successful authentication, you will need to get a "forwardable" TGT, and instruct your ssh client to actually transfer it over to the server (who will then use it to obtain an AFS token for your session). Use klist -f to check whether your TGT is "forwardable" - usually it then has a flag "F":

  Issued           Expires        Flags    Principal
Sep 29 13:24:41  Sep 30 13:24:03  FPR      krbtgt/CERN.CH@CERN.CH

To continue testing, use ssh -v afsusername@hostname.cern.ch klist -f\; tokens\; touch .sshtest (this should log you in, show your credentials - both Kerberos and AFS, and try to actually write something into your AFS home directory) - if all succeeds, you are set to use passwordless Kerberos. If you get error messages from these commands or are being asked for your passwords, the "verbose" output should allow you to pinpoint what went wrong. You may need to explicitly set some of the ssh client options.

Client configuration - Kerberos

In order to use CERN authentication, the name of the CERN KDC(s) need to be specified explicitly.

To get this done, a working config file (with CERN as default realm) can be copied from here. Default location for this configuration file is /etc/krb5.conf on most Linuxes and Mac OS X (here Library/Preferences/edu.mit.Kerberos would work as well), on some systems (Solaris) it is in /etc/krb5/.

You can edit the krb5.conf file using the following syntax, which is accepted by both the MIT and Heimdal Kerberos implementations.

As an alternative to editing the krb5.conf directly, the below information can typically also be entered via any Kerberos-configuring user interface your vendor may provide. If you do not have "root" privileges on the client machine, you can instruct MIT Kerberos to look at a different file by setting the KRB5_CONFIG environment variable.

[realms]
; AD  : This Kerberos configuration is for CERN's Active Directory realm.
[libdefaults]
 default_realm = CERN.CH
 ticket_lifetime = 25h
 renew_lifetime = 120h
 forwardable = true
 proxiable = true
 default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes256-cts-hmac-sha384-192 camellia256-cts-cmac aes128-cts-hmac-sha1-96 aes128-cts-hmac-sha256-128 camellia128-cts-cmac
 chpw_prompt = true
 rdns = false
 ...
[realms]
...
 CERN.CH = {
  default_domain = cern.ch
  kdc = cerndc.cern.ch
}
...
[domain_realm]
 .cern.ch = CERN.CH
...

Once you can get TGTs from the CERN realm as described above, the client-side Kerberos configuration is assumed to work.

Using multiple realms

In case you need to access different realms at the same time, you can take advantage of the credential cache collection feature of the MIT Kerberos client.

To do so, one of these credential cache types is required:

  • DIR (version 1.10)
  • KEYRING (version 1.12)
  • KCM (version 1.13)

Be sure to check the version of you Kerberos client before choosing:

$ klist -V

And configure the default_ccache_name parameter in /etc/krb5.conf or /etc/krb5.conf.d/*:

[libdefaults]
default_ccache_name = KEYRING:persistent:%{uid}
; OR
default_ccache_name = KCM:
; OR
default_ccache_name = DIR:/run/user/%{uid}/krb5cc

This way, you will be able to kinit multiple times and keep each credential cache active. You can list all of them by running:

$ klist -A

And you can clean up all of them with this command:

$ kdestroy -A

Client configuration - SSH

Two flavours of the SSH protocol exist, which support Kerberos differently: * SSH1: ancient and theoretically less secure, it had nevertheless direct support for Kerberos4, Kerberos5 and AFS. Typically, SSH programs nowadays needs to be specifically patched/recompiled to understand Kerberos4 and AFS. * SSH2: the modern successor that does not "speak" Kerberos directly, but uses an intermediate mechanism called GSSAPI ("Generic Security Services API", which itself uses Kerberos5). It is blissfully unaware of AFS, but widely available.

A common implementation of these protocols is found in OpenSSH, widely used in the Linux world, and also part of CERN Linux distributions.

ssh clients may need some help to try Kerberos for authentication. The following options should be enabled (via the command line or in ~/.ssh/config, see man ssh_config):

  • GSSAPIAuthentication: try Kerberos5 authentication
  • GSSAPIDelegateCredentials: tell the client to forward the Kerberos5 credentials to the remote side

The command-line switch "-v" (verbose) usually is very helpful - you should see messages mentioning Kerberos/AFS/GSSAPI (the following examples are from different sessions):

debug1: Trying Kerberos v5 authentication.
debug1: Kerberos v5 authentication accepted.
debug1: Kerberos v4 authentication accepted.
debug1: Kerberos v4 challenge successful.
debug1: Kerberos v4 TGT forwarded (afsaccount@CERN.CH).
debug1: AFS token for cell cern.ch forwarded.
debug1: Next authentication method: gssapi-with-mic
debug1: Delegating credentials
debug1: Authentication succeeded (gssapi-with-mic).

Some common SSH errors:

no credentials on the client
debug1: Kerberos v5: krb5_mk_req failed: No credentials cache found
debug1: Kerberos v4 krb_mk_req failed: Couldn't access ticket file (tf_util)
debug1: Miscellaneous failure
 No credentials cache found

This should get fixed by re-running kinit.

credentials not delegated

This appears to effect Mac OS Tiger (and newer?).

Symptons: You can login without having to specify a password, but you get

fs: You don't have the required access rights on '/afs/cern.ch/user/j/jouser'
hepix: E: /usr/bin/fs returned error, no tokens?

Symptons: You get "Permission denied" when using AFS or EOS paths, for example on LXPLUS or AIADM.

Make sure you use the following options either in your local ~/.ssh/config, or in your local systemwide /etc/ssh_config (could also be at /etc/ssh/ssh_config):

HOST lxplus*
    GSSAPITrustDns yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes

NOTE: Please be aware old kerberos configurations on MAC clients used different encryption that are not compatible with CentOS 8 / CentOS Stream 8. Refer to https://devices.docs.cern.ch/devices/mac/AboutKerberosAndSsh/ for further information.

Problem with IP aliasing

(this effects Mac OS Lion). Possible symptons: * You are asked for a password, despite having a valid token, or * you are logged in via your key, but you get no afs token.

In both cases, when using the "-v" option to ssh, you get output like

debug1: Next authentication method: gssapi-with-mic
debug1: Miscellaneous failure (see text)
Server (host/lx64slc5.cern.ch@CERN.CH) unknown while looking up
'host/lx64slc5.cern.ch@CERN.CH' (cached result, timeout in 853 sec)

debug1: An invalid name was supplied
unknown mech-code 0 for mech 1 2 752 43 14 2

Here the client does a reverse lookup of the host, but does not find anything for the alias used. The only solution is to add the following options to the ssh_config.

HOST lxplus*
    GSSAPITrustDns yes
for all clusters that use the IP alias round-robin for loadbalancing (lxplus, lxadm,...).

Please read the ssh_config man pages before modifying one of the config files.

While you are at it you might want to add the ForwardX11 yes for convenience in the HOST lxplus* section, but in that case do not forget to put the ForwardX11Trusted no there as well, for not making your setup too insecure. It is not recommended to put the GSSAPIDelegateCredentials yes or the ForwardX11 yes in the global section of the systemwide ssh_config file.

Buggy GSSAPI call inside openssh

(this e.g. affects MacOS ssh and most "vendor" ssh'es when connecting to one of the DNS-round-robin clusters such as LXPLUS, it will fall through to password authentication in such cases):

debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Miscellaneous failure
Generic error (see e-text)

or

debug1: Delegating credentials
debug1: An invalid name was supplied
No error

A symptom is that the credentials cache ("klist") contains a service ticket (host/lxplus123.cern.ch@CERN.CH) that does not match the machine ssh actually connected to:

debug1: Connecting to lxplus [137.138.4.22] port 22.
(in this case lxplus == 137.138.4.22 == lxplus207 != lxplus123).

This bug has been patched in the CERN SLC ssh client and in some other Linux distributions, it is being tracked upstream at the OpenSSH bugzilla. If you are lucky, your man ssh_config knows about a GSSAPITrustDNS parameter - in this case please try ssh -oGSSAPITrustDNS=yes .....

Other workarounds include using the SSH-1 protocol (ssh -1 ... - if your version still supports this, and supports Kerberos over SSH-1), or connecting to an individual node out of the cluster, instead of to the DNS alias (i.e. use ssh lxplus202.cern.ch instead of ssh lxplus.cern.ch). Obviously you will miss the benefits of the DNS alias - occasionally your target may be down, overloaded or simply no longer there.

Client configuration - other applications

It is possible to use Kerberos for local authentication (this is how CERN Linux machines are generally set up). However, such a setup only makes sense on machines that are permanently connected to the network, and have permanent connectivity to the CERN KDCs. In such cases, TGTs are typically obtained directly on login and get renewed via screensavers. For CERN Linux machines, a variant of Red Hat's pam_krb5 is being used to achieve this.

Besides SSH, other network services can also be made Kerberos-aware. Examples including web services (via the SPNEGO/GSSAPI authentication mechanism), ORACLE or CERN tools such as the Quattor CDB client.

Server-side configuration and troubleshooting

Kerberos is a shared-secret authentication system. This implies that both the users and the machines need to be registered at a central place, the KDC. These shared secrets are versioned, and both the machine and the KDC need to be using the same version of the secret. Most CERN Linux machines should get registered at install time, but in case of authentication problems the following may help to troubleshoot.

  • Check that /etc/krb5.keytab is present, non-empty, owned by "root" and readable only for "root".

    $ ls -l /etc/krb5.keytab
    -rw-------    1 root     root          197 Jan 14  2005 /etc/krb5.keytab
    

    If file is missing, run cern-get-keytab as "root" to (re-)create them. It will only work on the CERN network.

    Note: cern-get-keytab is available only for CERN supported Linux versions: Scientific Linux 5,6, .. (and Red Hat Enterprise Linux). If you run different system please see: msktutil paragraph below.

    This program (re-)creates the machine's secret on the KDC and stores a matching version inside these files. By running it, you invalidate existing service tickets for the machine in question (users will need to re-run kinit before they can connect again).

  • If these files exist, verify that the "key version" number (KVNO) is correct: On the server, run

    $ klist -k -t -e
    Keytab name: WRFILE:/etc/krb5.keytab
    KVNO Timestamp         Principal
    ---- ----------------- --------------------------------------------------------
    <b>3</b> 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (arcfour-hmac)
    <b>3</b> 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (aes128-cts-hmac-sha1-96)
    <b>3</b> 01/14/13 11:17:02 host/hostname.cern.ch@CERN.CH (aes256-cts-hmac-sha1-96)
    

    On a client, use

    $ kvno host/hostname.cern.ch@CERN.CH
    host/hostname.cern.ch@CERN.CH: kvno = <b>2</b>
    
    to get the KDC's idea of the current KVNO. If they are different (as in the example), remove /etc/krb5.keytab and recreate them as per above.

    This is a frequent problem on multi-boot systems when you use the same hostname for several systems. When the keytab is recreated on one of the systems, it invalidates the keytab files of the other systems, requiring you to use a password. In /var/log/secure you will see messages like

    Sep 27 14:08:55 myhost sshd[25361]: pam_krb5[25361]: authentication fails for 'jouser' (jouser@CERN.CH): Authentication failure (Success)
    Sep 27 14:08:55 myhost sshd[25361]: pam_krb5[25361]: TGT failed verification using keytab and key for 'host/myhost.cern.ch@CERN.CH': Key version number for principal in key table is incorrect
    

    One possible work-around is to copy the modified keytab file to the other systems on that node using the same hostname.

  • Check /var/log/messages for hints why a given service cannot use Kerberos, for example if it has trouble accessing the keytab files. For sshd, you might want to run in debug mode (sshd -d -d -d -p someport) for further hints.

msktutil: acquiring keytab file on unsupported linux system.

Note: For CERN supported Linux systems, it is strongly advised to use the cern-get-keytab utility exclusively. To prevent potential issues with outdated or corrupted keytabs, users are recommended to generate their keytab using cern-get-keytab --user on a RHEL 9 system, such as lxplus.cern.ch, as of the time of this writing.

In order to acquire keytab from CERN KDC msktutil version 1.1 (or newer) is needed. It may be included in your Linux distribution, if not, sources are available from msktutil site.

Obtaining initial keytab using msktutil

  • configure your /etc/krb5.conf as described in this document above.
  • check in Network Database if your account is authorized to manage the system for which you want to acquire the keytab.
  • As root on your system execute:

    $ curl -k --local-port 600-700 https://lxkerbwin.cern.ch/LxKerb.asmx/ResetComputerPasswordTimeCheck?service=host\&t=$(date -u +%FT%T)
    

    (--local-port option is available in curl 7.15.2 or newer only) Note: This will RESET current Kerberos host identity (if any), invalidating already existing keys/keytabs. * above curl command will return a result alike:

     <success>true</success>
     <principalname>host/HOSTNAME.cern.ch</principalname>
     <hostname>HOSTNAME</hostname>
     <samaccountname>SAMACCOUNTNAME$</samaccountname>
     <computerpassword>COMPUTERPASSWORD</computerpassword>
     

  • wait ~ 20 seconds (to allow resynchronization of all Domain Controllers).

  • Make sure that there are no user Kerberos credentials cached, clearing current ones:
    $ kdestroy; klist
    
  • As root on your system execute:
    $ msktutil --update --dont-expire-password --server cerndcXX.cern.ch --computer-name <HOSTNAME> --service host --old-account-password <COMPUTERPASSWORD> (--verbose)
    
    (please note: msktutil will reset computer account password in Active Directory to an random value, since this password is not needed anymore, for subsequent runs the existing keytab will be used to authenticate)
  • (to find exact name of KDC (cerndcXX), please use $ dig -t SRV _kerberos._tcp.cern.ch, do not use cerndc.cern.ch - that will fail)
  • New system keytab will be created by msktutil as /etc/krb5.keytab. (please verify permissions and selinux security context if SELinux in use, correct permissions are:
    $ ls -l /etc/krb5.keytab
    -rw-------. 1 root root 861 Feb 25 11:17 /etc/krb5.keytab
    
    correct selinux context is:
    $ ls -Z /etc/krb5.keytab
    -rw-------. root root system_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab
    
    use chown, chmod, chcon to change if needed)

Keytab recreation/acquiring additional service principal names using msktutil

Once initial keytab has been created (and is still valid) it can be regenerated using msktutil:

  • As root execute:
    $ msktutil --update --dont-expire-password --server cerndcXX.cern.ch (--verbose)
    
    (existing keytab will be used for authentication).

Additional service principal names (for example used for implementing http server kerberos authentication) / keys can be obtained also using msktutil: * As root execute:

$ msktutil --update --dont-expire-password --server cerndcXX.cern.ch --service HTTP (--verbose)