Linux exploit faq

CERN Linux Vulnerability FAQ

CERN Linux Vulnerability FAQ

How to check if a machine is vulnerable?

New security holes are unveiled continuously so, unless you upgrade your system regularly, it is very likely that your system is vulnerable.

If you use a certified/supported CERN Linux system (such as SLC5), most of the security holes have been fixed automatically for you. However, this only applies if you are running an automatic update mechanism, since new vulnerabilities may have been found between now and the time your version was installed.

If you do not use the currently supported CERN certified distributions (i.e. if you use another version of Red Hat Linux or another distribution), you have to check yourself that all the patches corresponding to your distribution have been applied.

What to do with compromised or vulnerable machines?

Compromised machines often have sniffers and backdoors installed and put other machines on the CERN network at risk: they must be switched off and reinstalled from scratch with a secure operating system as soon as possible. computer.security@cern.ch has to be informed as soon as you suspect a compromise, they will handle the case and escalate it as required - but cooperation by the machine responsible will still be required.

Vulnerable machines are dangerous and must be made secure as soon as possible to prevent breakins. The best solution is to reinstall a machine that was vulnerable for some period of time, since this will get rid of unnoticed compromises. Ideally, machine should never let become vulnerable:

  • handling a compromise is expensive (in terms of manpower) and disruptive (service stays down during the investigation)
  • reinstalling a vulnerable machine is less expensive (and can be planned beforehand to minimize service impact)
  • keeping a machine secured typically costs very little effort, since most updates can be applied automatically.

How to upgrade or (re)install a machine?

The recommended upgrade/installation procedure is to reinstall the machine from scratch:

  • this will remove any backdoor, sniffer, trojan horse, etc. left by an attacker, and will often remove data collected by those programs
  • it removes any temporary hack or modification that the owner or system manager forgot to clean afterwards and that may be disastrous for security
  • it is more reliable and more secure than upgrading only parts of the system
  • it forces the owner or system manager to know and understand what is really on the machine
  • it encourages the owner or system manager to automate the installation and this saves time (and therefore money) on the longer term

Note: reinstalling from scratch means that the Linux system partitions will be destroyed and recreated. If done carefully, the other partitions (containing Windows or data files) can be preserved.

The standard installation from scratch can be manual (interactive) or fully automated (even with customisations). See http://cern.ch/linux/ and/or contact linux.support@cern.ch for more information.

How to check if a machine has been compromised?

There is no reliable way to detect if a machine has been compromised. Attackers usually install "rootkits" that carefully hide their tracks, sometimes using kernel modules. This means a machine can not be trusted to correctly report information about itself.

As a first test, you can try to run /afs/cern.ch/project/security/cern/public/tools/check-rootkit/crk (you may need to copy the directory to local disk before running). This is not a comprehensive test (it will not detect all compromises). On SLC4, you can install a toolkit called rkhunter that can automate such checks from the central Linux software repository (# yum install rkhunter). Other such tools are chkrootkit or samhain (the latter requires a machine to be set up before a compromise would happen).

All these tools will not report all types of compromise, and may occasionally report legitimate activity or files as being suspicious (but these cases are usually worth inspecting more closely).

If you notice anything suspicious, please contact computer.security@cern.ch for assistance. If you have reason to suspect a machine (e.g. due to a low-privilege compromise or untrusted user on a machine missing security updates), please treat it as compromised until the incident has been fully understood.


old vulnerabilities (kept for historical purposes)

(no half-way current machine should be vulnerable to these anymore.)

December 2003

A vulnerability in the do_brk() kernel function allowed a local root level compromise, this hole is fixed via kernel 2.4.20-24. Please see this News article on how to upgrade your kernel.

March/April 2003

Please see the separate page on the "kernel ptrace" problem, it contains pointers to a checking script for the rootkit most widely used at the time..

October/November 2001

The rootkit used at CERN in October-November 2001 is fortunately easy to detect with the following recipe (for Red Hat Linux 6.x):

  1. check if the "sash" command (the stand-alone shell) is available on your system (usually in /sbin)
  2. if not, install it from the rpm /afs/cern.ch/project/linux/redhat/pro/RedHat/RPMS/sash-3.3-1.i386.rpm or copy it from /afs/cern.ch/project/security/tools/sash.redhat6
  3. run /sbin/sash -c "-ls /lib/libZ.a"
    if you do not get "/lib/libZ.a: No such file or directory", the rootkit is present
  4. run /sbin/sash -c "-ls /etc/ld.so.preload"
    if you do not get "/etc/ld.so.preload: No such file or directory" or only "libsafe.so" inside, the rootkit is present

Warning: if you do not detect a rootkit this way, it does not mean that you have no rootkit at all but only that this particular one is not present.