[Triumf-linux-managers] Security breach on ibm00

[Kelvin Raywood]

This is an important security message from the TRIUMF Core Computing and 
Networking group (CCN).

As many of you already know, there was a security breach on the public
Linux machine ibm00 which is part of an NIS cluster with trcomp01,
trcomp02. It appears that an intruder used a vulnerability in the Linux
kernel to gain root privileges from a non-root account though at this
point we don't know which account nor how the intruder gained access to
it. There are a large number of accounts on the cluster, some of them
unused for a long time. So someone's credentials may have been guessed
as part of a dictionary attack, or have been stolen by some other means.

With root privileges, the intruder installed modified versions of
/usr/sbin/sshd and /usr/bin/ssh that captured username, password and, in 
the case of outgoing ssh conections, the address of the remote host.
There is no indication that ssh-key passphrases were captured.  The
information was stored in a scrambled (not encrypted) file on the
system.  We believe that the modified sshd also enabled the intruder to
login without a log entry and retrieve the capture passwords.

*What we are doing about it*

We have restored the legitimate versions of ssh and sshd.

We fixed the kernel vulnerability that enabled the original privilege
escalation.

We have blocked all access to ibm00 from off-site. This is
inconvenient for some, but trcomp01 (SL-4) and trcomp02 (SL-5) are
still available.

We were able to unscramble the file and have notified those whose
passwords have been captured. We have also notified those whose
password on another system has been captured. All have been advised to
change their passwords.

We are checking the password age, and will disable accounts of those
with a captured password if it is not changed by the end of next week
(June 25); with another warning of course.

We are planning on decommissioning ibm00 completely (probably end of
July) as security updates are no longer available for its version of
Linux.

We are continuing to investigate the consequences of this breach and are 
also looking for similar occurrences on other TRIUMF machines.  Many of 
you allow us root access to your machines by not removing the optional 
package triumf-ccn-rootkey that was installed by the TRIUMF kickstart. 
We have found some other cracked machines and will contact their owners 
separately.  See below for how to check yourself.

*About the original kernel vulnerability*

I drew attention to this vulnerability in August last year, and made
recommendations for nullifying it.  See
http://lists.triumf.ca/pipermail/triumf-linux-managers/2009-August/000233.html

We did apply this to ibm00 but, from the timestamps, it appears that
the intrusion happened a couple of hours earlier.

The vulnerability was fixed in kernels for ScientificLinux 4 and 5.
SL-5 2.6.18-128.7.1.el5 and later.
Sl-4 2.6.9-89.0.9.EL and later.

If unsure, check the build date of your kernel with "uname -v". If it is
earlier than Aug-24-2009 then your kernel is vulnerable to the
privilege escalation but can be made safe as described in my message.


*How to check if your machine has been cracked*

Verify that /usr/sbin/sshd and /usr/bin/ssh have not been modified.

     rpm -V openssh-{server,clients}

You should see output like the following:

    S.5....T  c /etc/ssh/sshd_config
    S.5....T  c /etc/ssh/ssh_config

This indicates that the Size, md5-sum and timestamps of the config
files are different than those installed with the rpm. This is normal.
Those configuration files are changed by TRIUMF rpms and/or by the
machine manager for various reasons.

However, you should not see any indication that /usr/sbin/sshd or
/usr/bin/ssh have been modified.

Also, check for the existence of "/usr/include/linux/boot.h". It should
not exist. This was the file used by the modified versions to store
captured passwords.

If you suspect that your Linux machine has modified versions of sshd or
ssh, or has the captured-passwords file, please contact us through
<https://helpdesk.triumf.ca>.

--
Kel Raywood
Core Computing and Networking
TRIUMF