Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iD8DBQFObkU6QW+hXSf0t0IRAqMOAKC+GTtgmUT4jJU8TVtPaPUcaY1fNwCeKatj
96sDSwZ1uM3sAkIri7KhJDg=
=qf6u
-----END PGP SIGNATURE-----
Howdy,
Sad to be the bearer of bad news, but OSL was recently hacked (kernel.org and
linux.com are currently down), and that means naughty people had root access on
a machine on the OSL network for a while.
My recommendation to people with accounts on parrotvm.osuosl.org:
If you had an ssh private key stored there, disavow it.
If you ever ssh'ed from parrotvm to another machine, consider changing
that password/key as well.
We have no evidence yet (but we haven't looked hard) that our VM was hacked,
but someone could have very easily sniffed all the traffic from our VM.
Also, all password access to OSL machines has been removed (only ssh
key access is currently allowed).
If you accessed parrotvm via a password (like Coke++) you need to
email your ssh key to support@osuosl.org .
Duke
---------- Forwarded message ----------
From: Lance Albertson <lance@osuosl.org>
Date: Mon, Sep 12, 2011 at 10:45 AM
Subject: [directors] [Hosting] Security Alert: Please read immediately
To: hosting@osuosl.org
This message is long but very important. Please take a moment to read it
in its entirety.
As you may already know, on August 28, 2010, the Systems Administrator
for Kernel.org discovered that one of his primary servers, "Hera", had
been compromised. Multiple servers for Kernel.org are hosted at the OSUOSL.
Since discovery of the security breach, OSU Open Source Lab staff have
been cooperating with Kernel.org and Linux Foundation personnel to
uncover its source. We are also working together with a security expert
and the Linux Foundation to best understand the method of intrusion into
our hosted infrastructure.
Thus far, we have determined that attackers accessed at least one server
in addition to Hera, *cherry.osuosl.org*, but were not able to gain
full administrator access. We continue to work diligently on further
intrusion detection for all systems housed by the OSU Open Source Lab.
At this point, we have the following recommendations for you to perform
your own security audits on your machines. Even if OSUOSL staff
administer your machines, we encourage you to perform your own checks on
them. OSUOSL staff have completed these security audits on all machines
in our data center that we administer, but you are still encouraged to
perform your own checks even if we admin your machines.
Steps to Check for Compromise
* _Unexpected connections from hera.kernel.org (140.211.167.34)._
* Check for any unusual high numbered listening ports.
* Check for any suspicious SSH logins as far back as your logs go.
* Grep for Xnest in your kernel logs
If you find Xnest in your kernel logs, please follow up with us
*immediately*.
Steps to Increase Your Security (highly recommended, but not required)
* Have all your users update their passwords.
* Consider not allowing password ssh logins at all.
* Mark the following settings as "no" in sshd_config:
* ChallengeResponseAuthentication, PasswordAuthentication and
UsePAM
* Have all your users update their SSH keys.
* Ensure users aren't storing private SSH keys on hosts unless its
required.
What to Do if You Uncover Something:
_Whatever you do, do *not* wipe your machine after you uncover
something._ Maintain the state of the machine so that experts can assist
you and the OSUOSL staff in tracking down those responsible.
Staff at the OSUOSL are available to assist you, but consider that there
are many of you and fewer of us. Of course, please do let us know
*immediately* if you found something and are investigating it. Please do
so in a private fashion, e.g. private message to ramereth, jeff_s,
gchaix or lh on Freenode. We will likely refer you to your local FBI
field office for assistance: http://www.fbi.gov/contact-us/field
Even if you are not based in the United States, we will likely ask you
to work with the FBI on this investigation. They can help with
contacting any relevant law enforcement agencies local to you as needed.
Additional Steps for Users of cherry.osuosl.org:
We have rebuilt Cherry and disabled all shell account access moving
forward. If you require access to the OSL backend network, you will need
to contact OSUOSL staff for OpenVpn access. We ask that you contact us
via support@osuosl.org and thank you in advance for your patience as we
sort out all of these matters.
As a further precaution, we *HIGHLY* recommend that all users of Cherry
change their passwords and SSH keys.
Password Logins Disabled
We have disabled SSH password logins on all our hosts. If you attempt to
login to a machine and see the error "Permission denied (publickey)",
you may not be using a key or it may be that your account is in a locked
state on the box.
Please send email to support@osuosl.org to request help to resolve your
issue. Your patience is appreciated.
What We're Doing Going Forward to Make Things Better
We are working with our management to execute on the following plan:
* Disable all SSH password logins to our managed machines - done
* Replace root SSH keys on managed machines - done
* Audit all systems housed at OSUOSL that are managed by our systems
administration team for further security breaches - in progress
* Work with our hosted projects who handle their own administration
to ensure they conduct their own audits - in progress
* Audit our current system monitoring tools and processes to ensure
they're up to the task - in progress
* Add additional system monitoring support for each of our hosted
projects - planned
* Hire additional systems administration staff to ensure more eyes
watch this problem and are available to be assigned to these high
priority security related tasks - in progress
At this point, we are also planning to share whatever details of our
post-mortem analysis we will be able to share widely once the dust settles.
If you have any questions, please contact Jeff Sheltren or Leslie
Hawthorn, Jeff_S or lh on Freenode; jeff@osuosl.org and leslie@osuosl.org.
Thanks!
--
Lance Albertson
Systems Administrator / Architect Open Source Lab
Information Services Oregon State University
_______________________________________________
Hosting mailing list
Hosting@osuosl.org
http://lists.osuosl.org/mailman/listinfo/hosting
_______________________________________________
directors mailing list
directors@lists.parrot.org
http://lists.parrot.org/mailman/listinfo/parrot-directors
--
Jonathan "Duke" Leto <jonathan@leto.net>
Leto Labs LLC
209.691.DUKE // http://labs.leto.net
NOTE: Personal email is only checked twice a day at 10am/2pm PST,
please call/text for time-sensitive matters.
No comments:
Post a Comment