Strange goings-on at Fedora

Where Penguins and Daemons chill together in the warmth of the Sun.

Moderators: SecretSquirrel, notfred

Strange goings-on at Fedora

Postposted on Tue Aug 19, 2008 10:05 pm

So... there have been no Fedora 9 updates posted to the Fedora repositories in about a week, which is unusual. There have also been several cryptic messages posted to the Fedora-announce list (here, here, and here). Still no official word on what this "issue" might be, but the recommendation not to download or update any Fedora packages until further notice is disturbing.

Given what little information there is to go on, I'd say it sounds like maybe their servers got hacked, and they're trying to verify that they've closed the hole and that none of the packages in their repositories have been compromised. If so, this is pretty bad... if not from a real security standpoint, at least from a PR standpoint for Redhat.

The Register is apparently speculating along the same lines...
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Wed Aug 20, 2008 8:21 am

That would be a shame. Redhat was the first Linux distro I used and I tried several version of Fedora.

I have since moved on to Kubuntu, but Redhat really got a lot of people into Linux and are one of the few success stories.

Hopefully this turns out to be a minor issue and they are just being over cautious on this.
Image
"Give me a scotch. I'm starving" ~ Tony Stark
PRIME1
Darth Gerbil
 
Posts: 7561
Joined: Mon Apr 22, 2002 4:07 pm
Location: , location

Re: Strange goings-on at Fedora

Postposted on Wed Aug 20, 2008 8:46 am

I hope they recover from this quickly, and without affecting users security. We have nothing but Fedoras at work :-?
radix
Gerbil First Class
 
Posts: 103
Joined: Sun Jan 13, 2002 6:00 pm
Location: Mountain View, CA

Re: Strange goings-on at Fedora

Postposted on Wed Aug 20, 2008 8:56 am

radix wrote:We have nothing but Fedoras at work :-?

Same here. And I downloaded a few updates on Monday... hope everything is ok.
GA-EP45-DS3R · E8500@3.8 · Scythe Ninja+ · 8GB G.Skill DDR2-800 · MSI TwinFrozr HD7850 · Corsair VX450 · Antec Solo
180GB i520 SSD · WD1002FAEX · Plextor PX-755SA · HP ZR24w · Intel EXPI9301CT · X-Fi XtremeGamer · Win7/Mint
nerdrage
Graphmaster Gerbil
 
Posts: 1271
Joined: Thu Mar 06, 2003 1:49 pm
Location: Raleigh/Durham, NC

Re: Strange goings-on at Fedora

Postposted on Thu Aug 21, 2008 10:37 pm

They apparently changed their public keys and SSH fingerprints a couple of days ago. Either they've had some sort of major security breach, or they're being really paranoid. Or both.

Really wish they'd release more info and let people know what's going on...
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Fri Aug 22, 2008 7:12 am

Maybe someone who was in at the top has left acrimoniously and they are busy changing the locks? That would be a bit better than a security breach.
notfred
Grand Gerbil Poohbah
 
Posts: 3647
Joined: Tue Aug 10, 2004 9:10 am
Location: Ottawa, Canada

Re: Strange goings-on at Fedora

Postposted on Fri Aug 22, 2008 8:23 am

They've finally posted an update on the situation, here.
Fedora wrote:... based on our efforts, we have high confidence that the intruder was not able to capture the passphrase used to secure the Fedora package signing key. Based on our review to date, the passphrase was not used during the time of the intrusion on the system and the passphrase is not stored on any of the Fedora servers.
...
Among our other analyses, we have also done numerous checks of the Fedora package collection, and a significant amount of source verification as well, and have found no discrepancies that would indicate any loss of package integrity. These efforts have also not resulted in the discovery of additional security vulnerabilities in packages provided by Fedora.

Bottom line for Fedora users: Yes, there was a security breach at Fedora. One of the systems breached was the system they use to apply digital signatures to official Fedora packages. It does not appear that the signing key was accessed, but they have changed it as a precaution. The official package repositories have not been compromised.

It appears that RHEL servers experienced a similar breach, resulting in the possibility that a small number of packages obtained through third parties may contain what appears to be a legitimate Redhat digital signature, when in fact the package is not legit:
Redhat wrote:... we remain highly confident that our systems and processes prevented the intrusion from compromising RHN or the content distributed via RHN and accordingly believe that customers who keep their systems updated using Red Hat Network are not at risk. We are issuing this alert primarily for those who may obtain Red Hat binary packages via channels other than those of official Red Hat subscribers.

In connection with the incident, the intruder was able to sign a small number of OpenSSH packages relating only to Red Hat Enterprise Linux 4 (i386 and x86_64 architectures only) and Red Hat Enterprise Linux 5 (x86_64 architecture only). As a precautionary measure, we are releasing an updated version of these packages, and have published a list of the tampered packages and how to detect them at http://www.redhat.com/security/data/ope ... klist.html

Bottom line for RHEL users: Don't install any updates to the OpenSSH package that didn't come from a trusted source, even if the package appears to be legit (has a valid digital signature). Official package repositories have not been compromised.

Scary stuff. This sounds like a very deliberate (and sophisticated) attempt to compromise servers running RHEL in the field, by trying to trick administrators to install an OpenSSH package which has been tampered with (but appear to be a legit Redhat package).
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Fri Aug 22, 2008 8:29 am

Honestly, that looks much more concerted than a random hacker. In fact, I wouldn't be surprised if it was a an attack coordinated by a large entity, like the Chinese gov't.

Red Hat still has a large share of production linux boxes, don't they?
Usacomp2k3
Gerbil God
 
Posts: 21240
Joined: Thu Apr 01, 2004 3:53 pm
Location: Orlando, FL

Re: Strange goings-on at Fedora

Postposted on Fri Aug 22, 2008 8:44 am

Usacomp2k3 wrote:Honestly, that looks much more concerted than a random hacker. In fact, I wouldn't be surprised if it was a an attack coordinated by a large entity, like the Chinese gov't.

Red Hat still has a large share of production linux boxes, don't they?

Yes, they do.

To expand a bit on my previous post... reading between the lines, I'd say someone was preparing to launch a phishing attack against sysadmins of servers running RHEL, in an attempt to get them to install a trojaned OpenSSH package which appears to be a legit Redhat package (based on its digital signature). OpenSSH is a key part of the security infrastructure on any *NIX box, and successfully compromising it would pretty much give an attacker free reign to do anything they want on a system, including stealing anything they want off of the hard drives, tampering with files, installing rootkits, keyloggers, etc.

My first thought was Eastern European or Russian Mafia. Online fraud, botnets, etc. are supposedly big business for them.

Still left unsaid is how the Fedora and Redhat servers were breached in the first place. Was it just stupidity (someone used a weak password), an inside job (someone got paid off), or is there some other as yet undisclosed security hole which allowed the breach to occur?
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Fri Aug 22, 2008 11:58 pm

Hmm... check this out: http://slashdot.org/comments.pl?sid=654149&cid=24711327

If this is true, I think things are gonna get ugly. This looks like a really sophisticated attack, which tries to cover its tracks by (among other things) messing with the system date to mask when the compromised code was installed.

RPMforge is not an official package repository, but is apparently fairly popular among Redhat and CentOS users. If RPMforge was in fact serving up trojaned packages, there could be a lot of compromised servers out there!

Moral of story: If you run a mission critical server, don't add unofficial repositories to your automatic update configuration...

Edit: I should add, I'm not accusing RPMforge of any wrongdoing here. Until proven otherwise, they are just a (potential) vector for the attack, and not the perpetrator.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Thu Aug 28, 2008 9:04 am

Still no new package updates posted to the Fedora 9 repositories since 8/11.

I wonder if there's still more to this that we haven't been told yet.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Mon Sep 08, 2008 10:15 pm

Post-mortem...

All updates which have been released for Fedora 8 and 9 are in the process of being re-released and pushed out to the official mirrors with a new digital signature. Details here and here.

What a freaking mess...
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Fri Sep 26, 2008 4:13 pm

just brew it! wrote:So... there have been no Fedora 9 updates posted to the Fedora repositories in about a week, which is unusual. There have also been several cryptic messages posted to the Fedora-announce list (here, here, and here). Still no official word on what this "issue" might be, but the recommendation not to download or update any Fedora packages until further notice is disturbing.

Given what little information there is to go on, I'd say it sounds like maybe their servers got hacked, and they're trying to verify that they've closed the hole and that none of the packages in their repositories have been compromised. If so, this is pretty bad... if not from a real security standpoint, at least from a PR standpoint for Redhat.

The Register is apparently speculating along the same lines...


Pretty bad when a Linux distro gets hacked of all things.
JonMCC33
Gerbil XP
 
Posts: 408
Joined: Tue Jun 10, 2008 9:37 pm
Location: Browsing with ads...

Re: Strange goings-on at Fedora

Postposted on Sat Sep 27, 2008 9:13 am

JonMCC33 wrote:Pretty bad when a Linux distro gets hacked of all things.

We may never know all of the details of how the compromise occurred (heck, Redhat/Fedora may not even know), but ultimately it was probably some form of human error. People can (and will) make mistakes, no matter what OS they're using. I'd be willing to bet someone at Redhat used a weak password, logged in remotely from an insecure system that had been compromised with a keylogger, or something along those lines.

At least with an Open Source OS, there is (out of necessity) enough transparency that users are aware when stuff like this happens. While I'm not entirely pleased with how long it took Redhat to 'fess up to what was going on, if something similar had happened to Microsoft we would've probably never known, and remained completely oblivious to the fact that compromised updates may have been installed on our systems.
(this space intentionally left blank)
just brew it!
Administrator
Gold subscriber
 
 
Posts: 36902
Joined: Tue Aug 20, 2002 9:51 pm
Location: Somewhere, having a beer

Re: Strange goings-on at Fedora

Postposted on Wed Apr 01, 2009 6:48 pm

Image
"Give me a scotch. I'm starving" ~ Tony Stark
PRIME1
Darth Gerbil
 
Posts: 7561
Joined: Mon Apr 22, 2002 4:07 pm
Location: , location

Re: Strange goings-on at Fedora

Postposted on Wed Apr 01, 2009 6:58 pm

JonMCC33 wrote:Pretty bad when a Linux distro gets hacked of all things.

BTW, just about all open source softwares are hacked continuously. Otherwise, there'd be no open source software to be had. It's when they get cracked that it's bad.
The best things in life are free.
http://www.gentoo.org
Guy 1: Surely, you will fold with me.
Guy 2: Alright, but don't call me Shirley.
titan
Grand Gerbil Poohbah
 
Posts: 3276
Joined: Mon Feb 18, 2002 6:00 pm
Location: Great Smoky Mountains


Return to Linux, Unix, and Assorted Madness

Who is online

Users browsing this forum: No registered users and 1 guest