hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Boudnik <...@apache.org>
Subject Re: [ANNOUNCE] Hadoop-1.0.4 release, with Security fix
Date Mon, 15 Oct 2012 17:27:24 GMT
Thanks Matt - makes perfect sense!

Cos

On Sun, Oct 14, 2012 at 09:24PM, Matt Foley wrote:
> Hi Konstantin, Cos, & all,
> The sole purpose of the 1.0.4 release was to provide an immediate fix for
> Security issue CVE-2012-4449.  It was specifically requested by the Hadoop
> security subcommittee.  Of course, the way security fixes are handled in
> Hadoop, that reason couldn't be revealed until the release was actually
> done.
> 
> Other than that fix, 1.0.4 has only 3 straightforward fixes for rather
> severe issues, that were previously committed to branch-1.0:
>     HADOOP-7154 <https://issues.apache.org/jira/browse/HADOOP-7154> - set
> MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with glibc in
> RHEL-6
>     HDFS-3652 <https://issues.apache.org/jira/browse/HDFS-3652> - FSEditLog
> failure removes the wrong edit stream when storage dirs have same name
>     MAPREDUCE-4399 <https://issues.apache.org/jira/browse/MAPREDUCE-4399> - Fix
> (up to 3x) performance regression in shuffle
> 
> This upgrade should be easily adopted by users who just want a simple
> update to 1.0.x for the security issue.
> 
> Release 1.1.0, on the other hand, has approximately 135 enhancements and
> bug fixes compared to Hadoop-1.0.4, including:
> 
>    - many performance improvements in HDFS, backported from trunk
>    - improvements in Security to use SPNEGO instead of Kerberized SSL for
>    HTTP transactions
>    - lower default minimum heartbeat for task trackers from 3 sec to
>    300msec to increase job throughput on small clusters
>    - port Gridmix v3
>    - set MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with
>    glibc in RHEL-6
>    - splittable bzip2 files
> 
> This is a significant release with a lot of great improvements.  Of course
> it also has the security fix.  We can expect that 1.0.x users will want to
> upgrade to 1.1.0 to get the many improvements, but it may take longer than
> an update to 1.0.4.  In order to get the fix for CVE-2012-4449 into
> circulation as soon as possible, it made sense to release 1.0.4 as well.
> 
> Thanks,
> --Matt
> 
> On Sun, Oct 14, 2012 at 7:41 PM, Konstantin Boudnik <cos@apache.org> wrote:
> 
> > Yup, I was wondering about the same thing. BigTop is working on 0.3.1
> > release
> > based on Hadoop 1.1.0, so having and update for - essentially - 1.0.3 is a
> > bit
> > confusing.
> >
> > Thanks,
> >   Cos
> >
> > On Sun, Oct 14, 2012 at 12:16AM, Konstantin Shvachko wrote:
> > > Hi Matt,
> > >
> > > Could you please explain what is the difference between Hadoop 1.0.4
> > > just accepted and Hadoop 1.1.0 being
> > > voted at the same time? Also why is it important to keep and release
> > > both of these branches?
> > > I am lost here. I assume other people might have that question in mind
> > as well.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley <mattf@apache.org> wrote:
> > > > Hello,
> > > > The release of Hadoop-1.0.4 has been voted, accepted, and posted.
> > > > It is available in SVN and Maven, as well as at
> > > >     http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/
> > > >
> > > > It is still propagating to mirrors, and should be available on all
> > mirrors
> > > > by this time Saturday.
> > > > The documentation update is still being worked on and will be
> > available by
> > > > Monday.
> > > >
> > > > This release is noteworthy for including a Security bug fix, related to
> > > > CVE-2012-4449,
> > > > discovered by Daryn Sharp and fixed by Owen O'Malley.  The CVE
> > announcement
> > > > is below.
> > > >
> > > > Best regards,
> > > > --Matt Foley
> > > > Release Manager
> > > >
> > > > *CVE-2012-4449: Apache Hadoop security token vulnerabilities
> > > > *
> > > > Severity: Critical
> > > >
> > > > Vendor: The Apache Software Foundation
> > > >
> > > > Versions Affected:
> > > > 0.20.X: All versions
> > > > 0.23: All versions before 0.23.4
> > > > 1.0: All versions before 1.0.4
> > > > 2.0: All versions before 2.0.2
> > > >
> > > > Users affected:
> > > > Users who have enabled Hadoop's Kerberos security features.
> > > >
> > > > Impact:
> > > > Malicious users may crack the secret keys used to sign security
> > > > tokens, thus granting them the ability to fabricate tokens for
> > > > privilege escalation.  Malicious users may also launch unauthorized
> > > > tasks as an arbitrary user for privilege escalation.
> > > >
> > > > Description:
> > > > When Hadoop's security features are enabled, clients initially present
> > > > Kerberos credentials to authenticate to a service such as the
> > > > NameNode.  A client may then request a security token for subsequent
> > > > authentication within the Hadoop cluster.  The client receives a
> > > > security token and a corresponding signature for the token, generated
> > > > using the HMAC algorithm and a SHA1 hash.
> > > >
> > > > Token passwords are generated using a trivial secret key length (20
> > > > bits).  A key of this size can be brute forced in at most a few
> > > > seconds.  Once the secret is cracked, one can generate arbitrary
> > > > tokens to impersonate other users.  These fraudulent tokens may be
> > > > used to gain unauthorized access to data or disrupt services within
> > > > the cluster.  With default secret key rolling values, a cracked secret
> > > > may often be exploited for a couple days before another secret has to
> > > > be cracked.
> > > >
> > > > Some token-based services, such as the NameNode's delegation tokens
> > > > for the namespace, are immune from a compromised secret key because
> > > > they record the generated tokens.  A fraudulent token with a valid
> > > > password will rejected since the service will know it did not generate
> > > > the token.  Services that generate a token on behalf of another
> > > > service and rely on a shared secret for the other service to validate
> > > > the token's password are especially vulnerable.
> > > >
> > > > HDFS (all versions):
> > > > Malicious clients cannot gain unauthorized access to the namespace.
> > > > Malicious clients may however gain full access (read, write, and
> > > > delete) to any block based on knowledge of the block id.
> > > >
> > > > MapReduce (1.x):
> > > > Malicious clients may intercept task data, task logs, alter task
> > > > status, and disrupt tasks from executing or completing.  A malicious
> > > > client may also inject data into a Pipes-based job.
> > > >
> > > > Yarn (2.x only):
> > > > Malicious clients may perform the same attacks as MapReduce.  An
> > > > unauthorized yarn task may be launched unbeknownst to the
> > > > ResourceManager.  Additionally, the security tokens for launching
> > > > tasks do not contain the job submitter.  The user for task execution
> > > > is specified in an untrusted container launch context, thus allowing a
> > > > task to be launched as an arbitrary user.  When combined, an
> > > > unauthorized task may be launched as an arbitrary user.
> > > >
> > > > Other Hadoop projects:
> > > > Hadoop projects using the token management framework may be
> > > > susceptible if their services do not store the tokens issued, or if a
> > > > service generates tokens for other services.  This includes Apache
> > > > HBase version 0.92.0 or higher when the Kerberos-based security
> > > > features are enabled.
> > > >
> > > > Mitigation:
> > > > Users should immediately upgrade to the latest applicable release
> > > > (0.23.4, 1.0.4 or later, or 2.0.2), or should immediately apply the
> > > > patch provided below to their systems.
> > > >
> > > > Credit: This issue was discovered by Daryn Sharp of Yahoo! Inc.
> >

Mime
View raw message