Thanks Matt - makes perfect sense! Cos On Sun, Oct 14, 2012 at 09:24PM, Matt Foley wrote: > Hi Konstantin, Cos, & all, > The sole purpose of the 1.0.4 release was to provide an immediate fix for > Security issue CVE-2012-4449. It was specifically requested by the Hadoop > security subcommittee. Of course, the way security fixes are handled in > Hadoop, that reason couldn't be revealed until the release was actually > done. > > Other than that fix, 1.0.4 has only 3 straightforward fixes for rather > severe issues, that were previously committed to branch-1.0: > HADOOP-7154 - set > MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with glibc in > RHEL-6 > HDFS-3652 - FSEditLog > failure removes the wrong edit stream when storage dirs have same name > MAPREDUCE-4399 - Fix > (up to 3x) performance regression in shuffle > > This upgrade should be easily adopted by users who just want a simple > update to 1.0.x for the security issue. > > Release 1.1.0, on the other hand, has approximately 135 enhancements and > bug fixes compared to Hadoop-1.0.4, including: > > - many performance improvements in HDFS, backported from trunk > - improvements in Security to use SPNEGO instead of Kerberized SSL for > HTTP transactions > - lower default minimum heartbeat for task trackers from 3 sec to > 300msec to increase job throughput on small clusters > - port Gridmix v3 > - set MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with > glibc in RHEL-6 > - splittable bzip2 files > > This is a significant release with a lot of great improvements. Of course > it also has the security fix. We can expect that 1.0.x users will want to > upgrade to 1.1.0 to get the many improvements, but it may take longer than > an update to 1.0.4. In order to get the fix for CVE-2012-4449 into > circulation as soon as possible, it made sense to release 1.0.4 as well. > > Thanks, > --Matt > > On Sun, Oct 14, 2012 at 7:41 PM, Konstantin Boudnik wrote: > > > Yup, I was wondering about the same thing. BigTop is working on 0.3.1 > > release > > based on Hadoop 1.1.0, so having and update for - essentially - 1.0.3 is a > > bit > > confusing. > > > > Thanks, > > Cos > > > > On Sun, Oct 14, 2012 at 12:16AM, Konstantin Shvachko wrote: > > > Hi Matt, > > > > > > Could you please explain what is the difference between Hadoop 1.0.4 > > > just accepted and Hadoop 1.1.0 being > > > voted at the same time? Also why is it important to keep and release > > > both of these branches? > > > I am lost here. I assume other people might have that question in mind > > as well. > > > > > > Thanks, > > > --Konstantin > > > > > > On Fri, Oct 12, 2012 at 2:01 PM, Matt Foley wrote: > > > > Hello, > > > > The release of Hadoop-1.0.4 has been voted, accepted, and posted. > > > > It is available in SVN and Maven, as well as at > > > > http://www.us.apache.org/dist/hadoop/common/hadoop-1.0.4/ > > > > > > > > It is still propagating to mirrors, and should be available on all > > mirrors > > > > by this time Saturday. > > > > The documentation update is still being worked on and will be > > available by > > > > Monday. > > > > > > > > This release is noteworthy for including a Security bug fix, related to > > > > CVE-2012-4449, > > > > discovered by Daryn Sharp and fixed by Owen O'Malley. The CVE > > announcement > > > > is below. > > > > > > > > Best regards, > > > > --Matt Foley > > > > Release Manager > > > > > > > > *CVE-2012-4449: Apache Hadoop security token vulnerabilities > > > > * > > > > Severity: Critical > > > > > > > > Vendor: The Apache Software Foundation > > > > > > > > Versions Affected: > > > > 0.20.X: All versions > > > > 0.23: All versions before 0.23.4 > > > > 1.0: All versions before 1.0.4 > > > > 2.0: All versions before 2.0.2 > > > > > > > > Users affected: > > > > Users who have enabled Hadoop's Kerberos security features. > > > > > > > > Impact: > > > > Malicious users may crack the secret keys used to sign security > > > > tokens, thus granting them the ability to fabricate tokens for > > > > privilege escalation. Malicious users may also launch unauthorized > > > > tasks as an arbitrary user for privilege escalation. > > > > > > > > Description: > > > > When Hadoop's security features are enabled, clients initially present > > > > Kerberos credentials to authenticate to a service such as the > > > > NameNode. A client may then request a security token for subsequent > > > > authentication within the Hadoop cluster. The client receives a > > > > security token and a corresponding signature for the token, generated > > > > using the HMAC algorithm and a SHA1 hash. > > > > > > > > Token passwords are generated using a trivial secret key length (20 > > > > bits). A key of this size can be brute forced in at most a few > > > > seconds. Once the secret is cracked, one can generate arbitrary > > > > tokens to impersonate other users. These fraudulent tokens may be > > > > used to gain unauthorized access to data or disrupt services within > > > > the cluster. With default secret key rolling values, a cracked secret > > > > may often be exploited for a couple days before another secret has to > > > > be cracked. > > > > > > > > Some token-based services, such as the NameNode's delegation tokens > > > > for the namespace, are immune from a compromised secret key because > > > > they record the generated tokens. A fraudulent token with a valid > > > > password will rejected since the service will know it did not generate > > > > the token. Services that generate a token on behalf of another > > > > service and rely on a shared secret for the other service to validate > > > > the token's password are especially vulnerable. > > > > > > > > HDFS (all versions): > > > > Malicious clients cannot gain unauthorized access to the namespace. > > > > Malicious clients may however gain full access (read, write, and > > > > delete) to any block based on knowledge of the block id. > > > > > > > > MapReduce (1.x): > > > > Malicious clients may intercept task data, task logs, alter task > > > > status, and disrupt tasks from executing or completing. A malicious > > > > client may also inject data into a Pipes-based job. > > > > > > > > Yarn (2.x only): > > > > Malicious clients may perform the same attacks as MapReduce. An > > > > unauthorized yarn task may be launched unbeknownst to the > > > > ResourceManager. Additionally, the security tokens for launching > > > > tasks do not contain the job submitter. The user for task execution > > > > is specified in an untrusted container launch context, thus allowing a > > > > task to be launched as an arbitrary user. When combined, an > > > > unauthorized task may be launched as an arbitrary user. > > > > > > > > Other Hadoop projects: > > > > Hadoop projects using the token management framework may be > > > > susceptible if their services do not store the tokens issued, or if a > > > > service generates tokens for other services. This includes Apache > > > > HBase version 0.92.0 or higher when the Kerberos-based security > > > > features are enabled. > > > > > > > > Mitigation: > > > > Users should immediately upgrade to the latest applicable release > > > > (0.23.4, 1.0.4 or later, or 2.0.2), or should immediately apply the > > > > patch provided below to their systems. > > > > > > > > Credit: This issue was discovered by Daryn Sharp of Yahoo! Inc. > >