hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: [VOTE] Release candidate 0.20.203.0-rc0
Date Tue, 03 May 2011 23:58:50 GMT
Just to gauge what amount of stuff is in branch-0.20-security-203 I wrote a
quick script which does a comparison based on JIRAs mention in the commit
log. It output the following list of JIRAs that are in the branch but not
committed to trunk. I've marked many as N/A meaning that they don't apply to
trunk:

< HADOOP-1026
< HADOOP-6304 N/A
< HADOOP-6544
< HADOOP-6598 N/A
< HADOOP-6638
< HADOOP-6653 N/A
< HADOOP-6716 N/A
 < HADOOP-6718 N/A
< HADOOP-6728
< HADOOP-6745
< HADOOP-6757
< HADOOP-6776 N/A
< HADOOP-6808
< HADOOP-6810 N/A
< HADOOP-6832
< HADOOP-6855 N/A
< HADOOP-7143 N/A
< HADOOP-7190 N/A
< HADOOP-7232 N/A
< HADOOP-7243 N/A
< HADOOP-7246 (not sure about this)
< HADOOP-7247
< HADOOP-7253
< HDFS-1020 N/A
< HDFS-1022
< HDFS-1136 N/A
< HDFS-1153
< HDFS-1158
< HDFS-1313 N/A
< HDFS-495 N/A
< HDFS-6599 <-- must be a typo in commit log
< HDFS-740  N/A
< MAPREDUCE-1088
< MAPREDUCE-1100
< MAPREDUCE-1118
< MAPREDUCE-1207
< MAPREDUCE-1361 N/A
< MAPREDUCE-1376 N/A
< MAPREDUCE-1442 N/A
< MAPREDUCE-1521
< MAPREDUCE-1526 N/A
< MAPREDUCE-1550 N/A
< MAPREDUCE-1594 N/A
< MAPREDUCE-1641
< MAPREDUCE-1671
< MAPREDUCE-1672
< MAPREDUCE-1676
< MAPREDUCE-1677
< MAPREDUCE-1682
< MAPREDUCE-1687
< MAPREDUCE-1699 Unclear
< MAPREDUCE-1711 N/A
< MAPREDUCE-1713
< MAPREDUCE-1716
< MAPREDUCE-1730
< MAPREDUCE-1741
< MAPREDUCE-1744
< MAPREDUCE-1758
< MAPREDUCE-1759 N/A
< MAPREDUCE-1778 N/A
< MAPREDUCE-1790
< MAPREDUCE-1807 N/A
< MAPREDUCE-1854
< MAPREDUCE-1871
< MAPREDUCE-1872
< MAPREDUCE-1882
< MAPREDUCE-1889
< MAPREDUCE-1890
< MAPREDUCE-1914
< MAPREDUCE-1921
< MAPREDUCE-1933
< MAPREDUCE-1934
< MAPREDUCE-1938
< MAPREDUCE-1943
< MAPREDUCE-1954
< MAPREDUCE-1955
< MAPREDUCE-1960
< MAPREDUCE-1964
< MAPREDUCE-1966
< MAPREDUCE-1971
< MAPREDUCE-2005 N/A
< MAPREDUCE-2019
< MAPREDUCE-2055
< MAPREDUCE-2316
< MAPREDUCE-2355
< MAPREDUCE-291
< MAPREDUCE-323
< MAPREDUCE-339
< MAPREDUCE-517
< MAPREDUCE-6419 <-- doesnt exist, probably typo

Certainly some of these are new test cases, benchmark improvements, or
system tests. But many others are large new features (e.g new metrics
framework, separate JobHistory service). Others also introduce new
configurations (eg new JT based limits). In the list above there are 58 that
seem to be applicable, probably at least half of which are non-test code.

This list above doesn't include 192 patches that were committed to the
branch without reference to any JIRA in the commit message:
todd@todd-w510:~/git/hadoop-common$ for x in $(git rev-list
origin/branch-0.20..origin/branch-0.20-security-203 -- src) ; do git log -n1
$x | egrep -q '(MAPREDUCE|HDFS|HADOOP)[-:][0-9]+' || echo $x ; done | wc -l
192
Browsing through these, many have already been forward ported, or at least
had corresponding JIRAs opened. But it's very difficult to match them up and
evaluate which ones have been committed. Eli pointed out one earlier this
week that was done by a non-committer with no public review that introduced
an apparent performance regression; it's difficult to know whether there
might be others as well.

Rather than being a "maintenance release" (as is usually expected when
incrementing the third component of a version string) this is essentially a
separate trunk off of 0.20. I agree that the advancements in this branch are
many, and are a great set of contributions for the community. User limits
and security are two such that have been cited in this thread; unfortunately
the new improvements in limits haven't been committed to trunk, and the
security in trunk has a known root exploit. Do users really want to see us
putting these things in 20 without making sure they'll also show up in
future releases?

Looking at recent history of 204 it seems some more patches have gone in
there before going into trunk as well - for example MR-2429. Arun and Sid
are working on forward-porting it, and it's obviously not due to any kind of
bad intent that it was missed, but it underscores the dangers of having
essentially two trunks in ASF. I completely agree that there should be
long-term maintenance branches at the ASF, but we need to establish a clear
process to make sure that "maintenance" doesn't diverge into something else.

Here are two requests that others have made but I haven't seen an answer to
yet:

- Document the criteria by which developers can judge whether an improvement
should be included in branch-0.20-security. The inclusion criteria for the
branch as it stands is not clear -- given this branch's lineage, it clearly
used to be "things important for the Yahoo clusters", but that doesn't seem
like a reasonable community criterion. Up until now in Hadoop's history, the
criteria has always been "compatible bug fixes only", which doesn't describe
this branch either.

- Clearly establish the process that all patches must either be committed to
trunk first (and then backported), or include a comment on the JIRA
explaining why this is not necessary. Additionally we should decide whether
patches must be backported "through" 0.22 or if they may skip back from
trunk to 20-security. (I'm assuming 21 is dead here)

Perhaps this could go on a wiki page (or web site page) regarding the
currently active branches?

Thanks
-Todd

On Tue, May 3, 2011 at 10:02 AM, Eli Collins <eli@cloudera.com> wrote:

> I think we still need to incorporate the patches currently checked
> into branch 0.20.  For example, Owen identified a major bug
> (BooleanWritable's comparator is broken) and filed a jira
> (HADOOP-6928) to put it in branch-0.20, where I reviewed it and
> checked it in, so this bug would be fixed in the next stable release.
> However this change is not in branch-0.20-security-203. Unless we put
> the delta from branch-0.20 into this release, it is missing important
> bug fixes that will cause it to regress against 20.3 (if it ever is
> released).
>
> I am also nervous about changes like the one identified by
> HADOOP-7255. It looks like this change caused a significant regression
> in TestDFSIO throughput. It changes the core Task class, the commit
> log is a single line, and as far as I can tell it was not discussed or
> reviewed by anyone in the community. Don't changes like this at least
> deserve a jira before we release them?
>
> Thanks,
> Eli
>
> On Tue, May 3, 2011 at 1:39 AM, Konstantin Shvachko
> <shv.hadoop@gmail.com> wrote:
> > I think its a good idea to release hadoop-0.20.203. It moves Apache
> Hadoop a
> > step forward.
> >
> > Looks like the technical difficulties are resolved now with latest Arun's
> > commits.
> > Being a superset of hadoop-0.20.2 it can be considered based on one of
> the
> > official Apache releases.
> > I don't think there was a lack of discussions on the lists about the
> issues
> > included in the release candidate. Todd did a thorough review of the
> entire
> > security branch. Many developers participated in discussions.
> > Agreeing with Stack I wish HBase was considered a primary target for
> Hadoop
> > support. But it is not realistic to have it in hadoop-0.20.203.
> > I have some experience running a version of this release candidate on a
> > large cluster. It works. I would add a couple of patches, which make it
> run
> > on Windows for me like HADOOP-7110, HADOOP-7126. But those are not
> blockers.
> >
> > Thanks,
> > --Konstantin
> >
> >
> > On Mon, May 2, 2011 at 5:12 PM, Ian Holsman <hadoop@holsman.net> wrote:
> >
> >>
> >> On May 3, 2011, at 9:58 AM, Arun C Murthy wrote:
> >>
> >> >>
> >> >> Owen, Suresh and I have committed everything on this list except
> >> >> HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/
> >> >> necessary, I'll check with Cos.  Other than that hadoop-0.20.203 now
> a
> >> >> superset of hadoop-0.20.2.
> >> >>
> >> >
> >> > Missed adding HADOOP-5759 to that list, I'll check with Amareshwari
> >> before committing.
> >> >
> >> > Arun
> >>
> >> Thanks for doing this so fast Arun.
> >>
> >>
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message