hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Patch review process
Date Tue, 10 Feb 2015 10:10:55 GMT

On 9 February 2015 at 21:18:52, Colin P. McCabe (cmccabe@apache.org<mailto:cmccabe@apache.org>)

What happened with the Crucible experiment? Did we get a chance to
try that out? That would be a great way to speed up patch reviews,
and one that is well-integrated with JIRA.

I am -1 on Gerrit unless we can find a way to mirror the comments to
JIRA. I think splitting the discussion is a far worse thing that
letting a few minor patches languish for a while (even assuming that
gerrit would solve this, which seems unclear to me). The health of
the community is most important.

I think it is normal and healthy to post on hdfs-dev, email
developers, or hold a meeting to try to promote your patch and/or
idea. Some of the discussion here seems to be assuming that Hadoop is
a machine for turning patch available JIRAs into commits. It's not.
It's a community, and sometimes it is necessary to email people or
talk to them to get them to help with your JIRA.

I know your heart is in the right place, but the JIRA examples given
here are not that persuasive. Both of them are things that we would
not encounter on a real cluster (nobody uses Hadoop with ipv6, nobody
uses Hadoop without setting up DNS).

Got some bad news there. The real world is messy, and the way Hadoop tends to fail right now
leaves java stack traces that tend to leave people assuming its Hadoop side.

Messy networks are extra commonplace amongst people learning to use Hadoop themselves, future
community members, and when you are bringing up VMs.

In production, well, talk to your colleagues in support and say "how often do you field network-related
problems?", followed by "do you think Hadoop could do more to help here?"

But, if we find a specific set
of issues that the community has ignored (such as good error messages
in bad networking setups, configuration issues, etc.), then we could
create an umbrella JIRA and make a sustained effort to get it done.

Seems like a good strategy.

 I've just created https://issues.apache.org/jira/browse/HADOOP-11571, "get S3a production
ready". It shipped in Hadoop 2.6; now it's out in the wild the bug reports are starting to
come back in. Mostly scale related; some failure handling, some improvements to work behind
proxies and with non-AWS endpoints.

  1.  To date all the s3a code has come from none committers; the original codebase
  2.  Most of the ongoing dev from is Thomas Demoor at amplidata,
  3.  There's been some support via AWS (HADOOP-10714),
  4.  There's been a couple of patches from Ted Yu after hbase backups keeled over from too
many threads

One thing that is notable about the s3a (or any of the object store filesystems) is that Jenkins
does not run the tests. Anyone proposing to +1 a patch based on a Jenkins run (see HADOOP-11488)
is going to get a -1 from me; it takes 30-60 minutes for a test run. You get a bill of about
50c/month for participating this project

To date I've been the sole committer running the tests, reviewing the code and with a vague
idea of what's being going on. That's because (a) I care about object stores after my experience
with getting swift://  in, and (b) I'm not recommending that anyone use it in production until
its been field-tested more.

Who is going to assist me review and test these patches?

Perhaps we could also do things like batching findbugs fixes into
fewer JIRAs, as has been suggested before.

A detail. Findbugs is not the problem
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message