hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: Patch review process
Date Tue, 27 Jan 2015 22:59:27 GMT
> Andrew, can the community build on your distributed pre-commit work to make
> it production ready?
I'm happy to share it if someone is willing to take it across the finish
line. I think it'd be about two weeks of work full-time. I've already
cleaned it up some, but it still requires checking out a couple forked
repos, and it's strung together with a set of bash scripts.

I see the biggest challenges as the following though:
- Need to write a faster Swarming client. Running the unit tests just takes
5 mins, but using the Python client to package up all the dependencies
takes 15-20 minutes.
- It needs to be deployed on Apache infra, integrated with Jenkins JUnit
reporting / artifact archiving, and then hooked up to the precommit
scripts. I also worry about the ongoing maintenance cost. Brock set up
something like this for Hive, and he had a hard time with flaky EC2

One additional upside though is this same framework should work for other
mavenized projects too. There's nothing Hadoop specific in it right now, so
HBase, Solr, etc could leverage it too.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message