hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Yang <ey...@yahoo-inc.com>
Subject Re: Hadoop testing project
Date Fri, 18 Feb 2011 17:49:27 GMT

On 2/18/11 8:24 AM, "Steve Loughran" <stevel@apache.org> wrote:

> On 17/02/2011 18:36, Eric Yang wrote:
>> I think the bigger concern is that Hadoop ecosystem does not have a standard
>> method in linking dependencies.  Hbase depends on Zookeeper, and Pig depends
>> on Hadoop and Hbase.  Then pig decided to put hadoop-core jar in it's own
>> jar file.  Chukwa depends on pig + hbase + hadoop and zookeeper.  The
>> version incompatibility is probably what driving people nuts.  Hence, there
>> is a new proposal on how to integrate among hadoop ecosystem.  I urge
>> project owners to review the proposal and provide feedbacks.
>> The proposal is located at:
>> https://issues.apache.org/jira/secure/attachment/12470823/deployment.pdf
>> The related jiras are:
>> https://issues.apache.org/jira/browse/HADOOP-6255
>> https://issues.apache.org/jira/browse/PIG-1857
>> There are plans to file more jiras for related projects.
> I think a focus on RPMs/debs and then say "local VM for testing",
> because that gets people used to deploying on Linux from day one, rather
> than developing on windows and being surprised that things work
> differently in production. You still have the problem of debugging
> remotely to a VM in the VLAN, but that's tractable.

Packaging, VM testing, scale, then deployment automation.  My goal is
"deployment automation", and this requires polishing each steps along the
way.  For hadoop testing project, it depends on the type of testing that you
are focus on.  It will be easier to understand the scope and apply

>> The integration
>> would also be a lot easier if all related projects are using maven for
>> dependency management.
> Do you mean Maven the tool or the maven repo format?

I meant both, it is a little easier to manage dependencies in maven build
tools than ivy, and deploying jar files to maven repository is also an
essential mean for integrate project more closely.


View raw message