Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
Sender: cos@boudnik.org
In-Reply-To: <AANLkTinQ9iVjdjYr7eeLbXHts4+sFNA1-h4hY7KDw+wk@mail.gmail.com>
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
 <AANLkTinQ9iVjdjYr7eeLbXHts4+sFNA1-h4hY7KDw+wk@mail.gmail.com>
From: Konstantin Boudnik <cos@apache.org>
Date: Tue, 15 Feb 2011 15:54:35 -0800
Message-ID: <AANLkTi=RmQKWiG5fA+CSmF3p3Xn-S02zCT+Zhd1T2dgP@mail.gmail.com>
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 15, 2011 at 14:15, Eric Sammer <esammer@cloudera.com> wrote:
> I think this is a good idea. The only thing I think is that it may make
> sense to split such an effort into two components: one for the testing of
> Hadoop and the projects themselves and one to test end user applications =
and

I expect to see even greater number of component, to be honest. E.g. a
harness to run stacks testing (which as has been discussed with HBase
folks might utilize YCSB artifacts). Which doesn't invalidate the
purpose of central Hadoop testing project or whatever we might call
it.

> libraries. Performance testing tools like YCSB are probably more in the
> former camp where mrunit is the latter, as a for instance. I think it's
> important to have separate artifacts to minimize uber-jar issues (or
> contrib-like situations where release cycles are coupled).

Having separate artifacts/release cycles would be pretty important for
another reason too: test artifacts might undergo significant changes
between releases of a product. Thus requiring using different versions
of such validating artifacts for differently composed Hadoop stacks.
Uber-jar are proven to be inflexible and pain to deal with.

Cos

> On Tue, Feb 15, 2011 at 4:58 PM, Konstantin Boudnik <cos@apache.org> wrot=
e:
>
>> While MrUnit discussion draws to its natural conclusion I would like
>> to bring up another point which might be well aligned with that
>> discussion. Patrick Hunt has brought up this idea earlier today and I
>> believe it has to be elaborated further.
>>
>> A number of testing projects both for Hadoop and Hadoop-related
>> component were brought to life over last year or two. Among those are
>> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
>> focusing on more or less the same problem e.g. validation of Hadoop or
>> on-top-of-Hadoop components, or application level testing for Hadoop.
>> However, the fact that they all are spread across a wide variety of
>> projects seems to confuse/mislead Hadoop users.
>>
>> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
>> project which will take care about development and support of common
>> (where's possible) tools, frameworks and the like? Please feel free to
>> share your thoughts :)
>> --
>> =A0Take care,
>> Konstantin (Cos) Boudnik
>>
>>
>> On Tue, Feb 15, 2011 at 10:44, Eric Sammer <esammer@cloudera.com> wrote:
>> > I've started the wiki page proposal for Incubator for mrunit. I'll pin=
g
>> > people off list for mentoring. Much appreciated for all the help!
>> >
>> > On Tue, Feb 15, 2011 at 1:36 PM, Nigel Daley <ndaley@mac.com> wrote:
>> >
>> >> I'm happy to help mentor as well.
>> >>
>> >> Cheers,
>> >> Nige
>> >>
>> >> On Feb 11, 2011, at 11:52 AM, Patrick Hunt wrote:
>> >>
>> >> > On Fri, Feb 11, 2011 at 9:44 AM, Mattmann, Chris A (388J)
>> >> > <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >> >> Guys, BTW, if you need help or a mentor in Apache Incubator-ville =
for
>> >> MRUnit, I would be happy to help.
>> >> >
>> >> > I was going to suggest the same thing (mrunit to incubator). I woul=
d
>> >> > also be happy to be a mentor.
>> >> >
>> >> > Patrick
>>
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>