Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MailScanner-NULL-Check: 1298461028.36992@5x2sSncegTd3NQ21xZC26g
Message-ID: <4D5BB6E4.8040403@apache.org>
Date: Wed, 16 Feb 2011 11:37:08 +0000
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: general@hadoop.apache.org
Subject: Re: Hadoop testing project [Was: [VOTE] Abandon mrunit MapReduce
 contrib]
References: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
In-Reply-To: <AANLkTi=EvhSotauym5K080wy0+UORW+W2Nuh4by5JgdZ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 15/02/11 21:58, Konstantin Boudnik wrote:
> While MrUnit discussion draws to its natural conclusion I would like
> to bring up another point which might be well aligned with that
> discussion. Patrick Hunt has brought up this idea earlier today and I
> believe it has to be elaborated further.
>
> A number of testing projects both for Hadoop and Hadoop-related
> component were brought to life over last year or two. Among those are
> MRUnit, PigUnit, YCSB, Herriot, and perhaps a few more. They all
> focusing on more or less the same problem e.g. validation of Hadoop or
> on-top-of-Hadoop components, or application level testing for Hadoop.
> However, the fact that they all are spread across a wide variety of
> projects seems to confuse/mislead Hadoop users.
>
> How about incubating a bigger Hadoop (Pig, Oozie, HBase) testing
> project which will take care about development and support of common
> (where's possible) tools, frameworks and the like? Please feel free to
> share your thoughts :)
> --

I think it would be good though specific projects will need/have their 
own testing needs -I'd expect more focus for testing redistributables to 
be on helping Hadoop users test their stuff against subsets of data, 
rather than the hadoop-*-dev problem of "stressing the hadoop stack once 
your latest patch is applied".

That said, the whole problem of qualifying an OS, Java release and 
cluster is something we'd expect most end user teams to have to do 
-right now terasort is the main stress test.