From general-return-5078-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu May 10 04:54:33 2012 Return-Path: X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DA73AC917 for ; Thu, 10 May 2012 04:54:33 +0000 (UTC) Received: (qmail 77961 invoked by uid 500); 10 May 2012 04:54:28 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 77659 invoked by uid 500); 10 May 2012 04:54:27 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Delivered-To: moderator for general@hadoop.apache.org Received: (qmail 56343 invoked by uid 99); 10 May 2012 01:06:18 -0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andrew.purtell@gmail.com designates 209.85.214.176 as permitted sender) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to; bh=Wz8t2rloDxvTM8uv15xnHzAhYuMsJMVyxc4Dxi6v0/c=; b=J6ux8/NzevFqJVDi1h7jSQ5RDldo4k28GiidnClcHtAqQl30G9CYB7+T0iL+Q7FBph 1cWUjzFKEQhno+u68+iS2QYJZncOZpQBB4AvalVgHg26fXTqnKZnu4x5/088MLvIHN5u WlWAHJFp3s2C8RkuCGrKbnryQs7NdWTxA3yUnqZ5zkxheJFjO31TgCSa9jADqm+DHpaA oeVelFp4ZwjJfRqG/JGxhPacKv40UGip5MEH/9/sUFiU2qBynziB1IjePji0cHZVHHw6 Ytm0qe65QLruykL3RT/YCwSCRirqSEmnr1dSYZB2jRlBcG/JwR6qhvBZnRRSkc+GUF/F VE2g== References: <4C4776F7-94C7-453E-ACC8-E4663186FED0@hortonworks.com> <8BB5D1E9-0C05-41B7-ADBF-D60A8811DEB9@hortonworks.com> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <320D8CFB-73A9-41AF-850A-F6ED5C1255D9@gmail.com> Cc: "general@hadoop.apache.org" X-Mailer: iPhone Mail (9B206) From: Andrew Purtell Subject: Re: [VOTE] Release hadoop-2.0.0-alpha Date: Wed, 9 May 2012 18:05:50 -0700 To: "general@hadoop.apache.org" X-Virus-Checked: Checked by ClamAV on apache.org Hi Suresh, The unstable designation makes sense. As would one for MiniMRCluster.=20 I was over the top initially to surprise. I'm sure the MR minicluster seems a= minor detail.=20 Maybe it's worth thinking about the miniclusters differently? Please pardon i= f I am rehashing an old discussion. Things like MRUnit for applications and BigTop for full cluster tests can he= lp, but for as mentioned in the below annotation Pig, Hive, HBase, and other= parts of the stack use miniclusters for local end to end testing in unit te= sts. As the complexity of the stack increases and we consider cross version s= upport, unit tests on miniclusters I think will have no substitute.=20 As Hadoop 2 has been evolving there has been some difficulty keeping up with= minicluster changes. This makes sense. The attention to stability to client= APIs and such, and the lack thereof to the minicluster, I think is self evi= dent. But the need to fix up tests unpredictably introduces some friction th= at perhaps need not be there.=20 Would a JIRA to discuss defining a subset of the minicluster interfaces as m= ore stable be worthwhile? Best regards, - Andy On May 9, 2012, at 1:45 PM, Suresh Srinivas wrote: > For this reason, in HDFS, we change MiniDFSCluster to LimitedPrivate and > not treat it as such: >=20 > @InterfaceAudience.LimitedPrivate({"HBase", "HDFS", "Hive", "MapReduce", > "Pig"}) > @InterfaceStability.Unstable > public class MiniDFSCluster { ...} >=20 > On Wed, May 9, 2012 at 11:33 AM, Andrew Purtell wrot= e: >=20 >> Sounds good Arun. >>=20 >> How should we consider the suitability and stability of MiniMRCluster >> for downstream projects? >>=20 >> On Wed, May 9, 2012 at 11:30 AM, Arun C Murthy >> wrote: >>> No worries Andy. I can spin an rc1 once we can pin-point the bug. >>>=20 >>> thanks, >>> Arun >>>=20 >>> On May 9, 2012, at 10:17 AM, Andrew Purtell wrote: >>>=20 >>>> -1 (nonbinding), we are currently facing a minicluster semantic change >>>> of some kind, or more than one: >>>>=20 >>>> https://issues.apache.org/jira/browse/HBASE-5966 >>>>=20 >>>> There are other HBase JIRAs related to 2.0.0-alpha that we are working >>>> on, but I'd claim those are all our fault for breaking abstractions to >>>> solve issues. In one case there's a new helpful 2.x API >>>> (ShutdownHookManager, thank you!) that we can eventually move to. >>>>=20 >>>> However, the minicluster changes are causing us some repeated >>>> discomfort. It will break, we'll get some help fixing up our tests for >>>> that, then some time later it will break again, repeat. Perhaps we >>>> have no right to complain, the minicluster isn't meant to be used by >>>> downstream projects. If so then please disregard the complaint, but >>>> your assistance in helping to fix the breakage again would be much >>>> appreciated. And, if so, perhaps we can discuss what makes sense in >>>> terms of a stable minicluster consumable for downstream projects? >>>>=20 >>>> Best regards, >>>>=20 >>>> - Andy >>>>=20 >>>> On Wed, May 9, 2012 at 9:58 AM, Arun C Murthy >> wrote: >>>>> I've created a release candidate for hadoop-2.0.0-alpha that I would >> like to release. >>>>>=20 >>>>> It is available at: >> http://people.apache.org/~acmurthy/hadoop-2.0.0-alpha-rc0/ >>>>>=20 >>>>> The maven artifacts are available via repository.apache.org. >>>>>=20 >>>>> Please try the release and vote; the vote will run for the usual 7 >> days. >>>>>=20 >>>>> This is a big milestone for the Apache Hadoop community - >> congratulations and thanks for all the contributions! >>>>>=20 >>>>> thanks, >>>>> Arun >>>>>=20 >>>>>=20 >>>>> -- >>>>> Arun C. Murthy >>>>> Hortonworks Inc. >>>>> http://hortonworks.com/ >>>>>=20 >>>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> -- >>>> Best regards, >>>>=20 >>>> - Andy >>>>=20 >>>> Problems worthy of attack prove their worth by hitting back. - Piet >>>> Hein (via Tom White) >>>=20 >>> -- >>> Arun C. Murthy >>> Hortonworks Inc. >>> http://hortonworks.com/ >>>=20 >>>=20 >>=20 >>=20 >>=20 >> -- >> Best regards, >>=20 >> - Andy >>=20 >> Problems worthy of attack prove their worth by hitting back. - Piet >> Hein (via Tom White) >>=20