Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of todd@cloudera.com designates
 209.85.214.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <320D8CFB-73A9-41AF-850A-F6ED5C1255D9@gmail.com>
References: <4C4776F7-94C7-453E-ACC8-E4663186FED0@hortonworks.com>
 <CA+RK=_BOQ4hyPTu+cF2Ls8eoLhuGiRG7trwm3GADNO1Hg3uq0A@mail.gmail.com>
 <8BB5D1E9-0C05-41B7-ADBF-D60A8811DEB9@hortonworks.com>
 <CA+RK=_CO9qFsp9cjNeUmH0PGPkNY_muGQfT0AebE8cc40+r4Ww@mail.gmail.com>
 <CADdVdVGZirnXBDpjGd+2dgBfObVXCv1s9AwRsnnbEVYVF7a_Lg@mail.gmail.com>
 <320D8CFB-73A9-41AF-850A-F6ED5C1255D9@gmail.com>
From: Todd Lipcon <todd@cloudera.com>
Date: Wed, 9 May 2012 22:00:12 -0700
Message-ID: 
 <CADY20s7NzKOiTOQxh7aiJd7no2VTdcA9JrFS64+HBBni7CYV6w@mail.gmail.com>
Subject: Re: [VOTE] Release hadoop-2.0.0-alpha
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Andrew,

Have you seen the new MiniMRClientCluster class? It's meant to be what
you describe - a minicluster which only exposes "external" APIs --
most importantly a way of getting at a JobClient to submit jobs. We
have it implemented in both 1.x and 2.x at this point, though I don't
recall if it's in the 1.0.x releases or if it's only slated for 1.1+

-Todd

On Wed, May 9, 2012 at 6:05 PM, Andrew Purtell <andrew.purtell@gmail.com> w=
rote:
> Hi Suresh,
>
> The unstable designation makes sense. =A0As would one for MiniMRCluster.
>
> I was over the top initially to surprise. I'm sure the MR minicluster see=
ms a minor detail.
>
> Maybe it's worth thinking about the miniclusters differently? Please pard=
on if I am rehashing an old discussion.
>
> Things like MRUnit for applications and BigTop for full cluster tests can=
 help, but for as mentioned in the below annotation Pig, Hive, HBase, and o=
ther parts of the stack use miniclusters for local end to end testing in un=
it tests. As the complexity of the stack increases and we consider cross ve=
rsion support, unit tests on miniclusters I think will have no substitute.
>
> As Hadoop 2 has been evolving there has been some difficulty keeping up w=
ith minicluster changes. This makes sense. The attention to stability to cl=
ient APIs and such, and the lack thereof to the minicluster, I think is sel=
f evident. But the need to fix up tests unpredictably introduces some frict=
ion that perhaps need not be there.
>
> Would a JIRA to discuss defining a subset of the minicluster interfaces a=
s more stable be worthwhile?
>
> Best regards,
>
> =A0 =A0- Andy
>
>
> On May 9, 2012, at 1:45 PM, Suresh Srinivas <suresh@hortonworks.com> wrot=
e:
>
>> For this reason, in HDFS, we change MiniDFSCluster to LimitedPrivate and
>> not treat it as such:
>>
>> @InterfaceAudience.LimitedPrivate({"HBase", "HDFS", "Hive", "MapReduce",
>> "Pig"})
>> @InterfaceStability.Unstable
>> public class MiniDFSCluster { ...}
>>
>> On Wed, May 9, 2012 at 11:33 AM, Andrew Purtell <apurtell@apache.org> wr=
ote:
>>
>>> Sounds good Arun.
>>>
>>> How should we consider the suitability and stability of MiniMRCluster
>>> for downstream projects?
>>>
>>> On Wed, May 9, 2012 at 11:30 AM, Arun C Murthy <acm@hortonworks.com>
>>> wrote:
>>>> No worries Andy. I can spin an rc1 once we can pin-point the bug.
>>>>
>>>> thanks,
>>>> Arun
>>>>
>>>> On May 9, 2012, at 10:17 AM, Andrew Purtell wrote:
>>>>
>>>>> -1 (nonbinding), we are currently facing a minicluster semantic chang=
e
>>>>> of some kind, or more than one:
>>>>>
>>>>> =A0 https://issues.apache.org/jira/browse/HBASE-5966
>>>>>
>>>>> There are other HBase JIRAs related to 2.0.0-alpha that we are workin=
g
>>>>> on, but I'd claim those are all our fault for breaking abstractions t=
o
>>>>> solve issues. In one case there's a new helpful 2.x API
>>>>> (ShutdownHookManager, thank you!) that we can eventually move to.
>>>>>
>>>>> However, the minicluster changes are causing us some repeated
>>>>> discomfort. It will break, we'll get some help fixing up our tests fo=
r
>>>>> that, then some time later it will break again, repeat. Perhaps we
>>>>> have no right to complain, the minicluster isn't meant to be used by
>>>>> downstream projects. If so then please disregard the complaint, but
>>>>> your assistance in helping to fix the breakage again would be much
>>>>> appreciated. And, if so, perhaps we can discuss what makes sense in
>>>>> terms of a stable minicluster consumable for downstream projects?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> =A0 - Andy
>>>>>
>>>>> On Wed, May 9, 2012 at 9:58 AM, Arun C Murthy <acm@hortonworks.com>
>>> wrote:
>>>>>> I've created a release candidate for hadoop-2.0.0-alpha that I would
>>> like to release.
>>>>>>
>>>>>> It is available at:
>>> http://people.apache.org/~acmurthy/hadoop-2.0.0-alpha-rc0/
>>>>>>
>>>>>> The maven artifacts are available via repository.apache.org.
>>>>>>
>>>>>> Please try the release and vote; the vote will run for the usual 7
>>> days.
>>>>>>
>>>>>> This is a big milestone for the Apache Hadoop community -
>>> congratulations and thanks for all the contributions!
>>>>>>
>>>>>> thanks,
>>>>>> Arun
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Arun C. Murthy
>>>>>> Hortonworks Inc.
>>>>>> http://hortonworks.com/
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> =A0 - Andy
>>>>>
>>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>>> Hein (via Tom White)
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> =A0 - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>> Hein (via Tom White)
>>>


--=20
Todd Lipcon
Software Engineer, Cloudera