hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <andrew.purt...@gmail.com>
Subject Re: [VOTE] Release hadoop-2.0.0-alpha
Date Thu, 10 May 2012 01:05:50 GMT
Hi Suresh,

The unstable designation makes sense.  As would one for MiniMRCluster. 

I was over the top initially to surprise. I'm sure the MR minicluster seems a minor detail.


Maybe it's worth thinking about the miniclusters differently? Please pardon if I am rehashing
an old discussion.

Things like MRUnit for applications and BigTop for full cluster tests can help, but for as
mentioned in the below annotation Pig, Hive, HBase, and other parts of the stack use miniclusters
for local end to end testing in unit tests. As the complexity of the stack increases and we
consider cross version support, unit tests on miniclusters I think will have no substitute.


As Hadoop 2 has been evolving there has been some difficulty keeping up with minicluster changes.
This makes sense. The attention to stability to client APIs and such, and the lack thereof
to the minicluster, I think is self evident. But the need to fix up tests unpredictably introduces
some friction that perhaps need not be there. 

Would a JIRA to discuss defining a subset of the minicluster interfaces as more stable be
worthwhile?

Best regards,

    - Andy


On May 9, 2012, at 1:45 PM, Suresh Srinivas <suresh@hortonworks.com> wrote:

> For this reason, in HDFS, we change MiniDFSCluster to LimitedPrivate and
> not treat it as such:
> 
> @InterfaceAudience.LimitedPrivate({"HBase", "HDFS", "Hive", "MapReduce",
> "Pig"})
> @InterfaceStability.Unstable
> public class MiniDFSCluster { ...}
> 
> On Wed, May 9, 2012 at 11:33 AM, Andrew Purtell <apurtell@apache.org> wrote:
> 
>> Sounds good Arun.
>> 
>> How should we consider the suitability and stability of MiniMRCluster
>> for downstream projects?
>> 
>> On Wed, May 9, 2012 at 11:30 AM, Arun C Murthy <acm@hortonworks.com>
>> wrote:
>>> No worries Andy. I can spin an rc1 once we can pin-point the bug.
>>> 
>>> thanks,
>>> Arun
>>> 
>>> On May 9, 2012, at 10:17 AM, Andrew Purtell wrote:
>>> 
>>>> -1 (nonbinding), we are currently facing a minicluster semantic change
>>>> of some kind, or more than one:
>>>> 
>>>>   https://issues.apache.org/jira/browse/HBASE-5966
>>>> 
>>>> There are other HBase JIRAs related to 2.0.0-alpha that we are working
>>>> on, but I'd claim those are all our fault for breaking abstractions to
>>>> solve issues. In one case there's a new helpful 2.x API
>>>> (ShutdownHookManager, thank you!) that we can eventually move to.
>>>> 
>>>> However, the minicluster changes are causing us some repeated
>>>> discomfort. It will break, we'll get some help fixing up our tests for
>>>> that, then some time later it will break again, repeat. Perhaps we
>>>> have no right to complain, the minicluster isn't meant to be used by
>>>> downstream projects. If so then please disregard the complaint, but
>>>> your assistance in helping to fix the breakage again would be much
>>>> appreciated. And, if so, perhaps we can discuss what makes sense in
>>>> terms of a stable minicluster consumable for downstream projects?
>>>> 
>>>> Best regards,
>>>> 
>>>>   - Andy
>>>> 
>>>> On Wed, May 9, 2012 at 9:58 AM, Arun C Murthy <acm@hortonworks.com>
>> wrote:
>>>>> I've created a release candidate for hadoop-2.0.0-alpha that I would
>> like to release.
>>>>> 
>>>>> It is available at:
>> http://people.apache.org/~acmurthy/hadoop-2.0.0-alpha-rc0/
>>>>> 
>>>>> The maven artifacts are available via repository.apache.org.
>>>>> 
>>>>> Please try the release and vote; the vote will run for the usual 7
>> days.
>>>>> 
>>>>> This is a big milestone for the Apache Hadoop community -
>> congratulations and thanks for all the contributions!
>>>>> 
>>>>> thanks,
>>>>> Arun
>>>>> 
>>>>> 
>>>>> --
>>>>> Arun C. Murthy
>>>>> Hortonworks Inc.
>>>>> http://hortonworks.com/
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> 
>>>>   - Andy
>>>> 
>>>> Problems worthy of attack prove their worth by hitting back. - Piet
>>>> Hein (via Tom White)
>>> 
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>> 
>>> 
>> 
>> 
>> 
>> --
>> Best regards,
>> 
>>   - Andy
>> 
>> Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein (via Tom White)
>> 

Mime
View raw message