asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <>
Subject Re: Working with Hadoop
Date Thu, 21 Jul 2016 18:10:39 GMT
PS - I think the world has moved beyond MapReduce, so the Hadoop 
Compatibility Layer is no longer of interest. :-)

On 7/21/16 1:49 AM, abdullah alamoudi wrote:
> I think that list is all we've got. We only support Hadoop 2.x.x.
> We found that supporting both 1.x and 2.x has a cost that we couldn't
> afford. I believe there are fundamental differences between Hadoop 1.x and
> 2.x and that a good segment of Hadoop community still use 1.x. However, it
> has been a while since 1.x got a new release and so, I am not sure if it is
> worth investing time in making it work.
> Also, seems to me that our Hadoop support is mainly for attracting existing
> users of Hadoop and so, I really think we should not invest in that area
> anymore. The only thing that I think we should continue doing is maybe add
> more tests (for different formats,etc). That is just my opinion :)
> What happened to Hadoop Compatibility Layer? Is that still a thing?
> On Thu, Jul 21, 2016 at 5:24 AM, Ian Maxon <> wrote:
>> That's all the ways we use Hadoop at the moment that I can think of as
>> well. Maybe the two other minor ones are zookeeper and HDFS backup in
>> Managix.
>> For 1) and 2) it's using Hadoop 2.2.0 right now. In my experimental branch
>> for 3) I'm using 2.6.0, it doesn't cause any more issues for me than 2.2.0.
>> I believe 1) used to support Hadoop 0.20.0 and other 1.x versions but I'm
>> not sure if that works anymore.
>> On Wed, Jul 20, 2016 at 7:14 PM, Till Westmann <> wrote:
>>> Hi everybody,
>>> recently the topic of Hadoop support came up and I realized that my
>>> understanding is quite spotty so I’m trying to understand where we are.
>>> AFAIK we support
>>> 1) HDFS for (potentially indexed) external datasets,
>>> 2) YARN as a resource manager, and
>>> 3) HDFS as a basis for internal storage.
>>> Is this list complete or do we have other Hadoop touchpoints?
>>> I believe that 1) and 2) should be reasonable stable and that 3) is still
>>> in
>>> the works. Is that correct?
>>> Further I'm wondering
>>> a) which versions of Hadoop we support and
>>> b) which ones we should support for all the cases.
>>> Please chime in on this as well.
>>> Any other things that anybody working with AsterixDB and Hadoop should be
>>> aware
>>> of?
>>> Thanks!
>>> Till

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message