asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From abdullah alamoudi <bamou...@gmail.com>
Subject Re: Working with Hadoop
Date Thu, 21 Jul 2016 08:49:10 GMT
I think that list is all we've got. We only support Hadoop 2.x.x.
We found that supporting both 1.x and 2.x has a cost that we couldn't
afford. I believe there are fundamental differences between Hadoop 1.x and
2.x and that a good segment of Hadoop community still use 1.x. However, it
has been a while since 1.x got a new release and so, I am not sure if it is
worth investing time in making it work.

Also, seems to me that our Hadoop support is mainly for attracting existing
users of Hadoop and so, I really think we should not invest in that area
anymore. The only thing that I think we should continue doing is maybe add
more tests (for different formats,etc). That is just my opinion :)

What happened to Hadoop Compatibility Layer? Is that still a thing?

On Thu, Jul 21, 2016 at 5:24 AM, Ian Maxon <imaxon@uci.edu> wrote:

> That's all the ways we use Hadoop at the moment that I can think of as
> well. Maybe the two other minor ones are zookeeper and HDFS backup in
> Managix.
>
> For 1) and 2) it's using Hadoop 2.2.0 right now. In my experimental branch
> for 3) I'm using 2.6.0, it doesn't cause any more issues for me than 2.2.0.
> I believe 1) used to support Hadoop 0.20.0 and other 1.x versions but I'm
> not sure if that works anymore.
>
> On Wed, Jul 20, 2016 at 7:14 PM, Till Westmann <tillw@apache.org> wrote:
>
> > Hi everybody,
> >
> > recently the topic of Hadoop support came up and I realized that my
> > understanding is quite spotty so I’m trying to understand where we are.
> >
> > AFAIK we support
> > 1) HDFS for (potentially indexed) external datasets,
> > 2) YARN as a resource manager, and
> > 3) HDFS as a basis for internal storage.
> > Is this list complete or do we have other Hadoop touchpoints?
> >
> > I believe that 1) and 2) should be reasonable stable and that 3) is still
> > in
> > the works. Is that correct?
> >
> > Further I'm wondering
> > a) which versions of Hadoop we support and
> > b) which ones we should support for all the cases.
> > Please chime in on this as well.
> >
> > Any other things that anybody working with AsterixDB and Hadoop should be
> > aware
> > of?
> >
> > Thanks!
> > Till
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message