bigtop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evans Ye <evan...@apache.org>
Subject Re: Rebooting the conversation on the Future of bigtop: Abstracting the backplane ? Containers?
Date Wed, 17 Jun 2015 12:21:39 GMT
I personally like the idea of including mesos, docker stuff as
options we provided to users. No doubt that these solutions can gain more
eyeballs and attract new users/contributors to the bigtop family.
As mentioned above, the use case seems still not clear yet, but there might
be a chance that people will start to adopt them because of bigtop's
support.

Comming back to earth, to be honest I don't need thing like mesos, too. So
including it would be a little bit difficaut since there's no real demand
on it so far. Probably people who is intrested in the technology can start
with an alpha/experimental feature and see if lots of poeple has intrest on
it. The feature should be able to push in to our code base as long as
there's a maintainer who committed to maintain it. I can pair program with
Jay if you'd like to work on it. :)
 2015年6月17日 上午3:02於 "Andrew Purtell" <apurtell@apache.org>寫道:

> > thanks andy - i agree with most of your opinions around continuing to
> build
> standard packages.. but can you clarify what was offensive ?  must be a
> misinterpretation somewhere.
>
> Sure.
>
> A bit offensive.
>
> "gridgain or spark can do what 90% of the hadoop ecosystem already does,
> supporting streams, batch,sql all in one" -> This statement deprecates the
> utility of the labors of rest of the Hadoop ecosystem in favor of Gridgain
> and Spark. As a gross generalization it's unlikely to be a helpful
> statement in any case.
>
> It's fine if we all have our favorites, of course. I think we're set up
> well to empirically determine winners and losers, we don't need to make
> partisan statements. Those components that get some user interest in the
> form of contributions that keep them building and happy in Bigtop will stay
> in. Those that do not get the necessary attention will have to be culled
> out over time when and if they fail to compile or pass integration tests.
>
>
> On Mon, Jun 15, 2015 at 11:42 AM, jay vyas <jayunit100.apache@gmail.com>
> wrote:
>
>> thanks andy - i agree with most of your opinions around continuing to
>> build
>> standard packages.. but can you clarify what was offensive ?  must be a
>> misinterpretation somewhere.
>>
>> 1) To be clear, i am 100% behind supporting standard hadoop build rpms
>> that
>> we have now.   Thats the core product and will be for  the forseeable
>> future, absolutely !
>>
>> 2) The idea (and its just an idea i want to throw out - to keep us on our
>> toes), is that some folks may be interested in hacking around, in a
>> separate branch - on some bleeding edge bigdata deployments - which
>> attempts to incorporate resource managers and  containers as first-class
>> citizens.
>>
>> Again this is all just ideas - not in any way meant to derail the
>> packaging
>> efforts - but rather - just to gauge folks interest level in the bleeding
>> edge, docker, mesos, simplified  processing stacks, and so on.
>>
>>
>>
>> On Mon, Jun 15, 2015 at 12:39 PM, Andrew Purtell <apurtell@apache.org>
>> wrote:
>>
>> > > gridgain or spark can do what 90% of the hadoop ecosystem already
>> does,
>> > supporting streams, batch,sql all in one)
>> >
>> > If something like this becomes the official position of the Bigtop
>> > project, some day, then it will turn off people. I can see where you are
>> > coming from, I think. Correct me if I'm wrong: We have limited
>> bandwidth,
>> > we should move away from Roman et. al.'s vision of Bigtop as an
>> inclusive
>> > distribution of big data packages, and instead become highly opinionated
>> > and tightly focused. If that's accurate, I can sum up my concern as
>> > follows: To the degree we become more opinionated, the less we may have
>> to
>> > look at in terms of inclusion - both software and user communities. For
>> > example, I find the above quoted statement a bit offensive as a
>> participant
>> > on not-Spark and not-Gridgain projects. I roll my eyes sometimes at the
>> > Docker over-hype. Is there still a place for me here?
>> >
>> >
>> >
>> > On Mon, Jun 15, 2015 at 9:22 AM, jay vyas <jayunit100.apache@gmail.com>
>> > wrote:
>> >
>> >> Hi folks.   Every few months, i try to reboot the conversation about
>> the
>> >> next generation of bigtop.
>> >>
>> >> There are 3 things which i think we should consider : A backplane
>> (rather
>> >> than deploy to machines, the meaning of the term "ecosystem" in a
>> >> post-spark in-memory apacolypse, and containerization.
>> >>
>> >> 1) BACKPLANE: The new trend is to have a backplane that provides
>> >> networking abstractions for you (mesos, kubernetes, yarn, and so on).
>>  Is
>> >> it time for us to pick a resource manager?
>> >>
>> >> 2) ECOSYSTEM?: Nowadays folks don't necessarily need the whole hadoop
>> >> ecosystem, and there is a huge shift to in-memory, monolithic stacks
>> >> happening (i.e. gridgain or spark can do what 90% of the hadoop
>> ecosystem
>> >> already does, supporting streams, batch,sql all in one).
>> >>
>> >> 3) CONTAINERS:  we are doing a great job w/ docker in our build infra.
>> >> Is it time to start experimenting with running docker tarballs ?
>> >>
>> >> Combining 1+2+3 - i could see a useful bigdata upstream distro which
>> (1)
>> >> just installed an HCFS implementation (gluster,HDFS,...) along side,
>> say,
>> >> (2) mesos as a backplane for the tooling for [[ hbase + spark + ignite
>> ]]
>> >> --- and then (3) do the integration testing of available
>> mesos-framework
>> >> plugins for ignite and spark underneath.  If other folks are
>> interested,
>> >> maybe we could create the "1x" or "in-memory" branch to start hacking
>> on it
>> >> sometime ?    Maybe even bring the flink guys in as well, as they are
>> >> interested in bigtop packaging.
>> >>
>> >>
>> >>
>> >> --
>> >> jay vyas
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >    - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>> >
>>
>>
>>
>> --
>> jay vyas
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>

Mime
View raw message