kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adunuthula, Seshu" <sadunuth...@ebay.com>
Subject Re: [DISCUSS] about cluster management
Date Tue, 03 Nov 2015 13:39:50 GMT
What about Kubernetes? More heavy weight, but eBay overall is putting all
its eggs into that basket :-)
http://googlecloudplatform.blogspot.com/2015/07/Kubernetes-V1-Released.html


On 11/3/15, 5:26 AM, "Ted Dunning" <ted.dunning@gmail.com> wrote:

>You need to also decide whether Helix has momentum. The low traffic on the
>mailing list may be because the project is mature, but may also mean small
>user base.
>
>That is a very hard call to make.
>
>
>On Tue, Nov 3, 2015 at 2:00 AM, Li Yang <liyang@apache.org> wrote:
>
>> One important goal is the cluster management should be as simple as
>> possible and introduces minimal dependency. This will allow Kylin to be
>> self-contained, easy to deploy, and is able to run on different
>>computing
>> platform. Especially we don't want to be locked-in by any platform.
>>
>> Helix is good as it's just a lightweight 700 KB lib and the only
>>dependency
>> is zookeeper (which everyone depends on nowadays).
>>
>> YARN is great but implies Hadoop, meaning Kylin may have trouble
>>running on
>> Spark alone.
>>
>> Marathon and Mesos is super great but seems a overkill. All Kylin needs
>>at
>> the moment is leader selection and task dispatching.
>>
>>
>> On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <ted.dunning@gmail.com>
>>wrote:
>>
>> > True, but it is one of the few options for running outside of a
>>cluster.
>> >
>> >
>> >
>> > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <qhzhou@apache.org> wrote:
>> >
>> > > Marathon need Mesos as extra dependency. For those who use hdp or
>> > cloudera
>> > > as sandbox for POC or some production env, Mesos does not seem to be
>> the
>> > > standard component. I think it might be a concern for the users.
>> > >
>> > > Luke Han <luke.hq@gmail.com>于2015年11月3日周二 下午1:31写道:
>> > >
>> > > > Very interesting one, Marathon can run on Mesos which also could
>> manage
>> > > > containers...
>> > > > Worth to have a try...
>> > > >
>> > > > https://github.com/mesosphere/marathon
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Best Regards!
>> > > > ---------------------
>> > > >
>> > > > Luke Han
>> > > >
>> > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning
>><ted.dunning@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Try Marathon then.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <luke.hq@gmail.com>
>> wrote:
>> > > > >
>> > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster.
>> > > > > >
>> > > > > > Based on today's practices, Kylin's services always sit
beside
>> > > clusters
>> > > > > not
>> > > > > > inside to have capability to interactive with more than
2
>> clusters
>> > > with
>> > > > > > just one Kylin deployment.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Best Regards!
>> > > > > > ---------------------
>> > > > > >
>> > > > > > Luke Han
>> > > > > >
>> > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra <
>> > > > henry.saputra@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Why dont we just leverage Hadoop YARN?
>> > > > > > >
>> > > > > > > We could use Apache Twill [1] to wrap it as YARN
>>application.
>> > > > > > >
>> > > > > > > Helix is great but one of the limitations it could
not
>> provision
>> > > new
>> > > > > > > node when not already available.
>> > > > > > >
>> > > > > > > [1] http://twill.incubator.apache.org/
>> > > > > > >
>> > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <qhzhou@apache.org>
>>wrote:
>> > > > > > > > Hi, all
>> > > > > > > >      Kylin currently manages the cluster itself,
so far it
>> > works
>> > > > fine
>> > > > > > > since
>> > > > > > > > there are only 2 roles for the cluster, job engine
and
>>query
>> > > > engine.
>> > > > > > > >      In order to reduce the complexity of deployment
and
>> > support
>> > > > the
>> > > > > > > coming
>> > > > > > > > streaming feature, I suggest kylin have a better
cluster
>> > > > management.
>> > > > > > > >      There are several candidates I think can
be
>>leveraged.
>> > > Apache
>> > > > > > Helix
>> > > > > > > > seem suitable for Kylin in my opinion.
>> > > > > > > >      I would love to start a discussion about
cluster
>>manager
>> > of
>> > > > > Kylin,
>> > > > > > > how
>> > > > > > > > we can do it the better way, and which component
we should
>> > > choose.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>


Mime
View raw message