kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adunuthula, Seshu" <sadunuth...@ebay.com>
Subject Re: [DISCUSS] about cluster management
Date Tue, 03 Nov 2015 13:54:42 GMT
Lightweight cluster management may be ideal in terms of ease of use, but
if it doesn’t fit within the overall cluster/resource management
infrastructure of where Kylin is deployed, you would end up in creating
small islands of partitioned resources…

My personal preference would be (1)Mesos/Marathon which has larger
installed base (2)Kubernetes based on the momentum behind it and should be
optional for small scale deployments...



On 11/3/15, 5:39 AM, "Adunuthula, Seshu" <sadunuthula@ebay.com> wrote:

>What about Kubernetes? More heavy weight, but eBay overall is putting all
>its eggs into that basket :-)
>http://googlecloudplatform.blogspot.com/2015/07/Kubernetes-V1-Released.htm
>l
>
>
>On 11/3/15, 5:26 AM, "Ted Dunning" <ted.dunning@gmail.com> wrote:
>
>>You need to also decide whether Helix has momentum. The low traffic on
>>the
>>mailing list may be because the project is mature, but may also mean
>>small
>>user base.
>>
>>That is a very hard call to make.
>>
>>
>>On Tue, Nov 3, 2015 at 2:00 AM, Li Yang <liyang@apache.org> wrote:
>>
>>> One important goal is the cluster management should be as simple as
>>> possible and introduces minimal dependency. This will allow Kylin to be
>>> self-contained, easy to deploy, and is able to run on different
>>>computing
>>> platform. Especially we don't want to be locked-in by any platform.
>>>
>>> Helix is good as it's just a lightweight 700 KB lib and the only
>>>dependency
>>> is zookeeper (which everyone depends on nowadays).
>>>
>>> YARN is great but implies Hadoop, meaning Kylin may have trouble
>>>running on
>>> Spark alone.
>>>
>>> Marathon and Mesos is super great but seems a overkill. All Kylin needs
>>>at
>>> the moment is leader selection and task dispatching.
>>>
>>>
>>> On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <ted.dunning@gmail.com>
>>>wrote:
>>>
>>> > True, but it is one of the few options for running outside of a
>>>cluster.
>>> >
>>> >
>>> >
>>> > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <qhzhou@apache.org> wrote:
>>> >
>>> > > Marathon need Mesos as extra dependency. For those who use hdp or
>>> > cloudera
>>> > > as sandbox for POC or some production env, Mesos does not seem to
>>>be
>>> the
>>> > > standard component. I think it might be a concern for the users.
>>> > >
>>> > > Luke Han <luke.hq@gmail.com>于2015年11月3日周二 下午1:31写道:
>>> > >
>>> > > > Very interesting one, Marathon can run on Mesos which also could
>>> manage
>>> > > > containers...
>>> > > > Worth to have a try...
>>> > > >
>>> > > > https://github.com/mesosphere/marathon
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > Best Regards!
>>> > > > ---------------------
>>> > > >
>>> > > > Luke Han
>>> > > >
>>> > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning
>>><ted.dunning@gmail.com>
>>> > > wrote:
>>> > > >
>>> > > > > Try Marathon then.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <luke.hq@gmail.com>
>>> wrote:
>>> > > > >
>>> > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster.
>>> > > > > >
>>> > > > > > Based on today's practices, Kylin's services always
sit
>>>beside
>>> > > clusters
>>> > > > > not
>>> > > > > > inside to have capability to interactive with more than
2
>>> clusters
>>> > > with
>>> > > > > > just one Kylin deployment.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Best Regards!
>>> > > > > > ---------------------
>>> > > > > >
>>> > > > > > Luke Han
>>> > > > > >
>>> > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra <
>>> > > > henry.saputra@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Why dont we just leverage Hadoop YARN?
>>> > > > > > >
>>> > > > > > > We could use Apache Twill [1] to wrap it as YARN
>>>application.
>>> > > > > > >
>>> > > > > > > Helix is great but one of the limitations it could
not
>>> provision
>>> > > new
>>> > > > > > > node when not already available.
>>> > > > > > >
>>> > > > > > > [1] http://twill.incubator.apache.org/
>>> > > > > > >
>>> > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <qhzhou@apache.org>
>>>wrote:
>>> > > > > > > > Hi, all
>>> > > > > > > >      Kylin currently manages the cluster itself,
so far
>>>it
>>> > works
>>> > > > fine
>>> > > > > > > since
>>> > > > > > > > there are only 2 roles for the cluster, job
engine and
>>>query
>>> > > > engine.
>>> > > > > > > >      In order to reduce the complexity of
deployment and
>>> > support
>>> > > > the
>>> > > > > > > coming
>>> > > > > > > > streaming feature, I suggest kylin have a
better cluster
>>> > > > management.
>>> > > > > > > >      There are several candidates I think
can be
>>>leveraged.
>>> > > Apache
>>> > > > > > Helix
>>> > > > > > > > seem suitable for Kylin in my opinion.
>>> > > > > > > >      I would love to start a discussion about
cluster
>>>manager
>>> > of
>>> > > > > Kylin,
>>> > > > > > > how
>>> > > > > > > > we can do it the better way, and which component
we
>>>should
>>> > > choose.
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>


Mime
View raw message