kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: [DISCUSS] about cluster management
Date Tue, 03 Nov 2015 13:26:58 GMT
You need to also decide whether Helix has momentum. The low traffic on the
mailing list may be because the project is mature, but may also mean small
user base.

That is a very hard call to make.


On Tue, Nov 3, 2015 at 2:00 AM, Li Yang <liyang@apache.org> wrote:

> One important goal is the cluster management should be as simple as
> possible and introduces minimal dependency. This will allow Kylin to be
> self-contained, easy to deploy, and is able to run on different computing
> platform. Especially we don't want to be locked-in by any platform.
>
> Helix is good as it's just a lightweight 700 KB lib and the only dependency
> is zookeeper (which everyone depends on nowadays).
>
> YARN is great but implies Hadoop, meaning Kylin may have trouble running on
> Spark alone.
>
> Marathon and Mesos is super great but seems a overkill. All Kylin needs at
> the moment is leader selection and task dispatching.
>
>
> On Tue, Nov 3, 2015 at 3:03 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>
> > True, but it is one of the few options for running outside of a cluster.
> >
> >
> >
> > On Mon, Nov 2, 2015 at 10:43 PM, 周千昊 <qhzhou@apache.org> wrote:
> >
> > > Marathon need Mesos as extra dependency. For those who use hdp or
> > cloudera
> > > as sandbox for POC or some production env, Mesos does not seem to be
> the
> > > standard component. I think it might be a concern for the users.
> > >
> > > Luke Han <luke.hq@gmail.com>于2015年11月3日周二 下午1:31写道:
> > >
> > > > Very interesting one, Marathon can run on Mesos which also could
> manage
> > > > containers...
> > > > Worth to have a try...
> > > >
> > > > https://github.com/mesosphere/marathon
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards!
> > > > ---------------------
> > > >
> > > > Luke Han
> > > >
> > > > On Tue, Nov 3, 2015 at 1:12 PM, Ted Dunning <ted.dunning@gmail.com>
> > > wrote:
> > > >
> > > > > Try Marathon then.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 2, 2015 at 5:06 PM, Luke Han <luke.hq@gmail.com>
> wrote:
> > > > >
> > > > > > Run on YARN requires to run kylin on "one" Hadoop cluster.
> > > > > >
> > > > > > Based on today's practices, Kylin's services always sit beside
> > > clusters
> > > > > not
> > > > > > inside to have capability to interactive with more than 2
> clusters
> > > with
> > > > > > just one Kylin deployment.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best Regards!
> > > > > > ---------------------
> > > > > >
> > > > > > Luke Han
> > > > > >
> > > > > > On Tue, Nov 3, 2015 at 12:12 AM, Henry Saputra <
> > > > henry.saputra@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Why dont we just leverage Hadoop YARN?
> > > > > > >
> > > > > > > We could use Apache Twill [1] to wrap it as YARN application.
> > > > > > >
> > > > > > > Helix is great but one of the limitations it could not
> provision
> > > new
> > > > > > > node when not already available.
> > > > > > >
> > > > > > > [1] http://twill.incubator.apache.org/
> > > > > > >
> > > > > > > On Sun, Nov 1, 2015 at 7:01 PM, 周千昊 <qhzhou@apache.org>
wrote:
> > > > > > > > Hi, all
> > > > > > > >      Kylin currently manages the cluster itself, so
far it
> > works
> > > > fine
> > > > > > > since
> > > > > > > > there are only 2 roles for the cluster, job engine
and query
> > > > engine.
> > > > > > > >      In order to reduce the complexity of deployment
and
> > support
> > > > the
> > > > > > > coming
> > > > > > > > streaming feature, I suggest kylin have a better cluster
> > > > management.
> > > > > > > >      There are several candidates I think can be leveraged.
> > > Apache
> > > > > > Helix
> > > > > > > > seem suitable for Kylin in my opinion.
> > > > > > > >      I would love to start a discussion about cluster
manager
> > of
> > > > > Kylin,
> > > > > > > how
> > > > > > > > we can do it the better way, and which component we
should
> > > choose.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message