kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Yang <liy...@apache.org>
Subject Re: Kylin for MapR
Date Mon, 17 Jul 2017 20:03:27 GMT
I knew some commercial version of Kylin could store metadata and cube
outside of HBase.

On Tue, Jul 11, 2017 at 5:19 AM, Nirav Patel <npatel@xactlycorp.com> wrote:

> "Please also note that Kylin's metadata is persisted to HBase by default,
> you need have another implementation for ResourceStore."
>
> I found almost 600 matches for string "hbase" in modules other then
> hbase-storage. Hbase seems tightly integrated into modules where it should
> not be. Even if it is used for metadata it should been pluggable and not
> directly called from other modules. I can't even run DebugTomcat.java
> because it relies on classes like AclHBaseStorage (part of server-base) and
> calls methods of Hbase storage directly.  I think we should make Hbase
> dependency injection little bit more pluggable then what it is now. Or use
> embedded database like derby or rdbms like mysql which are not so costly to
> deploy like hbase.
>
> Thanks,
> Nirav
>
> On Mon, Jul 10, 2017 at 11:03 AM, Nirav Patel <npatel@xactlycorp.com>
> wrote:
>
> > No equivalent. Instead they came up with Drill - MaprDB connector for
> > performance and flexibility.  So like earlier version of Kylin I should
> be
> > able to make drill calls but still do filtering and aggregation on
> cluster.
> > Only thing is we need to run separate drill-bit service under Yarn for
> > this to work.
> >
> > Thanks,
> > Nirav
> >
> > On Sun, Jul 9, 2017 at 5:06 PM, ShaoFeng Shi <shaofengshi@apache.org>
> > wrote:
> >
> >> Is there some coprocessor-equivalent mechanism in maprdb? If yes, you
> can
> >> try to implement the logic into it. Otherwise, the performance might be
> >> unacceptable when there is bunch of data in each region.
> >>
> >> I rememer in very early version, Kylin doesn't have coprocessor, all
> data
> >> be fetched to Kylin server and then do filtering and aggregation by
> >> Calcite. Later coprocessor was added and improved continiously. Now
> there
> >> is no option to disable it. You have to modify the code if don't want to
> >> trigger it.
> >>
> >>
> >>
> >> 2017-07-09 1:34 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:
> >>
> >> >
> >> > Thanks Shao for some pointers. I would like some more advice on how I
> >> can
> >> > replace maprdb as datasource.
> >> >
> >> > Currently, you can  create cubes on maprdb using hbase-storage module
> as
> >> > maprdb does support hbase client/admin APIs. Only problem is while
> >> querying
> >> > it throws an error due to Lack of support of Co-processor.
> >> >
> >> > My initial goal is to get past coprocessor error and return query
> >> result.
> >> > Then find a better way to implement that strategy - may be a separate
> >> > maprdb-storage module. Then look into how to improve performance
> without
> >> > coprocessors. Any advice on these approaches ?
> >> >
> >> > Thanks,
> >> > Nirav
> >> >
> >> > On Sat, Jul 8, 2017 at 7:54 AM, ShaoFeng Shi <shaofengshi@apache.org>
> >> > wrote:
> >> >
> >> >> The core modules like core-cube, core-storage are totally independent
> >> of
> >> >> HBase; while some others like engine-mr, engine-spark has
> dependencies
> >> on
> >> >> HBase. If you want to replace it, you need implement new cubing
> engine
> >> as
> >> >> well. Please also note that Kylin's metadata is persisted to HBase
by
> >> >> default, you need have another implementation for ResourceStore.
> >> >>
> >> >> In a short, the plug-in architecture works (we have verfied that),
> >> while
> >> >> changing the storage is a complex task which take some time to be
> >> function
> >> >> complete and performance stable.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 2017-07-08 5:19 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:
> >> >>
> >> >>> Yes, MapR-DB doesn't support coprocessors.
> >> >>>
> >> >>> Here's the thing though - based on kylin plugin architecture it
> >> >>> shouldn't be a problem ideally. Aggregation as well as other DML/DDL
> >> >>> operation on datasources should be done transparently. i.e. using
> >> >>> kylin-hbase adapter written with calcite or something. It's upto
the
> >> writer
> >> >>> of those adapters to implement aggregates however they want. i.e.
> >> either
> >> >>> using coprocessors, or in-memory on application server side or
using
> >> spark.
> >> >>> http://kylin.apache.org/development/plugin_arch.html
> >> >>>
> >> >>> I think for mapr-db we can leverage mapr-drill with secondary
> indexes
> >> >>> for faster filtering and drill does parallel aggregation as well.
> >> Other
> >> >>> option is to use mapr-spark with mapr-db which can does the same.
> >> >>>
> >> >>> Do you know how tightly hbase is coupled with other modules of
kylin
> >> >>> source other then hbase-storage.
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Thu, Jul 6, 2017 at 6:53 PM, ShaoFeng Shi <
> shaofengshi@apache.org>
> >> >>> wrote:
> >> >>>
> >> >>>> Hi Nirav,
> >> >>>>
> >> >>>> I googled that "HBase coprocessors are not present in MapR
DB", is
> >> >>>> this true? You know Kylin relies on HBase coprocessor to do
> filering
> >> and
> >> >>>> aggregation in each region local; If coprocessor is not available,
> >> the
> >> >>>> performance will be a problem.
> >> >>>>
> >> >>>> 2017-07-07 1:29 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:
> >> >>>>
> >> >>>>> Hi, We are a mapr users. You will need to deploy separate
Hbase
> >> >>>>> Cluster alongside your mapr Cluster or on top of it. You
won't be
> >> able to
> >> >>>>> use Mapr-DB with Kylin.
> >> >>>>> I am looking into writing Kyling - MaprDB adapter.
> >> >>>>>
> >> >>>>> On Tue, Jul 4, 2017 at 8:23 AM, <jun.liu@nomura.com>
wrote:
> >> >>>>>
> >> >>>>>> V5.2,
> >> >>>>>>
> >> >>>>>> I’m trying to figure out how to deploy Kylin on a
MapR cluster,
> >> >>>>>> whether Kylin has to be installed on the same cluster
of the MapR
> >> cluster
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> *From:* Luke Han [mailto:luke.hq@gmail.com]
> >> >>>>>> *Sent:* Tuesday, July 04, 2017 10:04 PM
> >> >>>>>> *To:* user
> >> >>>>>> *Subject:* Re: Kylin for MapR
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Kylin support MapR, which MapR version you are using
now?
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> Best Regards!
> >> >>>>>> ---------------------
> >> >>>>>>
> >> >>>>>> Luke Han
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Tue, Jul 4, 2017 at 4:19 PM, <jun.liu@nomura.com>
wrote:
> >> >>>>>>
> >> >>>>>> Hi
> >> >>>>>>
> >> >>>>>> Does kylin support MapR version of the Hadoop?
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> This e-mail (including any attachments) is private
and
> >> confidential,
> >> >>>>>> may contain proprietary or privileged information and
is intended
> >> for the
> >> >>>>>> named recipient(s) only. Unintended recipients are
strictly
> >> prohibited from
> >> >>>>>> taking action on the basis of information in this e-mail
and must
> >> contact
> >> >>>>>> the sender immediately, delete this e-mail (and all
attachments)
> >> and
> >> >>>>>> destroy any hard copies. Nomura will not accept responsibility
or
> >> liability
> >> >>>>>> for the accuracy or completeness of, or the presence
of any virus
> >> or
> >> >>>>>> disabling code in, this e-mail. If verification is
sought please
> >> request a
> >> >>>>>> hard copy. Any reference to the terms of executed transactions
> >> should be
> >> >>>>>> treated as preliminary only and subject to formal written
> >> confirmation by
> >> >>>>>> Nomura. Nomura reserves the right to retain, monitor
and
> intercept
> >> e-mail
> >> >>>>>> communications through its networks (subject to and
in accordance
> >> with
> >> >>>>>> applicable laws). No confidentiality or privilege is
waived or
> >> lost by
> >> >>>>>> Nomura by any mistransmission of this e-mail. Any reference
to
> >> "Nomura" is
> >> >>>>>> a reference to any entity in the Nomura Holdings, Inc.
group.
> >> Please read
> >> >>>>>> our Electronic Communications Legal Notice which forms
part of
> >> this e-mail:
> >> >>>>>> http://www.Nomura.com/email_disclaimer.htm
> >> >>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.
> >> Nomura.com_email-5Fdisclaimer.htm&d=DwMFaQ&c=dCBwIlVXJsYZrY6
> >> gpNt0LA&r=nFqGYx5SOTBIei-Hwk8uz_JfhmSu2UlelQfdmGX2k5I&
> >> m=s6jR2dvhFFTZIuHhCE8WU8EYNQ7M5u-9SggFAoyn5-M&s=iXZr3lmnuJ4X
> >> k2-qORnbYZQALda0jRLZ3kzGy816bco&e=>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> This e-mail (including any attachments) is private
and
> >> confidential,
> >> >>>>>> may contain proprietary or privileged information and
is intended
> >> for the
> >> >>>>>> named recipient(s) only. Unintended recipients are
strictly
> >> prohibited from
> >> >>>>>> taking action on the basis of information in this e-mail
and must
> >> contact
> >> >>>>>> the sender immediately, delete this e-mail (and all
attachments)
> >> and
> >> >>>>>> destroy any hard copies. Nomura will not accept responsibility
or
> >> liability
> >> >>>>>> for the accuracy or completeness of, or the presence
of any virus
> >> or
> >> >>>>>> disabling code in, this e-mail. If verification is
sought please
> >> request a
> >> >>>>>> hard copy. Any reference to the terms of executed transactions
> >> should be
> >> >>>>>> treated as preliminary only and subject to formal written
> >> confirmation by
> >> >>>>>> Nomura. Nomura reserves the right to retain, monitor
and
> intercept
> >> e-mail
> >> >>>>>> communications through its networks (subject to and
in accordance
> >> with
> >> >>>>>> applicable laws). No confidentiality or privilege is
waived or
> >> lost by
> >> >>>>>> Nomura by any mistransmission of this e-mail. Any reference
to
> >> "Nomura" is
> >> >>>>>> a reference to any entity in the Nomura Holdings, Inc.
group.
> >> Please read
> >> >>>>>> our Electronic Communications Legal Notice which forms
part of
> >> this e-mail:
> >> >>>>>> http://www.Nomura.com/email_disclaimer.htm
> >> >>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> [image: What's New with Xactly]
> >> >>>>> <http://www.xactlycorp.com/email-click/>
> >> >>>>>
> >> >>>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> >> >>>>> <https://www.linkedin.com/company/xactly-corporation>
 [image:
> >> >>>>> Twitter] <https://twitter.com/Xactly>  [image: Facebook]
> >> >>>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> >> >>>>> <http://www.youtube.com/xactlycorporation>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> --
> >> >>>> Best regards,
> >> >>>>
> >> >>>> Shaofeng Shi 史少锋
> >> >>>>
> >> >>>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> [image: What's New with Xactly] <http://www.xactlycorp.com/ema
> >> il-click/>
> >> >>>
> >> >>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> >> >>> <https://www.linkedin.com/company/xactly-corporation>  [image:
> >> Twitter]
> >> >>> <https://twitter.com/Xactly>  [image: Facebook]
> >> >>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> >> >>> <http://www.youtube.com/xactlycorporation>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Best regards,
> >> >>
> >> >> Shaofeng Shi 史少锋
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > [image: What's New with Xactly] <http://www.xactlycorp.com/
> email-click/
> >> >
> >> >
> >> > <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> >> > <https://www.linkedin.com/company/xactly-corporation>  [image:
> Twitter]
> >> > <https://twitter.com/Xactly>  [image: Facebook]
> >> > <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> >> > <http://www.youtube.com/xactlycorporation>
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >>
> >
> >
>
> --
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
> <https://twitter.com/Xactly>  [image: Facebook]
> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> <http://www.youtube.com/xactlycorporation>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message