kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Kylin for MapR
Date Mon, 10 Jul 2017 00:06:52 GMT
Is there some coprocessor-equivalent mechanism in maprdb? If yes, you can
try to implement the logic into it. Otherwise, the performance might be
unacceptable when there is bunch of data in each region.

I rememer in very early version, Kylin doesn't have coprocessor, all data
be fetched to Kylin server and then do filtering and aggregation by
Calcite. Later coprocessor was added and improved continiously. Now there
is no option to disable it. You have to modify the code if don't want to
trigger it.



2017-07-09 1:34 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:

>
> Thanks Shao for some pointers. I would like some more advice on how I can
> replace maprdb as datasource.
>
> Currently, you can  create cubes on maprdb using hbase-storage module as
> maprdb does support hbase client/admin APIs. Only problem is while querying
> it throws an error due to Lack of support of Co-processor.
>
> My initial goal is to get past coprocessor error and return query result.
> Then find a better way to implement that strategy - may be a separate
> maprdb-storage module. Then look into how to improve performance without
> coprocessors. Any advice on these approaches ?
>
> Thanks,
> Nirav
>
> On Sat, Jul 8, 2017 at 7:54 AM, ShaoFeng Shi <shaofengshi@apache.org>
> wrote:
>
>> The core modules like core-cube, core-storage are totally independent of
>> HBase; while some others like engine-mr, engine-spark has dependencies on
>> HBase. If you want to replace it, you need implement new cubing engine as
>> well. Please also note that Kylin's metadata is persisted to HBase by
>> default, you need have another implementation for ResourceStore.
>>
>> In a short, the plug-in architecture works (we have verfied that), while
>> changing the storage is a complex task which take some time to be function
>> complete and performance stable.
>>
>>
>>
>>
>>
>>
>> 2017-07-08 5:19 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:
>>
>>> Yes, MapR-DB doesn't support coprocessors.
>>>
>>> Here's the thing though - based on kylin plugin architecture it
>>> shouldn't be a problem ideally. Aggregation as well as other DML/DDL
>>> operation on datasources should be done transparently. i.e. using
>>> kylin-hbase adapter written with calcite or something. It's upto the writer
>>> of those adapters to implement aggregates however they want. i.e. either
>>> using coprocessors, or in-memory on application server side or using spark.
>>> http://kylin.apache.org/development/plugin_arch.html
>>>
>>> I think for mapr-db we can leverage mapr-drill with secondary indexes
>>> for faster filtering and drill does parallel aggregation as well. Other
>>> option is to use mapr-spark with mapr-db which can does the same.
>>>
>>> Do you know how tightly hbase is coupled with other modules of kylin
>>> source other then hbase-storage.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Jul 6, 2017 at 6:53 PM, ShaoFeng Shi <shaofengshi@apache.org>
>>> wrote:
>>>
>>>> Hi Nirav,
>>>>
>>>> I googled that "HBase coprocessors are not present in MapR DB", is
>>>> this true? You know Kylin relies on HBase coprocessor to do filering and
>>>> aggregation in each region local; If coprocessor is not available, the
>>>> performance will be a problem.
>>>>
>>>> 2017-07-07 1:29 GMT+08:00 Nirav Patel <npatel@xactlycorp.com>:
>>>>
>>>>> Hi, We are a mapr users. You will need to deploy separate Hbase
>>>>> Cluster alongside your mapr Cluster or on top of it. You won't be able
to
>>>>> use Mapr-DB with Kylin.
>>>>> I am looking into writing Kyling - MaprDB adapter.
>>>>>
>>>>> On Tue, Jul 4, 2017 at 8:23 AM, <jun.liu@nomura.com> wrote:
>>>>>
>>>>>> V5.2,
>>>>>>
>>>>>> I’m trying to figure out how to deploy Kylin on a MapR cluster,
>>>>>> whether Kylin has to be installed on the same cluster of the MapR
cluster
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Luke Han [mailto:luke.hq@gmail.com]
>>>>>> *Sent:* Tuesday, July 04, 2017 10:04 PM
>>>>>> *To:* user
>>>>>> *Subject:* Re: Kylin for MapR
>>>>>>
>>>>>>
>>>>>>
>>>>>> Kylin support MapR, which MapR version you are using now?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards!
>>>>>> ---------------------
>>>>>>
>>>>>> Luke Han
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 4, 2017 at 4:19 PM, <jun.liu@nomura.com> wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> Does kylin support MapR version of the Hadoop?
>>>>>>
>>>>>>
>>>>>>
>>>>>> This e-mail (including any attachments) is private and confidential,
>>>>>> may contain proprietary or privileged information and is intended
for the
>>>>>> named recipient(s) only. Unintended recipients are strictly prohibited
from
>>>>>> taking action on the basis of information in this e-mail and must
contact
>>>>>> the sender immediately, delete this e-mail (and all attachments)
and
>>>>>> destroy any hard copies. Nomura will not accept responsibility or
liability
>>>>>> for the accuracy or completeness of, or the presence of any virus
or
>>>>>> disabling code in, this e-mail. If verification is sought please
request a
>>>>>> hard copy. Any reference to the terms of executed transactions should
be
>>>>>> treated as preliminary only and subject to formal written confirmation
by
>>>>>> Nomura. Nomura reserves the right to retain, monitor and intercept
e-mail
>>>>>> communications through its networks (subject to and in accordance
with
>>>>>> applicable laws). No confidentiality or privilege is waived or lost
by
>>>>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura"
is
>>>>>> a reference to any entity in the Nomura Holdings, Inc. group. Please
read
>>>>>> our Electronic Communications Legal Notice which forms part of this
e-mail:
>>>>>> http://www.Nomura.com/email_disclaimer.htm
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.Nomura.com_email-5Fdisclaimer.htm&d=DwMFaQ&c=dCBwIlVXJsYZrY6gpNt0LA&r=nFqGYx5SOTBIei-Hwk8uz_JfhmSu2UlelQfdmGX2k5I&m=s6jR2dvhFFTZIuHhCE8WU8EYNQ7M5u-9SggFAoyn5-M&s=iXZr3lmnuJ4Xk2-qORnbYZQALda0jRLZ3kzGy816bco&e=>
>>>>>>
>>>>>>
>>>>>>
>>>>>> This e-mail (including any attachments) is private and confidential,
>>>>>> may contain proprietary or privileged information and is intended
for the
>>>>>> named recipient(s) only. Unintended recipients are strictly prohibited
from
>>>>>> taking action on the basis of information in this e-mail and must
contact
>>>>>> the sender immediately, delete this e-mail (and all attachments)
and
>>>>>> destroy any hard copies. Nomura will not accept responsibility or
liability
>>>>>> for the accuracy or completeness of, or the presence of any virus
or
>>>>>> disabling code in, this e-mail. If verification is sought please
request a
>>>>>> hard copy. Any reference to the terms of executed transactions should
be
>>>>>> treated as preliminary only and subject to formal written confirmation
by
>>>>>> Nomura. Nomura reserves the right to retain, monitor and intercept
e-mail
>>>>>> communications through its networks (subject to and in accordance
with
>>>>>> applicable laws). No confidentiality or privilege is waived or lost
by
>>>>>> Nomura by any mistransmission of this e-mail. Any reference to "Nomura"
is
>>>>>> a reference to any entity in the Nomura Holdings, Inc. group. Please
read
>>>>>> our Electronic Communications Legal Notice which forms part of this
e-mail:
>>>>>> http://www.Nomura.com/email_disclaimer.htm
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [image: What's New with Xactly]
>>>>> <http://www.xactlycorp.com/email-click/>
>>>>>
>>>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>>>> <https://www.linkedin.com/company/xactly-corporation>  [image:
>>>>> Twitter] <https://twitter.com/Xactly>  [image: Facebook]
>>>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>>>> <http://www.youtube.com/xactlycorporation>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>
>>>
>>>
>>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>>
>>> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
>>> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
>>> <https://twitter.com/Xactly>  [image: Facebook]
>>> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
>>> <http://www.youtube.com/xactlycorporation>
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
>
> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>
> <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn]
> <https://www.linkedin.com/company/xactly-corporation>  [image: Twitter]
> <https://twitter.com/Xactly>  [image: Facebook]
> <https://www.facebook.com/XactlyCorp>  [image: YouTube]
> <http://www.youtube.com/xactlycorporation>
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message