kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: multiple EMRs sync
Date Mon, 06 Aug 2018 02:55:55 GMT
Yea that would be great if Kylin can have a centralized metastore in RDS.

The big problem for us now is this:

2 emr clusters each running kylin on master node.  Both share hbase s3 root
dir.

Cluster A creates a cube and does a build.  Cluster B can see the cube as
it builds in “monitor”, but once cube is finished.  Cube is “ready” only in
cluster A (job launched from).

We need somewhat isolated kylin nodes that can still share the same
backend.  This is a big win since then each cluster can scale read/write
independently in EMR - this is our goal.  Having read/write in the same
cluster doesn’t work for various reasons...

It seems kylin is really close since the monitoring of the cube is in sync
when sharing same hbase backend.

Using read replica did not work - when we try to login from the replica
kylin want able to work



On Sun, Aug 5, 2018 at 7:01 PM ShaoFeng Shi <shaofengshi@apache.org> wrote:

> Hi Sonny,
>
> EMR HBase read replica is a great feature, but we didn't try. Are you
> going to using this feature? or just want to deploy Kylin as a cluster?
>
> If putting Kylin metadata to RDS, can it be easier for you?
>
> 2018-08-04 0:05 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>
>> we'd like to use emr hbase read replicas if possible.  We had some issues
>> using this stragety since kylin requires write capability from all nodes
>> (on login for example).
>>
>> idea is to cluster kylin using multiple EMRs on master node.  If this
>> isn't possible we may go with separate instance approach, but that is prone
>> to errors as emr libs have to copied around..
>>
>> ref:
>>
>> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
>>
>> Anyone else have experience or can share their use case on emr?
>>
>> Thanks!
>>
>> On Thu, Aug 2, 2018 at 2:32 PM Sonny Heer <sonnyheer@gmail.com> wrote:
>>
>>> Is it possible in the new version of kylin to have multiple EMR clusters
>>> with Kylin installed on master node but talking to the same S3 location.
>>>
>>> e.g. one Write EMR cluster and one Read EMR cluster
>>>
>>> ?
>>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>

Mime
View raw message