kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Which version of spark is kylin 2.0 Beta using ?
Date Mon, 06 Mar 2017 02:11:05 GMT
Yes it is expected; the child RDD is based on parent RDD; as parent RDD
already be generated so some stages will be skipped (each layer's DAG is a
full DAG from source data), those stages can be skipped. See this
screenshot:

[image: 内嵌图片 1]

2017-03-06 10:01 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com>:

> A more problem is confusing me now, I read this blog
> http://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/ ,and got
>  the key  point of by-layer spark cubing is "a RDD  per layer”, and during
> the cubing, I take a look at Spark DAG, is that a spark job for a layer ?
>
> my cube has 11 dimensions, the first spark job has 2 stages, Is the stage
> 1 get the first layer( 11-D Cuboid) RDD(in your blog is  RDD1) ?  In second
> job, it has 3 stages, the first stage is skipped , Are the remained 2
> stages get the second layer(10-D cuboids) RDD (RDD2) ? and In every follow
> spark jobs, always skipped some former stages ,just leave last 2 stages to
> implements.
>
> Please forgive my ignorance of this.
>
> Thanks.
>
>
> 在 2017年3月3日,下午6:10,ShaoFeng Shi <shaofengshi@gmail.com> 写道:
>
> I'm afraid not that simple; the embedded spark in Kylin is v1.6.3, and the
> Kylin code is compiled with 1.6.3 API; if you want to switch to 2.1, need
> recompile and re-package. That is doable, you can take a try.
>
> Get Outlook for iOS <https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 2, 2017 at 1:37 PM +0700, "Mars Xu" <xujiao.mycafe@gmail.com>
> wrote:
>
> Hi, Shaofeng,
>>
>> Can I change the spark version to spark 2.1 using
>> “kylin.engine.spark-conf” to config ?
>>
>> 在 2017年3月1日,下午5:27,ShaoFeng Shi <shaofengshi@apache.org> 写道:
>>
>> Kylin embed a spark, which is spark v1.6.3; I added a blog to briefly
>> introduce this, later will add a doc to instruct on how to config:
>>
>> https://kylin.apache.org/blog/2017/02/23/by-layer-spark-cubing/
>>
>> To customize the spark conf, please do that in kylin.properties with
>> prefix "kylin.engine.spark-conf."
>>
>> 2017-03-01 15:49 GMT+08:00 Mars Xu <xujiao.mycafe@gmail.com>:
>>
>>> Hi ,
>>>    I have install kylin 2.0 beta on my cluster which has already deploy
>>> a spark 2.1, but when I startup kylin server and build cube, the Web UI of
>>> Cube Build Step show it is spark 1.6.3.
>>>
>>>    Is that kylin already include a spark version ? or some other else I
>>> need to configure ?
>>>
>>>
>>> Thanks
>>>
>>> Mars.
>>
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message