kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: running spark on kylin 2.2
Date Thu, 01 Mar 2018 00:26:11 GMT
Kylin 2.1/2.2/2.3  compiles (and ships) with Spark 2.1; If you want to run
with Spark 1.6, please use Kylin 2.0;

2018-03-01 6:45 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:

> Please use vendor's forum.
>
> Thanks
>
> -------- Original message --------
> From: Sonny Heer <sonnyheer@gmail.com>
> Date: 2/28/18 2:35 PM (GMT-08:00)
> To: user@kylin.apache.org
> Subject: Re: running spark on kylin 2.2
>
> So when I run it with just spark-submit it gets further.  but now there is
> a API difference.  Does Kylin 2.2 work with Spark 1.6.1 ?  This is the
> version that comes with HDP 2.4.2.0-258
>
> ERROR:
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/
> String;)Lorg/apache/spark/sql/Dataset;
> at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(
> SparkCubingByLayer.java:167
>
>
> It appears it only supports spark 2.x?  Please advise what we can do to
> make this work on HDP 2.4...
>
> Thanks
>
> On Wed, Feb 28, 2018 at 2:07 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
>
>> I don't see spark-libs.jar under $KYLIN_HOME/spark/jars
>>
>> per this doc: http://kylin.apache.org/docs21/tutorial/cube_spark.html
>>
>> On Wed, Feb 28, 2018 at 10:30 AM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>
>>> Hi Billy
>>> Looks like the current error is this:
>>>
>>> Error: Could not find or load main class org.apache.spark.deploy.yarn.A
>>> pplicationMaster
>>>
>>> End of LogType:stderr
>>>
>>> Thanks
>>>
>>> On Wed, Feb 28, 2018 at 8:04 AM, Billy Liu <billyliu@apache.org> wrote:
>>>
>>>> Any exception in logs?
>>>>
>>>> With Warm regards
>>>>
>>>> Billy Liu
>>>>
>>>>
>>>> 2018-02-28 22:53 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
>>>> > Anyone know what I need to set in order for spark-submit to use the
>>>> HDP
>>>> > version of spark and not the internal one?
>>>> >
>>>> > currently i see:
>>>> >
>>>> > export HADOOP_CONF_DIR=/ebs/kylin/hadoop-conf &&
>>>> > /ebs/kylin/apache-kylin-2.2.0-bin/spark/bin/spark-submit
>>>> >
>>>> >
>>>> > I see in the kylin.properties files:
>>>> > ## Spark conf (default is in spark/conf/spark-defaults.conf)
>>>> >
>>>> > Although it doesn't how how I can change this to use the HDP
>>>> spark-submit.
>>>> >
>>>> > Also HDP is on 1.6.1 version of spark and kylin internally uses 2.x.
>>>> Not
>>>> > sure if that matters during submit.  I can't seem to get more than 2
>>>> > executors to run without it failing with other errors.  We have about
>>>> 44
>>>> > slots on our cluster.
>>>> >
>>>> > Also uncommented:
>>>> > ## uncomment for HDP
>>>> >
>>>> > kylin.engine.spark-conf.spark.driver.extraJavaOptions=-Dhdp.
>>>> version=current
>>>> >
>>>> > kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=-Dhdp
>>>> .version=current
>>>> >
>>>> > kylin.engine.spark-conf.spark.executor.extraJavaOptions=-Dhd
>>>> p.version=current
>>>> >
>>>> > see attached for other properties set.
>>>>
>>>
>>>
>>
>


-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message