From user-return-3050-archive-asf-public=cust-asf.ponee.io@kylin.apache.org Thu Mar 1 01:26:55 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2DA29180657 for ; Thu, 1 Mar 2018 01:26:55 +0100 (CET) Received: (qmail 22990 invoked by uid 500); 1 Mar 2018 00:26:54 -0000 Mailing-List: contact user-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kylin.apache.org Delivered-To: mailing list user@kylin.apache.org Received: (qmail 22979 invoked by uid 99); 1 Mar 2018 00:26:54 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2018 00:26:54 +0000 Received: from mail-pf0-f174.google.com (mail-pf0-f174.google.com [209.85.192.174]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id C06BA6E0 for ; Thu, 1 Mar 2018 00:26:52 +0000 (UTC) Received: by mail-pf0-f174.google.com with SMTP id a16so1713254pfn.9 for ; Wed, 28 Feb 2018 16:26:52 -0800 (PST) X-Gm-Message-State: APf1xPCrwlRHtmXOp8TpBD3cuTv84jYSzB2Nr4dEQE/kLXOzUpR3YFZs QGdNqre7YXJA18qiCHIzy8iSRZ4zKZndHofxv7c= X-Google-Smtp-Source: AG47ELug1SAtBDMiIzda1kiHybB6UKr1v8PLgIYVOq/ypbWnqtlx/FDrLhpeN3vX3Tl9CwTgfnO2jpmqRuHwXxVfKRU= X-Received: by 10.101.76.134 with SMTP id m6mr7373444pgt.445.1519864011418; Wed, 28 Feb 2018 16:26:51 -0800 (PST) MIME-Version: 1.0 Received: by 10.100.160.240 with HTTP; Wed, 28 Feb 2018 16:26:11 -0800 (PST) In-Reply-To: <5a9730f5.c361620a.64254.acfc@mx.google.com> References: <5a9730f5.c361620a.64254.acfc@mx.google.com> From: ShaoFeng Shi Date: Thu, 1 Mar 2018 08:26:11 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: running spark on kylin 2.2 To: user Content-Type: multipart/alternative; boundary="089e08234c1081f41105664eea7a" --089e08234c1081f41105664eea7a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Kylin 2.1/2.2/2.3 compiles (and ships) with Spark 2.1; If you want to run with Spark 1.6, please use Kylin 2.0; 2018-03-01 6:45 GMT+08:00 Ted Yu : > Please use vendor's forum. > > Thanks > > -------- Original message -------- > From: Sonny Heer > Date: 2/28/18 2:35 PM (GMT-08:00) > To: user@kylin.apache.org > Subject: Re: running spark on kylin 2.2 > > So when I run it with just spark-submit it gets further. but now there i= s > a API difference. Does Kylin 2.2 work with Spark 1.6.1 ? This is the > version that comes with HDP 2.4.2.0-258 > > ERROR: > Exception in thread "main" java.lang.NoSuchMethodError: > org.apache.spark.sql.hive.HiveContext.table(Ljava/lang/ > String;)Lorg/apache/spark/sql/Dataset; > at org.apache.kylin.engine.spark.SparkCubingByLayer.execute( > SparkCubingByLayer.java:167 > > > It appears it only supports spark 2.x? Please advise what we can do to > make this work on HDP 2.4... > > Thanks > > On Wed, Feb 28, 2018 at 2:07 PM, Sonny Heer wrote: > >> I don't see spark-libs.jar under $KYLIN_HOME/spark/jars >> >> per this doc: http://kylin.apache.org/docs21/tutorial/cube_spark.html >> >> On Wed, Feb 28, 2018 at 10:30 AM, Sonny Heer wrote= : >> >>> Hi Billy >>> Looks like the current error is this: >>> >>> Error: Could not find or load main class org.apache.spark.deploy.yarn.A >>> pplicationMaster >>> >>> End of LogType:stderr >>> >>> Thanks >>> >>> On Wed, Feb 28, 2018 at 8:04 AM, Billy Liu wrote: >>> >>>> Any exception in logs? >>>> >>>> With Warm regards >>>> >>>> Billy Liu >>>> >>>> >>>> 2018-02-28 22:53 GMT+08:00 Sonny Heer : >>>> > Anyone know what I need to set in order for spark-submit to use the >>>> HDP >>>> > version of spark and not the internal one? >>>> > >>>> > currently i see: >>>> > >>>> > export HADOOP_CONF_DIR=3D/ebs/kylin/hadoop-conf && >>>> > /ebs/kylin/apache-kylin-2.2.0-bin/spark/bin/spark-submit >>>> > >>>> > >>>> > I see in the kylin.properties files: >>>> > ## Spark conf (default is in spark/conf/spark-defaults.conf) >>>> > >>>> > Although it doesn't how how I can change this to use the HDP >>>> spark-submit. >>>> > >>>> > Also HDP is on 1.6.1 version of spark and kylin internally uses 2.x. >>>> Not >>>> > sure if that matters during submit. I can't seem to get more than 2 >>>> > executors to run without it failing with other errors. We have abou= t >>>> 44 >>>> > slots on our cluster. >>>> > >>>> > Also uncommented: >>>> > ## uncomment for HDP >>>> > >>>> > kylin.engine.spark-conf.spark.driver.extraJavaOptions=3D-Dhdp. >>>> version=3Dcurrent >>>> > >>>> > kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=3D-Dhdp >>>> .version=3Dcurrent >>>> > >>>> > kylin.engine.spark-conf.spark.executor.extraJavaOptions=3D-Dhd >>>> p.version=3Dcurrent >>>> > >>>> > see attached for other properties set. >>>> >>> >>> >> > --=20 Best regards, Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B --089e08234c1081f41105664eea7a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Kylin 2.1/2.2/2.3=C2=A0 compiles (and ships) with Spark 2.= 1; If you want to run with Spark 1.6, please use Kylin 2.0;

2018-03-01 6:45 GMT+08:00 T= ed Yu <yuzhihong@gmail.com>:
Please use vendor's forum.=C2=A0<= /font>

Thanks

--------= Original message --------
From: Sonny Heer <sonnyheer@gmail.com>
<= div>Date: 2/28/18 2:35 PM (GMT-08:00)
Subject: Re: running spark on kylin 2.2

So when I run it with just spark-submit it gets further.=C2=A0 bu= t now there is a API difference.=C2=A0 Does Kylin 2.2 work with Spark 1.6.1= ?=C2=A0 This is the version that comes with HDP=20 2.4.2.0-258

ERROR:
Exception in thread "main" java.lang.NoSuchMethodError: org.apach= e.spark.sql.hive.HiveContext.table(Ljava/lang/String;)Lorg/apache= /spark/sql/Dataset;
at org.apache.kylin.engine.spark.SparkCubi= ngByLayer.execute(SparkCubingByLayer.java:167


It appears it only supports spark 2.x?=C2=A0 Please advise = what we can do to make this work on HDP 2.4...

Tha= nks

On= Wed, Feb 28, 2018 at 2:07 PM, Sonny Heer <sonnyheer@gmail.com> wrote:
I don't= see spark-libs.jar under $KYLIN_HOME/spark/jars


On Wed, Feb 28, 2018 at 10:30 AM, Sonny Heer <sonnyheer@gm= ail.com> wrote:
Hi Billy
Looks like the current error is this:

Error: Could not find or load main class org.apache.spark.deploy.yarn.= ApplicationMaster

End of LogType:stderr


Thanks

On Wed, = Feb 28, 2018 at 8:04 AM, Billy Liu <billyliu@apache.org> w= rote:
Any exception in logs?

With Warm regards

Billy Liu


2018-02-28 22:53 GMT+08:00 Sonny Heer <sonnyheer@gmail.com>:
> Anyone know what I need to set in order for spark-submit to use the HD= P
> version of spark and not the internal one?
>
> currently i see:
>
> export HADOOP_CONF_DIR=3D/ebs/kylin/hadoop-conf &&
> /ebs/kylin/apache-kylin-2.2.0-bin/spark/bin/spark-submit
>
>
> I see in the kylin.properties files:
> ## Spark conf (default is in spark/conf/spark-defaults.conf)
>
> Although it doesn't how how I can change this to use the HDP spark= -submit.
>
> Also HDP is on 1.6.1 version of spark and kylin internally uses 2.x.= =C2=A0 Not
> sure if that matters during submit.=C2=A0 I can't seem to get more= than 2
> executors to run without it failing with other errors.=C2=A0 We have a= bout 44
> slots on our cluster.
>
> Also uncommented:
> ## uncomment for HDP
>
> kylin.engine.spark-conf.spark.driver.extraJavaOptions=3D-Dhdp.version=3Dcurrent
>
> kylin.engine.spark-conf.spark.yarn.am.extraJavaOptions=3D-Dhdp.version=3Dcurrent
>
> kylin.engine.spark-conf.spark.executor.extraJavaOptions=3D-Dhdp.version=3Dcurrent
>
> see attached for other properties set.






-= -
Best regards,

Shaofe= ng Shi =E5=8F=B2=E5=B0=91=E9=94=8B

--089e08234c1081f41105664eea7a--