kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chao Long <chao.long0...@gmail.com>
Subject Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
Date Wed, 14 Aug 2019 06:46:37 GMT
这个参数配置了么,*kylin.engine.spark-conf.spark.yarn.archive,*
是否配置为自己集群所在的hdfs目录,而不是文档上的hdfs://sandbox.hortonworks.com:8020

On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 <bobrave@163.com> wrote:

> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Updating cube instance
> 'kylin_sales_cube_clone_spark'
> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Remove segment
> kylin_sales_cube_clone_spark[20120101000000_20140201235500]
>
> Action - Delete Segement can Remove segment , and then rebuild it.
>
> Main problem is
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function。
>
> 不确定是否是jar报冲突,spark下面所有的jar都一起打包放到hdfs了,里面有spark-core。
> spark版本:2.3.2
> spark-core_2.11-2.3.2
>
>
> 在 2019-08-14 14:08:03,"邵志鹏" <bobrave@163.com> 写道:
>
> Hi kylin team:
>
> kylin-2.6.3
> 使用MapReduce已经成功构建smaple cube。
> 然后clone了一份kylin_sales_cube修改为Spark,build 报错。
>
> Spark是使用脚本下载的,配置参考了
> http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
>
>
> $KYLIN_HOME/bin/download-spark.sh
> jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
> hadoop fs -mkdir -p /kylin/spark/
> hadoop fs -put spark-libs.jar /kylin/spark/
> ...
>
> 报错日志:
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/function/Function
>         at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         ... 77 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function
>         at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
> ~[catalina.jar:7.0.91]
>         at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
> ~[catalina.jar:7.0.91]
>         at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>
>
> 报错之后,再去提交build,又报新的错:
> Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube
> kylin_sales_cube_clone_spark has segments
> [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of
> them is READY. It's not allowed for parallel building
>         at
> org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>         ... 77 more
>
>
> Monitor里面是没有新的构建任务的。
>
> Thanks.
>
> By 邵志鹏
>
>
>
>
>
>
>
>
>

Mime
View raw message