kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 邵志鹏 <bobr...@163.com>
Subject Re:Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
Date Wed, 14 Aug 2019 09:42:00 GMT

设置了kylin.engine.spark-conf.spark.yarn.archive,我也很奇怪,看上去就是没有加载到spark-libs.jar。



## manually upload spark-assembly jar to HDFS and then set this property will avoid repeatedly
uploading jar at runtime
kylin.engine.spark-conf.spark.yarn.archive=hdfs://master:9000/kylin/spark/spark-libs.jar


$ hadoop fs -ls hdfs://master:9000/kylin/spark/
Found 1 items
-rw-r--r--   3 admin supergroup  200878688 2019-08-14 17:16 hdfs://master:9000/kylin/spark/spark-libs.jar

在 2019-08-14 14:46:37,"Chao Long" <chao.long0101@gmail.com> 写道:

这个参数配置了么,kylin.engine.spark-conf.spark.yarn.archive,是否配置为自己集群所在的hdfs目录,而不是文档上的hdfs://sandbox.hortonworks.com:8020


On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏 <bobrave@163.com> wrote:

2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Updating
cube instance 'kylin_sales_cube_clone_spark'
2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7] org.apache.kylin.cube.CubeManager - Remove
segment kylin_sales_cube_clone_spark[20120101000000_20140201235500]


Action - Delete Segement can Remove segment , and then rebuild it.


Main problem is 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function。


不确定是否是jar报冲突,spark下面所有的jar都一起打包放到hdfs了,里面有spark-core。
spark版本:2.3.2
spark-core_2.11-2.3.2



在 2019-08-14 14:08:03,"邵志鹏" <bobrave@163.com> 写道:

Hi kylin team:


kylin-2.6.3
使用MapReduce已经成功构建smaple cube。
然后clone了一份kylin_sales_cube修改为Spark,build 报错。


Spark是使用脚本下载的,配置参考了http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。




$KYLIN_HOME/bin/download-spark.sh
jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
hadoop fs -mkdir -p /kylin/spark/
hadoop fs -put spark-libs.jar /kylin/spark/
...


报错日志:
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function
        at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
~[kylin-engine-spark-2.6.3.jar:2.6.3]
        at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
~[kylin-engine-spark-2.6.3.jar:2.6.3]
        at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
~[kylin-core-job-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
~[kylin-server-base-2.6.3.jar:2.6.3]
        ... 77 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.api.java.function.Function
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
~[catalina.jar:7.0.91]
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
~[catalina.jar:7.0.91]
        at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.<init>(SparkBatchCubingJobBuilder2.java:53)
~[kylin-engine-spark-2.6.3.jar:2.6.3]
        at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
~[kylin-engine-spark-2.6.3.jar:2.6.3]
        at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
~[kylin-core-job-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
~[kylin-server-base-2.6.3.jar:2.6.3]




报错之后,再去提交build,又报新的错:
Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube kylin_sales_cube_clone_spark
has segments [kylin_sales_cube_clone_spark[20120101000000_20140201235500]], but none of them
is READY. It's not allowed for parallel building
        at org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422)
~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221)
~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) ~[kylin-server-base-2.6.3.jar:2.6.3]
        at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
~[kylin-server-base-2.6.3.jar:2.6.3]
        ... 77 more




Monitor里面是没有新的构建任务的。


Thanks.


By 邵志鹏






 





 
Mime
View raw message