kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jon shoberg (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KYLIN-3607) can't build cube with spark in v2.5.0
Date Wed, 05 Dec 2018 18:35:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16710462#comment-16710462
] 

jon shoberg edited comment on KYLIN-3607 at 12/5/18 6:34 PM:
-------------------------------------------------------------

I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR (non-HDP)

HBase 1.4.8 is the version currently being used and made sure there are no other HBase versioned
jars which would conflict this. Hadoop Version is 2.8.5.
{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit --class
org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=40  --conf spark.yarn.queue=default 
--conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history  --conf spark.master=yarn 
--conf spark.hadoop.yarn.timeline-service.enabled=false  --conf spark.executor.memory=4G 
--conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///kylin/spark-history 
--conf spark.yarn.executor.memoryOverhead=1024  --conf spark.driver.memory=2G  --conf spark.shuffle.service.enabled=true
--jars /opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
/opt/kylin/lib/kylin-job-2.5.2.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile
-partitions hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-00000_hfile
-counterOutput hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
-cubename HoldingNodeCube -output hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
-input hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
-segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
-hbaseConfPath hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}


was (Author: jshoberg):
I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR (non-HDP)

HBase 1.4.8 is the version currently being used and made sure there are no other HBase versioned
jars which would conflict this.


{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit --class
org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=40  --conf spark.yarn.queue=default 
--conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history  --conf spark.master=yarn 
--conf spark.hadoop.yarn.timeline-service.enabled=false  --conf spark.executor.memory=4G 
--conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///kylin/spark-history 
--conf spark.yarn.executor.memoryOverhead=1024  --conf spark.driver.memory=2G  --conf spark.shuffle.service.enabled=true
--jars /opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
/opt/kylin/lib/kylin-job-2.5.2.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile
-partitions hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-00000_hfile
-counterOutput hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
-cubename HoldingNodeCube -output hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
-input hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
-segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
-hbaseConfPath hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}

> can't build cube with spark in v2.5.0
> -------------------------------------
>
>                 Key: KYLIN-3607
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3607
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: ANIL KUMAR
>            Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, the following
is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException:
error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted
due to stage failure: Task 0 in stage 1.0 failed 4 times, java.lang.ExceptionInInitializerError
>  at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory
Is the hadoop compatibility jar on the classpath?
>  at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message