kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Failed to find metadata store when cubing by spark
Date Sat, 03 Jun 2017 13:05:54 GMT
Spark couldn't connect with the HBase cluster for metadata. Some possible
causes I can think of:
1) the correct hbase-site.xml wasn't in the HADOOP_CONF_DIR
2) some HBase dependent jar wasn't submitted with the application

You may need check more logs, including the Spark executor logs. Start
Spark history sever and then check that job's detail log.

2017-06-03 18:36 GMT+08:00 qiumingming <csurjqmm@gmail.com>:

> Hi All:
>     I got a error when cubing by spark which make me confused. When I start
> kylin I got these log:
>     2017-06-03 18:09:19,647 INFO  [Thread-8-EventThread]
> state.ConnectionStateManager:228 : State
>     change: CONNECTED
>     2017-06-03 18:09:19,680 INFO  [Thread-8] execution.ExecutableManager:53
> : Using metadata url:
>     kylin_v2_metadata@hbase
>    2017-06-03 18:09:19,682 INFO  [Thread-8] dao.ExecutableDao:75 : Using
> metadata url:
>    kylin_v2_metadata@hbase
>    2017-06-03 18:09:19,687 INFO  [Thread-8] persistence.ResourceStore:88 :
> Using metadata url
>    kylin_v2_metadata@hbase for resource store
>    2017-06-03 18:09:19,708 DEBUG [Thread-8] hbase.HBaseConnection:274 :
> HTable
>    'kylin_v2_metadata' already exists
>    2017-06-03 18:09:19,710 DEBUG [Thread-8] metadata.MetadataManager:424 :
> Reloading SourceTable
>    from folder kylin_v2_metadata(key='/table')@kylin_v2_metadata@hbase
>
>    It seems the metadata store works well. But when I start cubing by spark
> , the log file shows these info:
>     17/06/03 18:14:16 WARN scheduler.TaskSetManager: Lost task 0.0 in stage
> 0.0 (TID 0, n6-129-021.byted.org): java.lang.IllegalArgumentException:
> Failed to find metadata store by url: kylin_v2_metadata@hbase
>         at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(
> ResourceStore.java:99)
>         at
> org.apache.kylin.common.persistence.ResourceStore.
> getStore(ResourceStore.java:110)
>         at
> org.apache.kylin.cube.CubeDescManager.getStore(CubeDescManager.java:370)
>         at
> org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
> CubeDescManager.java:298)
>         at
> org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:109)
>         at
> org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
>         at
> org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:109)
>         at
> org.apache.kylin.cube.CubeSegment.getCubeDesc(CubeSegment.java:119)
>         at
> org.apache.kylin.cube.CubeSegment.isEnableSharding(CubeSegment.java:467)
>         at
> org.apache.kylin.cube.kv.RowKeyEncoder.<init>(RowKeyEncoder.java:48)
>         at
> org.apache.kylin.cube.kv.AbstractRowKeyEncoder.createInstance(
> AbstractRowKeyEncoder.java:48)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:209)
>         at
> org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(
> SparkCubingByLayer.java:197)
>         at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(
> JavaPairRDD.scala:1025)
>         at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(
> JavaPairRDD.scala:1025)
>         at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
>         at
> org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.
> scala:191)
>         at
> org.apache.spark.shuffle.sort.SortShuffleWriter.write(
> SortShuffleWriter.scala:64)
>         at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
>         at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:218)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
>       I don't know why this happened, I need your help. Thanks.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Failed-to-find-metadata-store-when-cubing-by-spark-tp8167.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message