kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Fixed - Re: MetricRegistry - NoClassDefFoundError - Hbase Jar Issue - Spark - (Step - Convert Cuboid Data to HFile)?
Date Thu, 13 Jun 2019 13:37:47 GMT
Hi Joh, thank you for your sharing. :)

Best regards,

Shaofeng Shi 史少锋
Apache Kylin PMC
Email: shaofengshi@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org




Jon Shoberg <jon.shoberg@gmail.com> 于2019年6月11日周二 上午7:07写道:

> This has been fixed now.  I had to manually search the jar files include
> the proper jars in the "jars folder" of my spark setup.
>
> find /opt -name "*.jar" | xargs grep
> org.apache.hadoop.hbase.metrics.MetricRegistry
>
> Afterwards the step completed and finished the job on the upgraded cluster.
>
> J
>
>
> On Mon, Jun 10, 2019 at 8:24 AM Jon Shoberg <jon.shoberg@gmail.com> wrote:
>
>> Good Morning,
>>
>>
>>  [On Error and Question]
>>
>>
>>   Has anyone fixed the error in the "Convert Cuboid Data to HFile" step; spark 2.3.2
and Kylin 2.6.2
>>
>>
>>    * java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hbase/metrics/MetricRegistry;
>>
>>
>>  I think this is a classpath issue. On all the slaves and master servers I have the
library present and in the classpath.
>>
>>
>>  There is a JIRA issue which should have fixed this in 2.6.0 but I'm still getting
the error message.
>>
>>
>>
>>  [On The Command]
>>
>>
>>   Following is the command for the step and I've manually added what I think are
the required JARs but I get the same failure
>>
>>
>>  export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/kylin/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry  --conf spark.max-partition=500  --conf spark.executor.instances=40
 --conf spark.yarn.queue=default  --conf spark.history.fs.logDirectory=hdfs:///kylin/spark-history
 --conf spark.io.compression.codec=snappy  --conf spark.dynamicAllocation.enabled=true  --conf
spark.driver.extraClassPath=/opt/spark/jars/snappy*.jar  --conf spark.master=yarn  --conf
spark.dynamicAllocation.schedulerBacklogTimeout=1  --conf spark.dynamicAllocation.executorIdleTimeout=300
 --conf spark.hadoop.yarn.timeline-service.enabled=false  --conf spark.executor.memory=10G
 --conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs:///kylin/spark-history
 --conf spark.dynamicAllocation.minExecutors=1  --conf spark.local.dir=/opt/volume/tmp1,/opt/volume/tmp2,/opt/volume/tmp3
 --conf spark.executor.cores=1  --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=true
 --conf spark.executor.memoryOverhead=1024  --conf spark.hadoop.dfs.replication=2  --conf
spark.dynamicAllocation.maxExecutors=1000  --conf spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.DefaultCodec
 --conf spark.driver.extraLibraryPath=/opt/hadoop/lib/native  --conf spark.driver.memory=10G
 --conf spark.submit.deployMode=cluster  --conf spark.executor.extraLibraryPath=/opt/hadoop/lib/native
 --conf spark.shuffle.service.enabled=true --jars /opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/hbase-metrics-1.4.8.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/hbase-hadoop2-compat-1.4.8.jar,/opt/hbase/lib/hbase-metrics-api-1.4.8.jar,
/opt/kylin/lib/kylin-job-2.6.2.jar -className org.apache.kylin.storage.hbase.steps.SparkCubeHFile
-partitions hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/matt_vw_nodes/rowkey_stats/part-r-00000_hfile
-counterOutput hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/matt_vw_nodes/counter
-cubename matt_vw_nodes -output hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/matt_vw_nodes/hfile
-input hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/matt_vw_nodes/cuboid/
-segmentId 91dbfc05-1e54-8fd1-6550-855a6dd657da -metaUrl kylin_metadata@hdfs,path=hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/matt_vw_nodes/metadata
-hbaseConfPath hdfs://master:9000/kylin/kylin_metadata/kylin-15389716-d96c-4778-ffe6-aa7ec8945c65/hbase-conf.xml
>>
>>
>>  [On Diagnostics]
>>
>>     Here is information on the error.
>>
>>
>> [image: image.png]
>>
>>
>>
>> 2019-06-10 08:09:40 INFO  TransportClientFactory:184 - Found inactive connection
to master.hadoop.lan/192.168.1.20:7337, creating a new one.
>> 2019-06-10 08:09:40 INFO  TransportClientFactory:267 - Successfully created connection
to master.hadoop.lan/192.168.1.20:7337 after 3 ms (0 ms spent in bootstraps)
>> 2019-06-10 08:09:41 INFO  TransportClientFactory:184 - Found inactive connection
to slave3.hadoop.lan/192.168.1.23:7337, creating a new one.
>> 2019-06-10 08:09:41 INFO  TransportClientFactory:267 - Successfully created connection
to slave3.hadoop.lan/192.168.1.23:7337 after 2 ms (0 ms spent in bootstraps)
>> 2019-06-10 08:10:33 INFO  FileOutputCommitter:108 - File Output Committer Algorithm
version is 1
>> 2019-06-10 08:10:34 INFO  FileOutputCommitter:108 - File Output Committer Algorithm
version is 1
>> 2019-06-10 08:10:34 INFO  MetricsConfig:112 - loaded properties from hadoop-metrics2.properties
>> 2019-06-10 08:10:35 INFO  MetricsSystemImpl:375 - Scheduled snapshot period at 10
second(s).
>> 2019-06-10 08:10:35 INFO  MetricsSystemImpl:192 - HBase metrics system started
>> 2019-06-10 08:10:35 ERROR Utils:91 - Aborting task
>> java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hbase/metrics/MetricRegistry;
>> 	at java.lang.Class.getDeclaredFields0(Native Method)
>> 	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
>> 	at java.lang.Class.getDeclaredFields(Class.java:1916)
>> 	at org.apache.hadoop.util.ReflectionUtils.getDeclaredFieldsIncludingInherited(ReflectionUtils.java:323)
>> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.initRegistry(MetricsSourceBuilder.java:92)
>> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.<init>(MetricsSourceBuilder.java:56)
>> 	at org.apache.hadoop.metrics2.lib.MetricsAnnotations.newSourceBuilder(MetricsAnnotations.java:43)
>> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:224)
>> 	at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
>> 	at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:44)
>> 	at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:36)
>> 	at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:73)
>> 	at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
>> 	at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:191)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
>> 	at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
>> 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:109)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> 	at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.metrics.MetricRegistry
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> 	... 30 more
>> 2019-06-10 08:10:35 ERROR SparkHadoopWriter:70 - Task attempt_20190610074916_0003_r_000000_0
aborted.
>> 2019-06-10 08:10:35 ERROR Executor:91 - Exception in task 0.0 in stage 1.0 (TID 56)
>> org.apache.spark.SparkException: Task failed while writing rows
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:155)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
>> 	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>> 	at org.apache.spark.scheduler.Task.run(Task.scala:109)
>> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> 	at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hbase/metrics/MetricRegistry;
>> 	at java.lang.Class.getDeclaredFields0(Native Method)
>> 	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
>> 	at java.lang.Class.getDeclaredFields(Class.java:1916)
>> 	at org.apache.hadoop.util.ReflectionUtils.getDeclaredFieldsIncludingInherited(ReflectionUtils.java:323)
>> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.initRegistry(MetricsSourceBuilder.java:92)
>> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder.<init>(MetricsSourceBuilder.java:56)
>> 	at org.apache.hadoop.metrics2.lib.MetricsAnnotations.newSourceBuilder(MetricsAnnotations.java:43)
>> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:224)
>> 	at org.apache.hadoop.hbase.metrics.BaseSourceImpl.<init>(BaseSourceImpl.java:115)
>> 	at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:44)
>> 	at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:36)
>> 	at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:73)
>> 	at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
>> 	at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:191)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229)
>> 	at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
>> 	at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.write(SparkHadoopWriter.scala:356)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:130)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$4.apply(SparkHadoopWriter.scala:127)
>> 	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1415)
>> 	at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:139)
>> 	... 8 more
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.metrics.MetricRegistry
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> 	... 30 more
>> 2019-06-10 08:10:35 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 57
>> 2019-06-10 08:10:35 INFO  Executor:54 - Running task 0.1 in stage 1.0 (TID 57)
>> 2019-06-10 08:10:35 INFO  ShuffleBlockFetcherIterator:54 - Getting 56 non-empty blocks
out of 56 blocks
>> 2019-06-10 08:10:35 INFO  TransportClientFactory:184 - Found inactive connection
to slave1.hadoop.lan/192.168.1.21:7337, creating a new one.
>> 2019-06-10 08:10:35 INFO  TransportClientFactory:267 - Successfully created connection
to slave1.hadoop.lan/192.168.1.21:7337 after 2 ms (0 ms spent in bootstraps)
>> 2019-06-10 08:10:35 INFO  ShuffleBlockFetcherIterator:54 - Started 1 remote fetches
in 9 ms
>> 2019-06-10 08:10:35 INFO  TransportClientFactory:184 - Found inactive connection
to slave2.hadoop.lan/192.168.1.22:7337, creating a new one.
>> 2019-06-10 08:10:35 INFO  TransportClientFactory:267 - Successfully created connection
to slave2.hadoop.lan/192.168.1.22:7337 after 2 ms (0 ms spent in bootstraps)
>> 2019-06-10 08:11:07 INFO  ExternalSorter:54 - Thread 112 spilling in-memory map of
5.2 GB to disk (1 time so far)
>>
>>

Mime
View raw message