kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "skyyws"<sky...@163.com>
Subject Build sample error with spark on kylin 2.0.0
Date Mon, 19 Jun 2017 13:55:47 GMT
Hi all,
I met an error when using spark engine build kylin sample on step "Build Cube with Spark",
here is the exception log:
-----------------------------------------------------------------------------------------
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
        at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
        at org.apache.hadoop.util.NativeCrc32.calculateChunkedSumsByteArray(NativeCrc32.java:86)
        at org.apache.hadoop.util.DataChecksum.calculateChunkedSums(DataChecksum.java:430)
        at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunks(FSOutputSummer.java:202)
        at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:124)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:110)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
        at java.io.DataOutputStream.write(DataOutputStream.java:107)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:80)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
        at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
        at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:317)
        at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:407)
        at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$5.apply(Client.scala:446)
        at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$5.apply(Client.scala:444)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:444)
        at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:727)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:142)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
        at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:150)
        at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
        at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/06/19 21:22:06 INFO storage.DiskBlockManager: Shutdown hook called
17/06/19 21:22:06 INFO util.ShutdownHookManager: Shutdown hook called
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/httpd-9bcb9a5d-569f-4f28-ad89-038a9020eda8
17/06/19 21:22:06 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-0d1d3709-86cd-446c-b728-5070f168de28/userFiles-2e9ff265-3d37-40e0-8894-6fd4d1a3ad8b

        at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
        at org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:124)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

-----------------------------------------------------------------------------------------
I can use the kylin in-build spark-shell to do some operations like:
-----------------------------------------------------------------------------------------
var textFile = sc.textFile("hdfs://xxxx/xxxx/README.md")
textFile.count()
textFile.first()
textFile.filter(line => line.contains("hello")).count()
-----------------------------------------------------------------------------------------
Here is the env info:
kylin version is 2.0.0
hadoop version is 2.7.*
spark version is 1.6.*
-----------------------------------------------------------------------------------------
Anyone can help me?THX


2017-06-19
skyyws 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message