kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: spark on yarn is fine,but not kylin.
Date Fri, 17 Nov 2017 11:05:59 GMT
Hi,

To check why Spark is failed, please start Spark history server and then
check the detail executor logs; The tutorial has the guide on how to do
this:
https://kylin.apache.org/docs21/tutorial/cube_spark.html

2017-11-17 18:48 GMT+08:00 Prasanna <prasanna.p@trinitymobility.com>:

> Hi,
>
> I tried spark-shell in yarn mode,its working fine. If I am using kylin
> cube build on spark engine its failing at 7th step #7 Step Name: Build
> Cube with Spark . I am not able to understand why its failing. Yarn job is
> starting but after 10.0% completion its failing.
>
>
>
> application_1510896792536_0042
> <http://192.168.1.135:8088/cluster/app/application_1510896792536_0042>
>
> hdfs
>
> org.apache.kylin.common.util.SparkEntry
>
> SPARK
>
> default
>
> Fri Nov 17 16:02:03 +0550 2017
>
> Fri Nov 17 16:02:44 +0550 2017
>
> FINISHED
>
> FAILED
>
>
>
>
>
> These are the logs i am getting in kylin , Please go through this and
> suggest me what is my mistake. Give me suggestions as early as possible.
>
>
>
> [image: http://192.168.1.135:7070/kylin/image/logo.png] Kylin
> <http://192.168.1.135:7070/kylin/query>
>
> ·
>
> test_sample
>
> o    test_sample
>
> o    -- Choose Project --
>
> ·         Insight <http://192.168.1.135:7070/kylin/query>
>
> ·         Model <http://192.168.1.135:7070/kylin/models>
>
> ·         Monitor <http://192.168.1.135:7070/kylin/jobs>
>
> ·         System <http://192.168.1.135:7070/kylin/admin>
>
> ·         Welcome, ADMIN  <http://192.168.1.135:7070/kylin/>
>
>
>
>    - Jobs
>    - Slow Queries
>
> *Cube Name:*
>
> Jobs in:  NEWPENDINGRUNNINGSTOPPEDFINISHEDERRORDISCARDED
>
> *Job Name *
>
> *Cube *
>
> *Progress*
>
> *Last Modified Time *
>
> *Duration*
>
> *Actions*
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171117140000 - GMT+08:00
> 2017-11-17 17:50:28
>
> test_sample_cube
>
> ERROR
>
> 2017-11-17 18:32:45 GMT+8
>
> 9.62 mins
>
> Action
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171117090000 - GMT+08:00
> 2017-11-17 13:35:05
>
> test_sample_cube
>
> 54.55%
>
> 2017-11-17 17:50:12 GMT+8
>
> 9.80 mins
>
> Action
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171117090000 - GMT+08:00
> 2017-11-17 12:17:00
>
> test_sample_cube
>
> 54.55%
>
> 2017-11-17 13:34:52 GMT+8
>
> 62.88 mins
>
> Action
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171116120000 - GMT+08:00
> 2017-11-16 21:23:15
>
> test_sample_cube
>
> 54.55%
>
> 2017-11-17 12:16:42 GMT+8
>
> 5.88 mins
>
> Action
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171114140000 - GMT+08:00
> 2017-11-15 21:02:27
>
> test_sample_cube
>
> 54.55%
>
> 2017-11-16 21:23:03 GMT+8
>
> 12.78 mins
>
> Action
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171115120000 - GMT+08:00
> 2017-11-15 20:34:06
>
> test_sample_cube
>
> 54.55%
>
> 2017-11-15 21:02:00 GMT+8
>
> 7.15 mins
>
> Action
>
> *Total: 6*
>
> ·          Detail Information
>
> *Job Name*
>
> BUILD CUBE - test_sample_cube - 20160101120000_20171117140000 - GMT+08:00
> 2017-11-17 17:50:28
>
> *Job ID*
>
> 15e7edc3-e6b6-4a67-954d-7458dca8ca94
>
> *Status*
>
> *ERROR*
>
> *Duration*
>
> 9.62 mins
>
> *MapReduce Waiting*
>
> 0.32 mins
>
> ·         *Start   2017-11-17 17:50:43 GMT+8*
>
> ·          2017-11-17 17:50:43 GMT+8
>
> #1 Step Name: Create Intermediate Flat Hive Table
> Duration: 0.73 mins Waiting: 0 seconds
>
> ·          2017-11-17 17:51:27 GMT+8
>
> #2 Step Name: Redistribute Flat Hive Table
> Duration: 0.56 mins Waiting: 0 seconds
>
> ·          2017-11-17 17:52:01 GMT+8
>
> #3 Step Name: Extract Fact Table Distinct Columns
>
> Data Size: 18.00 MB
>
> Duration: 3.34 mins Waiting: 19 seconds
>
>
>
> ·          2017-11-17 17:55:21 GMT+8
>
> #4 Step Name: Build Dimension Dictionary
> Duration: 0.07 mins Waiting: 0 seconds
>
>
>
> ·          2017-11-17 17:55:25 GMT+8
>
> #5 Step Name: Save Cuboid Statistics
> Duration: 0.04 mins Waiting: 0 seconds
>
> ·          2017-11-17 17:55:28 GMT+8
>
> #6 Step Name: Create HTable
> Duration: 0.07 mins Waiting: 0 seconds
>
>
>
> ·          2017-11-17 18:31:51 GMT+8
>
> #7 Step Name: Build Cube with Spark
> Duration: 0.90 mins Waiting: 0 seconds
>
> ·         #8 Step Name: Convert Cuboid Data to HFile
> Duration: 0 seconds Waiting: 0 seconds
>
> ·         #9 Step Name: Load HFile to HBase Table
> Duration: 0 seconds Waiting: 0 seconds
>
> ·         #10 Step Name: Update Cube Info
> Duration: 0 seconds Waiting: 0 seconds
>
> ·         #11 Step Name: Hive Cleanup
> Duration: 0 seconds Waiting: 0 seconds
>
> ·         *End   *
>
>  Apache Kylin <http://kylin.apache.org/> |  Apache Kylin Community
> <http://kylin.apache.org/community/>
> Output
>
> OS command error exit with return code: 1, error message: Ivy Default Cache set to: /home/hdfs/.ivy2/cache
>
> The jars for the packages stored in: /home/hdfs/.ivy2/jars
>
> :: loading settings :: url = jar:file:/usr/local/kylin/spark/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
>
> com.databricks#spark-csv_2.11 added as a dependency
>
> :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
>
>          confs: [default]
>
>          found com.databricks#spark-csv_2.11;1.4.0 in central
>
>          found org.apache.commons#commons-csv;1.1 in central
>
>          found com.univocity#univocity-parsers;1.5.1 in central
>
> :: resolution report :: resolve 277ms :: artifacts dl 7ms
>
>          :: modules in use:
>
>          com.databricks#spark-csv_2.11;1.4.0 from central in [default]
>
>          com.univocity#univocity-parsers;1.5.1 from central in [default]
>
>          org.apache.commons#commons-csv;1.1 from central in [default]
>
>          ---------------------------------------------------------------------
>
>          |                  |            modules            ||   artifacts   |
>
>          |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
>
>          ---------------------------------------------------------------------
>
>          |      default     |   3   |   0   |   0   |   0   ||   3   |   0   |
>
>          ---------------------------------------------------------------------
>
> :: retrieving :: org.apache.spark#spark-submit-parent
>
>          confs: [default]
>
>          0 artifacts copied, 3 already retrieved (0kB/8ms)
>
> 17/11/17 16:01:59 INFO client.RMProxy: Connecting to ResourceManager at master01.trinitymobility.local/192.168.1.135:8032
>
> 17/11/17 16:01:59 INFO yarn.Client: Requesting a new application from cluster with 1
NodeManagers
>
> 17/11/17 16:01:59 INFO yarn.Client: Verifying our application has not requested more
than the maximum memory capability of the cluster (4608 MB per container)
>
> 17/11/17 16:01:59 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including
384 MB overhead
>
> 17/11/17 16:01:59 INFO yarn.Client: Setting up container launch context for our AM
>
> 17/11/17 16:01:59 INFO yarn.Client: Setting up the launch environment for our AM container
>
> 17/11/17 16:01:59 INFO yarn.Client: Preparing resources for our AM container
>
> 17/11/17 16:02:00 INFO yarn.Client: Source and destination file systems are the same.
Not copying hdfs://trinitybdhdfs/kylin/spark/spark-libs.jar
>
> 17/11/17 16:02:00 INFO yarn.Client: Uploading resource file:/usr/local/kylin/lib/kylin-job-2.2.0.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/kylin-job-2.2.0.jar
>
> 17/11/17 16:02:02 INFO yarn.Client: Uploading resource file:/usr/local/kylin/spark/jars/htrace-core-3.0.4.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/htrace-core-3.0.4.jar
>
> 17/11/17 16:02:02 INFO yarn.Client: Uploading resource file:/usr/hdp/2.4.3.0-227/hbase/lib/htrace-core-3.1.0-incubating.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/htrace-core-3.1.0-incubating.jar
>
> 17/11/17 16:02:02 INFO yarn.Client: Uploading resource file:/usr/hdp/2.4.3.0-227/hbase/lib/metrics-core-2.2.0.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/metrics-core-2.2.0.jar
>
> 17/11/17 16:02:02 INFO yarn.Client: Uploading resource file:/usr/hdp/2.4.3.0-227/hbase/lib/guava-12.0.1.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/guava-12.0.1.jar
>
> 17/11/17 16:02:03 INFO yarn.Client: Uploading resource file:/home/hdfs/.ivy2/jars/com.databricks_spark-csv_2.11-1.4.0.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/com.databricks_spark-csv_2.11-1.4.0.jar
>
> 17/11/17 16:02:03 INFO yarn.Client: Uploading resource file:/home/hdfs/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/org.apache.commons_commons-csv-1.1.jar
>
> 17/11/17 16:02:03 INFO yarn.Client: Uploading resource file:/home/hdfs/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/com.univocity_univocity-parsers-1.5.1.jar
>
> 17/11/17 16:02:03 INFO yarn.Client: Uploading resource file:/tmp/spark-5f534e08-aaa8-45ea-afd6-bc6568665e5a/__spark_conf__8429925006438057896.zip
-> hdfs://trinitybdhdfs/user/hdfs/.sparkStaging/application_1510896792536_0042/__spark_conf__.zip
>
> 17/11/17 16:02:03 WARN yarn.Client: spark.yarn.am.extraJavaOptions will not take effect
in cluster mode
>
> 17/11/17 16:02:03 INFO spark.SecurityManager: Changing view acls to: hdfs
>
> 17/11/17 16:02:03 INFO spark.SecurityManager: Changing modify acls to: hdfs
>
> 17/11/17 16:02:03 INFO spark.SecurityManager: Changing view acls groups to:
>
> 17/11/17 16:02:03 INFO spark.SecurityManager: Changing modify acls groups to:
>
> 17/11/17 16:02:03 INFO spark.SecurityManager: SecurityManager: authentication disabled;
ui acls disabled; users  with view permissions: Set(hdfs); groups with view permissions: Set();
users  with modify permissions: Set(hdfs); groups with modify permissions: Set()
>
> 17/11/17 16:02:03 INFO yarn.Client: Submitting application application_1510896792536_0042
to ResourceManager
>
> 17/11/17 16:02:03 INFO impl.YarnClientImpl: Submitted application application_1510896792536_0042
>
> 17/11/17 16:02:04 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:04 INFO yarn.Client:
>
>           client token: N/A
>
>           diagnostics: N/A
>
>           ApplicationMaster host: N/A
>
>           ApplicationMaster RPC port: -1
>
>           queue: default
>
>           start time: 1510914723600
>
>           final status: UNDEFINED
>
>           tracking URL: http://master01.trinitymobility.local:8088/proxy/application_1510896792536_0042/
>
>           user: hdfs
>
> 17/11/17 16:02:05 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:06 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:07 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:08 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:09 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:09 INFO yarn.Client:
>
>           client token: N/A
>
>           diagnostics: N/A
>
>           ApplicationMaster host: 192.168.1.135
>
>           ApplicationMaster RPC port: 0
>
>           queue: default
>
>           start time: 1510914723600
>
>           final status: UNDEFINED
>
>           tracking URL: http://master01.trinitymobility.local:8088/proxy/application_1510896792536_0042/
>
>           user: hdfs
>
> 17/11/17 16:02:10 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:11 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:12 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:13 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:14 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:15 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:16 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:17 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:18 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:19 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:20 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:21 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:22 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:23 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:24 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:24 INFO yarn.Client:
>
>           client token: N/A
>
>           diagnostics: N/A
>
>           ApplicationMaster host: N/A
>
>           ApplicationMaster RPC port: -1
>
>           queue: default
>
>           start time: 1510914723600
>
>           final status: UNDEFINED
>
>           tracking URL: http://master01.trinitymobility.local:8088/proxy/application_1510896792536_0042/
>
>           user: hdfs
>
> 17/11/17 16:02:25 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:26 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:27 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:28 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:29 INFO yarn.Client: Application report for application_1510896792536_0042
(state: ACCEPTED)
>
> 17/11/17 16:02:30 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:30 INFO yarn.Client:
>
>           client token: N/A
>
>           diagnostics: N/A
>
>           ApplicationMaster host: 192.168.1.135
>
>           ApplicationMaster RPC port: 0
>
>           queue: default
>
>           start time: 1510914723600
>
>           final status: UNDEFINED
>
>           tracking URL: http://master01.trinitymobility.local:8088/proxy/application_1510896792536_0042/
>
>           user: hdfs
>
> 17/11/17 16:02:31 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:32 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:33 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:34 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:35 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:36 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:37 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:38 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:39 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:40 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:41 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:42 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:43 INFO yarn.Client: Application report for application_1510896792536_0042
(state: RUNNING)
>
> 17/11/17 16:02:44 INFO yarn.Client: Application report for application_1510896792536_0042
(state: FINISHED)
>
> 17/11/17 16:02:44 INFO yarn.Client:
>
>           client token: N/A
>
>           diagnostics: User class threw exception: java.lang.RuntimeException: error
execute org.apache.kylin.engine.spark.SparkCubingByLayer
>
>           ApplicationMaster host: 192.168.1.135
>
>           ApplicationMaster RPC port: 0
>
>           queue: default
>
>           start time: 1510914723600
>
>           final status: FAILED
>
>           tracking URL: http://master01.trinitymobility.local:8088/proxy/application_1510896792536_0042/
>
>           user: hdfs
>
> Exception in thread "main" org.apache.spark.SparkException: Application application_1510896792536_0042
finished with failed status
>
>          at org.apache.spark.deploy.yarn.Client.run(Client.scala:1180)
>
>          at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1226)
>
>          at org.apache.spark.deploy.yarn.Client.main(Client.scala)
>
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>          at java.lang.reflect.Method.invoke(Method.java:498)
>
>          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
>
>          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
>
>          at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
>
>          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
>
>          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
> 17/11/17 16:02:44 INFO util.ShutdownHookManager: Shutdown hook called
>
> 17/11/17 16:02:44 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-5f534e08-aaa8-45ea-afd6-bc6568665e5a
>
> The command is:
>
> export HADOOP_CONF_DIR=/usr/local/kylin/hadoop-conf && /usr/local/kylin/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=1  --conf
spark.yarn.archive=hdfs://trinitybdhdfs/kylin/spark/spark-libs.jar  --conf spark.yarn.queue=default
 --conf spark.yarn.am.extraJavaOptions=-Dhdp.version=current  --conf spark.history.fs.logDirectory=hdfs://trinitybdhdfs/kylin/spark-history
 --conf spark.driver.extraJavaOptions=-Dhdp.version=current  --conf spark.master=yarn  --conf
spark.executor.extraJavaOptions=-Dhdp.version=current  --conf spark.hadoop.yarn.timeline-service.enabled=false
 --conf spark.executor.memory=1G  --conf spark.eventLog.enabled=true  --conf spark.eventLog.dir=hdfs://trinitybdhdfs/kylin/spark-history
 --conf spark.executor.cores=2  --conf spark.submit.deployMode=cluster --jars /usr/local/kylin/spark/jars/htrace-core-3.0.4.jar,/usr/hdp/2.4.3.0-227/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.4.3.0-227/hbase/lib/metrics-core-2.2.0.jar,/usr/hdp/2.4.3.0-227/hbase/lib/guava-12.0.1.jar,
/usr/local/kylin/lib/kylin-job-2.2.0.jar -className org.apache.kylin.engine.spark.SparkCubingByLayer
-hiveTable default.kylin_intermediate_test_sample_cube_175ff321_fa23_4fa8_8923_e39e2f37df4f
-output hdfs://trinitybdhdfs/kylin/kylin_metadata/kylin-15e7edc3-e6b6-4a67-954d-7458dca8ca94/test_sample_cube/cuboid/
-segmentId 175ff321-fa23-4fa8-8923-e39e2f37df4f -metaUrl kylin_metadata@hdfs,path=hdfs://trinitybdhdfs/kylin/kylin_metadata/metadata/175ff321-fa23-4fa8-8923-e39e2f37df4f
-cubename test_sample_cube
>
>
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message