kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Manoj H" <manoj.h.ku...@jpmorgan.com>
Subject RE: apache kylin 2.1 - Spark Cube Building
Date Wed, 07 Feb 2018 04:21:17 GMT
Yes I did that.. I will come back with more logs.

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org]
Sent: Wednesday, February 07, 2018 6:46 AM
To: user <user@kylin.apache.org>
Subject: Re: apache kylin 2.1 - Spark Cube Building

The default configuration for spark is very small; You need to tweak some parameters (like
below) or enable Spark dynamic resource allocation;

kylin.engine.spark-conf.spark.executor.memory=1G
kylin.engine.spark-conf.spark.executor.cores=2
kylin.engine.spark-conf.spark.executor.instances=1
If you already took these actions, the performance still not good, then you need a deep tunning.

2018-02-06 12:43 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
While running Spark Cube process, I noticed that this is taking other Cube tables into the
consideration , Rather it should take the cube which it isdoing. Not sure why its taking data
model of other cubes. Normally its being noticed that Spark is taking almost same time as
Maprecuce is taking. We assume that Spark should be faster than MR jobs.



2018-02-05 23:35:07,825 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
: 18/02/05 23:35:07 WARN CubeDescManager: Broken cube desc /cube_desc/FRI_CUBE_update.json
2018-02-05 23:35:07,825 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
: java.lang.IllegalArgumentException: Table not found by LOAN_POSITION_BAL_009_SS1
2018-02-05 23:35:07,825 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.metadata.model.DataModelDesc.findTable(DataModelDesc.java:314)
2018-02-05 23:35:07,826 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.model.DimensionDesc.init(DimensionDesc.java:61)
2018-02-05 23:35:07,826 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:587)
2018-02-05 23:35:07,826 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:196)
2018-02-05 23:35:07,826 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:321)
2018-02-05 23:35:07,826 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:114)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:81)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:811)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:789)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:147)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:105)
2018-02-05 23:35:07,827 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:161)
2018-02-05 23:35:07,828 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
2018-02-05 23:35:07,828 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
2018-02-05 23:35:07,828 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2018-02-05 23:35:07,828 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2018-02-05 23:35:07,828 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2018-02-05 23:35:07,829 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at java.lang.reflect.Method.invoke(Method.java:606)
2018-02-05 23:35:07,829 INFO  [Job 53f2a470-2973-46ad-9d97-8ddcec1933cc-315] spark.SparkExecutable:38
:         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$r

Regards,
Manoj


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi 史少锋


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.
Mime
View raw message