kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From macwork <15554329...@163.com>
Subject spark构建出现错误从网络上没有找到答案,能否提供帮助谢谢
Date Tue, 30 Jan 2018 11:09:58 GMT
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/tmp/nm-local-dir/filecache/13/spark-libs.jar/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/software/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for TERM
18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for HUP
18/01/30 19:05:28 INFO util.SignalUtils: Registered signal handler for INT
18/01/30 19:05:29 INFO yarn.ApplicationMaster: Preparing Local resources
18/01/30 19:05:30 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1516010594436_0185_000002
18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls to: hadoop
18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls groups to: 
18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls groups to: 
18/01/30 19:05:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set();
users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/01/30 19:05:30 INFO yarn.ApplicationMaster: Starting the user application in a separate
Thread
18/01/30 19:05:30 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
18/01/30 19:05:30 INFO spark.SparkContext: Running Spark version 2.1.2
18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls to: hadoop
18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/01/30 19:05:30 INFO spark.SecurityManager: Changing view acls groups to: 
18/01/30 19:05:30 INFO spark.SecurityManager: Changing modify acls groups to: 
18/01/30 19:05:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set();
users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/01/30 19:05:31 INFO util.Utils: Successfully started service 'sparkDriver' on port 42339.
18/01/30 19:05:31 INFO spark.SparkEnv: Registering MapOutputTracker
18/01/30 19:05:31 INFO spark.SparkEnv: Registering BlockManagerMaster
18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper
for getting topology information
18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/01/30 19:05:31 INFO storage.DiskBlockManager: Created local directory at /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/blockmgr-e9044b90-ff87-4ad3-9ae7-cf85a131ba4e
18/01/30 19:05:31 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
18/01/30 19:05:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/01/30 19:05:31 INFO util.log: Logging initialized @3601ms
18/01/30 19:05:31 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/01/30 19:05:31 INFO server.Server: jetty-9.2.z-SNAPSHOT
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1727271{/jobs,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a8c27d3{/jobs/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@201b73c{/jobs/job,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6dc9732e{/jobs/job/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7f3a751c{/stages,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@44336797{/stages/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4b84807a{/stages/stage,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6d72da27{/stages/stage/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41ccc000{/stages/pool,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1e1c0871{/stages/pool/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@71f4dc98{/storage,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a2c9363{/storage/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2d479ee6{/storage/rdd,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@53e654bf{/storage/rdd/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4215247{/environment,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6af87e2b{/environment/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7bd97d3c{/executors,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1afb9d54{/executors/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14c64726{/executors/threadDump,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3de67ab3{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@d66c0ae{/static,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4bb1ed6b{/,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6de87c4c{/api,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6355a0a0{/jobs/job/kill,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@580d6fd4{/stages/stage/kill,null,AVAILABLE,@Spark}
18/01/30 19:05:31 INFO server.ServerConnector: Started Spark@36b4f4c5{HTTP/1.1}{0.0.0.0:37939}
18/01/30 19:05:31 INFO server.Server: Started @3740ms
18/01/30 19:05:31 INFO util.Utils: Successfully started service 'SparkUI' on port 37939.
18/01/30 19:05:31 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.184:37939
18/01/30 19:05:31 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
18/01/30 19:05:31 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services
with app application_1516010594436_0185 and attemptId Some(appattempt_1516010594436_0185_000002)
18/01/30 19:05:31 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService'
on port 36754.
18/01/30 19:05:31 INFO netty.NettyBlockTransferService: Server created on 192.168.1.184:36754
18/01/30 19:05:31 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy
for block replication policy
18/01/30 19:05:31 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver,
192.168.1.184, 36754, None)
18/01/30 19:05:31 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.184:36754
with 366.3 MB RAM, BlockManagerId(driver, 192.168.1.184, 36754, None)
18/01/30 19:05:31 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver,
192.168.1.184, 36754, None)
18/01/30 19:05:31 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver,
192.168.1.184, 36754, None)
18/01/30 19:05:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@57ef634b{/metrics/json,null,AVAILABLE,@Spark}
18/01/30 19:05:32 INFO scheduler.EventLoggingListener: Logging events to hdfs:///kylin/spark-history/application_1516010594436_0185_2
18/01/30 19:05:32 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster
registered as NettyRpcEndpointRef(spark://YarnAM@192.168.1.184:42339)
18/01/30 19:05:32 INFO yarn.ApplicationMaster: 
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
    SPARK_YARN_STAGING_DIR -> hdfs://192.168.1.171:9000/user/hadoop/.sparkStaging/application_1516010594436_0185
    SPARK_USER -> hadoop
    SPARK_YARN_MODE -> true

  command:
    {{JAVA_HOME}}/bin/java \ 
      -server \ 
      -Xmx1024m \ 
      '-Dhdp.version=current' \ 
      -Djava.io.tmpdir={{PWD}}/tmp \ 
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ 
      -XX:OnOutOfMemoryError='kill %p' \ 
      org.apache.spark.executor.CoarseGrainedExecutorBackend \ 
      --driver-url \ 
      spark://CoarseGrainedScheduler@192.168.1.184:42339 \ 
      --executor-id \ 
      <executorId> \ 
      --hostname \ 
      <hostname> \ 
      --cores \ 
      2 \ 
      --app-id \ 
      application_1516010594436_0185 \ 
      --user-class-path \ 
      file:$PWD/__app__.jar \ 
      --user-class-path \ 
      file:$PWD/htrace-core-3.1.0-incubating.jar \ 
      --user-class-path \ 
      file:$PWD/metrics-core-2.2.0.jar \ 
      --user-class-path \ 
      file:$PWD/guava-12.0.1.jar \ 
      1><LOG_DIR>/stdout \ 
      2><LOG_DIR>/stderr

  resources:
    htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "192.168.1.171"
port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/htrace-core-3.1.0-incubating.jar"
} size: 1475955 timestamp: 1517310297655 type: FILE visibility: PRIVATE
    __app__.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/kylin-job-2.2.0.jar"
} size: 16695259 timestamp: 1517310297574 type: FILE visibility: PRIVATE
    guava-12.0.1.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file:
"/user/hadoop/.sparkStaging/application_1516010594436_0185/guava-12.0.1.jar" } size: 1795932
timestamp: 1517310297780 type: FILE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file:
"/user/hadoop/.sparkStaging/application_1516010594436_0185/__spark_conf__.zip" } size: 86203
timestamp: 1517310297898 type: ARCHIVE visibility: PRIVATE
    metrics-core-2.2.0.jar -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000
file: "/user/hadoop/.sparkStaging/application_1516010594436_0185/metrics-core-2.2.0.jar" }
size: 82123 timestamp: 1517310297714 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "192.168.1.171" port: 9000 file:
"/kylin/spark/spark-libs.jar" } size: 200781646 timestamp: 1517295215668 type: ARCHIVE visibility:
PUBLIC

===============================================================================
18/01/30 19:05:32 INFO client.RMProxy: Connecting to ResourceManager at /192.168.1.171:8030
18/01/30 19:05:32 INFO yarn.YarnRMClient: Registering the ApplicationMaster
18/01/30 19:05:32 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with
2 core(s) and 1408 MB memory (including 384 MB of overhead)
18/01/30 19:05:32 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
18/01/30 19:05:32 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat
: 3000, initial allocation : 200) intervals
18/01/30 19:05:32 INFO impl.AMRMClientImpl: Received new token for : sx-hadoop-dn02:32911
18/01/30 19:05:32 INFO yarn.YarnAllocator: Launching container container_1516010594436_0185_02_000002
on host sx-hadoop-dn02
18/01/30 19:05:32 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors
on 1 of them.
18/01/30 19:05:32 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies
: 0
18/01/30 19:05:32 INFO impl.ContainerManagementProtocolProxy: Opening proxy : sx-hadoop-dn02:32911
18/01/30 19:05:36 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor
NettyRpcEndpointRef(null) (192.168.1.181:54402) with ID 1
18/01/30 19:05:36 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for
scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/01/30 19:05:36 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
18/01/30 19:05:36 INFO storage.BlockManagerMasterEndpoint: Registering block manager sx-hadoop-dn02:42882
with 366.3 MB RAM, BlockManagerId(1, sx-hadoop-dn02, 42882, None)
18/01/30 19:05:36 INFO common.AbstractHadoopJob: Ready to load KylinConfig from uri: kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
18/01/30 19:05:36 INFO cube.CubeManager: Initializing CubeManager with config kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
18/01/30 19:05:36 INFO persistence.ResourceStore: Using metadata url kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
for resource store
18/01/30 19:05:36 INFO hdfs.HDFSResourceStore: hdfs meta path : hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
18/01/30 19:05:36 INFO cube.CubeManager: Loading Cube from folder hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a/cube
18/01/30 19:05:37 INFO cube.CubeDescManager: Initializing CubeDescManager with config kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
18/01/30 19:05:37 INFO cube.CubeDescManager: Reloading Cube Metadata from folder hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a/cube_desc
18/01/30 19:05:37 INFO project.ProjectManager: Initializing ProjectManager with metadata url
kylin_metadata@hdfs,path=hdfs://192.168.1.171:9000/kylin/kylin_metadata/metadata/c3468331-ab55-4368-b691-bf08e2b5fd2a
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: Checking custom measure types from kylin
config
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering COUNT_DISTINCT(hllc), class
org.apache.kylin.measure.hllc.HLLCMeasureType$Factory
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering COUNT_DISTINCT(bitmap), class
org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering EXTENDED_COLUMN(extendedcolumn),
class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory
18/01/30 19:05:37 INFO measure.MeasureTypeFactory: registering PERCENTILE(percentile), class
org.apache.kylin.measure.percentile.PercentileMeasureType$Factory
18/01/30 19:05:37 INFO metadata.MetadataManager: Reloading data model at /model_desc/kylin_sales_model.json
18/01/30 19:05:37 INFO cube.CubeDescManager: Loaded 1 Cube(s)
18/01/30 19:05:37 INFO cube.CubeManager: Reloaded cube kylin_sales_cube_clone being CUBE[name=kylin_sales_cube_clone]
having 2 segments
18/01/30 19:05:37 INFO cube.CubeManager: Loaded 1 cubes, fail on 0 cubes
18/01/30 19:05:37 INFO spark.SparkCubingByLayer: RDD Output path: hdfs://192.168.1.171:9000/kylin/kylin_metadata/kylin-5b2617ee-90ec-4c4b-86ff-699d0f14223e/kylin_sales_cube_clone/cuboid/
18/01/30 19:05:37 INFO spark.SparkCubingByLayer: All measure are normal (agg on all cuboids)
? : true
18/01/30 19:05:38 INFO internal.SharedState: Warehouse path is 'file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/spark-warehouse'.
18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@75fae875{/SQL,null,AVAILABLE,@Spark}
18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3785cc60{/SQL/json,null,AVAILABLE,@Spark}
18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6299f0a5{/SQL/execution,null,AVAILABLE,@Spark}
18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@578d227{/SQL/execution/json,null,AVAILABLE,@Spark}
18/01/30 19:05:38 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@29f2a18{/static/sql,null,AVAILABLE,@Spark}
18/01/30 19:05:38 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.2.1
using Spark classes.
18/01/30 19:05:38 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/01/30 19:05:39 INFO metastore.ObjectStore: ObjectStore, initialize called
18/01/30 19:05:39 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown
unknown - will be ignored
18/01/30 19:05:39 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown
- will be ignored
18/01/30 19:05:45 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/01/30 19:05:47 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
18/01/30 19:05:47 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
18/01/30 19:05:51 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema"
is tagged as "embedded-only" so does not have its own datastore table.
18/01/30 19:05:51 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder"
is tagged as "embedded-only" so does not have its own datastore table.
18/01/30 19:05:52 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/01/30 19:05:52 INFO metastore.ObjectStore: Initialized ObjectStore
18/01/30 19:05:52 WARN metastore.ObjectStore: Version information not found in metastore.
hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/01/30 19:05:52 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
18/01/30 19:05:53 INFO metastore.HiveMetaStore: Added admin role in metastore
18/01/30 19:05:53 INFO metastore.HiveMetaStore: Added public role in metastore
18/01/30 19:05:53 INFO metastore.HiveMetaStore: No user is added in admin role, since config
is empty
18/01/30 19:05:53 INFO metastore.HiveMetaStore: 0: get_all_databases
18/01/30 19:05:53 INFO HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_all_databases

18/01/30 19:05:53 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
18/01/30 19:05:53 INFO HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_functions:
db=default pat=*	
18/01/30 19:05:53 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri"
is tagged as "embedded-only" so does not have its own datastore table.
18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/hadoop
18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5_resources
18/01/30 19:05:54 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5
18/01/30 19:05:54 INFO session.SessionState: Created local directory: /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/tmp/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5
18/01/30 19:05:54 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/4990ced4-7b5c-46d6-a0f6-9ecfdcbc22c5/_tmp_space.db
18/01/30 19:05:54 INFO client.HiveClientImpl: Warehouse location for Hive client (version
1.2.1) is file:/home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/container_1516010594436_0185_02_000001/spark-warehouse
18/01/30 19:05:54 INFO metastore.HiveMetaStore: 0: get_database: default
18/01/30 19:05:54 INFO HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_database:
default	
18/01/30 19:05:54 INFO metastore.HiveMetaStore: 0: get_database: global_temp
18/01/30 19:05:54 INFO HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_database:
global_temp	
18/01/30 19:05:54 WARN metastore.ObjectStore: Failed to get database global_temp, returning
NoSuchObjectException
18/01/30 19:05:54 INFO execution.SparkSqlParser: Parsing command: default.kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a
18/01/30 19:05:55 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a
18/01/30 19:05:55 INFO HiveMetaStore.audit: ugi=hadoop	ip=unknown-ip-addr	cmd=get_table :
db=default tbl=kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a

18/01/30 19:05:55 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException:
error execute org.apache.kylin.engine.spark.SparkCubingByLayer
java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer
	at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
	at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:636)
Caused by: org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'kylin_intermediate_kylin_sales_cube_clone_c3468331_ab55_4368_b691_bf08e2b5fd2a'
not found in database 'default';
	at org.apache.spark.sql.hive.client.HiveClient$$anonfun$getTable$1.apply(HiveClient.scala:74)
	at org.apache.spark.sql.hive.client.HiveClient$$anonfun$getTable$1.apply(HiveClient.scala:74)
	at scala.Option.getOrElse(Option.scala:121)
	at org.apache.spark.sql.hive.client.HiveClient$class.getTable(HiveClient.scala:74)
	at org.apache.spark.sql.hive.client.HiveClientImpl.getTable(HiveClientImpl.scala:78)
	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable$1.apply(HiveExternalCatalog.scala:118)
	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable$1.apply(HiveExternalCatalog.scala:118)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
	at org.apache.spark.sql.hive.HiveExternalCatalog.org$apache$spark$sql$hive$HiveExternalCatalog$$getRawTable(HiveExternalCatalog.scala:117)
	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:628)
	at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$getTable$1.apply(HiveExternalCatalog.scala:628)
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
	at org.apache.spark.sql.hive.HiveExternalCatalog.getTable(HiveExternalCatalog.scala:627)
	at org.apache.spark.sql.hive.HiveMetastoreCatalog.lookupRelation(HiveMetastoreCatalog.scala:124)
	at org.apache.spark.sql.hive.HiveSessionCatalog.lookupRelation(HiveSessionCatalog.scala:70)
	at org.apache.spark.sql.SparkSession.table(SparkSession.scala:586)
	at org.apache.spark.sql.SparkSession.table(SparkSession.scala:582)
	at org.apache.spark.sql.SQLContext.table(SQLContext.scala:708)
	at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:167)
	at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
	... 6 more
18/01/30 19:05:55 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason:
User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer)
18/01/30 19:05:55 INFO spark.SparkContext: Invoking stop() from shutdown hook
18/01/30 19:05:55 INFO server.ServerConnector: Stopped Spark@36b4f4c5{HTTP/1.1}{0.0.0.0:0}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@580d6fd4{/stages/stage/kill,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6355a0a0{/jobs/job/kill,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6de87c4c{/api,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4bb1ed6b{/,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@d66c0ae{/static,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@3de67ab3{/executors/threadDump/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@14c64726{/executors/threadDump,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1afb9d54{/executors/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7bd97d3c{/executors,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6af87e2b{/environment/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4215247{/environment,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@53e654bf{/storage/rdd/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@2d479ee6{/storage/rdd,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@a2c9363{/storage/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@71f4dc98{/storage,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1e1c0871{/stages/pool/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@41ccc000{/stages/pool,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6d72da27{/stages/stage/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@4b84807a{/stages/stage,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@44336797{/stages/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@7f3a751c{/stages,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@6dc9732e{/jobs/job/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@201b73c{/jobs/job,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@5a8c27d3{/jobs/json,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO handler.ContextHandler: Stopped o.s.j.s.ServletContextHandler@1727271{/jobs,null,UNAVAILABLE,@Spark}
18/01/30 19:05:55 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.184:37939
18/01/30 19:05:55 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
18/01/30 19:05:55 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
18/01/30 19:05:55 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor
to shut down
18/01/30 19:05:55 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
18/01/30 19:05:55 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint
stopped!
18/01/30 19:05:55 INFO memory.MemoryStore: MemoryStore cleared
18/01/30 19:05:55 INFO storage.BlockManager: BlockManager stopped
18/01/30 19:05:55 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/01/30 19:05:55 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
18/01/30 19:05:55 INFO spark.SparkContext: Successfully stopped SparkContext
18/01/30 19:05:55 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED
(diag message: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer)
18/01/30 19:05:55 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
18/01/30 19:05:55 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://192.168.1.171:9000/user/hadoop/.sparkStaging/application_1516010594436_0185
18/01/30 19:05:55 INFO util.ShutdownHookManager: Shutdown hook called
18/01/30 19:05:55 INFO util.ShutdownHookManager: Deleting directory /home/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1516010594436_0185/spark-502e0062-d08e-47a0-9663-4b284f141d3f


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message