kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "雨日听风" <491245...@qq.com>
Subject Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Date Mon, 19 Dec 2016 10:35:34 GMT
When I build a cube in kylin1.6, I get error in step #2: Redistribute Flat Hive Table 

Please help! Thank you very much!

env: kylin1.6 is in a independent server, and have 2 other server cluster: clusterA(hive only)
and clusterB(hbase only).
Error is:

2016-12-19 10:28:00,641 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Compute
row count of flat hive table, cmd: 
2016-12-19 10:28:00,642 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : hive -e
"USE boco;
SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapreduce.job.queuename=young;
SET tez.queue.name=young;

set hive.exec.compress.output=false;

set hive.exec.compress.output=false;
INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count'
SELECT count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a;

"
2016-12-19 10:28:03,277 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : WARNING:
Use "yarn jar" to launch YARN applications.
2016-12-19 10:28:04,444 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:04,445 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Logging
initialized using configuration in file:/etc/hive/conf/hive-log4j.properties
2016-12-19 10:28:14,700 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:14,703 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken:
0.935 seconds
2016-12-19 10:28:15,559 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Query ID
= young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
2016-12-19 10:28:15,560 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Total jobs
= 1
2016-12-19 10:28:15,575 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Launching
Job 1 out of 1
2016-12-19 10:28:22,842 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:22,842 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Status:
Running (Executing on YARN cluster with App id application_1473415773736_1063281)
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: -/-
Reducer 2: 0/1 
2016-12-19 10:28:23,307 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2
Reducer 2: 0/1 
2016-12-19 10:28:26,363 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2
Reducer 2: 0/1 
2016-12-19 10:28:26,567 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+1)/2
Reducer 2: 0/1 
2016-12-19 10:28:26,596 INFO  [pool-7-thread-1] threadpool.DefaultScheduler:118 : Job Fetcher:
1 should running, 1 actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
2016-12-19 10:28:26,769 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2
Reducer 2: 0/1 
2016-12-19 10:28:29,810 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2
Reducer 2: 0/1 
2016-12-19 10:28:30,217 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 1(+1)/2
Reducer 2: 0(+1)/1 
2016-12-19 10:28:30,826 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2
Reducer 2: 0(+1)/1 
2016-12-19 10:28:31,232 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2
Reducer 2: 1/1 
2016-12-19 10:28:31,319 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Moving data
to: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count
2016-12-19 10:28:31,406 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:31,454 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken:
16.701 seconds
2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
execute finished with exception
java.io.FileNotFoundException: File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)

 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
 at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
 at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242)
 at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227)
 at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215)
 at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303)
 at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269)
 at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261)
 at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
 at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:303)
 at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:299)
 at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
 at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.readRowCountFromFile(HiveMRInput.java:294)
 at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:337)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
 at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
 at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does
not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)

 at org.apache.hadoop.ipc.Client.call(Client.java:1427)
 at org.apache.hadoop.ipc.Client.call(Client.java:1358)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
 at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
 at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
 at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240)
 ... 20 more
2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output,
id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] hbase.HBaseResourceStore:262 : Update row
/execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 from oldTs: 1482114480638, to newTs:
1482114515078, operation result: true
2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output,
id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
Mime
View raw message