kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "雨日听风" <491245...@qq.com>
Subject Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Date Tue, 20 Dec 2016 06:51:41 GMT
Thank you! 
We checked the yarn and hard disk. But not found any error. Hard disk space and memory and
so on is working well.
Last time its error code was "unknownhost clusterB",now in new server env it cant find clusterB(hbase
only). but cant find rowCount file.
 ===
the follow command runs ok:
  hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp

And "ls" cant find file "000000_0"  which it said "file does not exist".

  

 ------------------ 原始邮件 ------------------
  发件人: "Alberto Ramón";<a.ramonportoles@gmail.com>;
 发送时间: 2016年12月19日(星期一) 晚上9:13
 收件人: "user"<user@kylin.apache.org>; 
 
 主题: Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist

 

 i think i had this error last nigth  :)  (go to yarn to find detailed error & find on
internet)
 in my case was free space less than 10% of hard disk. Check this please

 
 El 19/12/2016 11:35, "雨日听风" <491245229@qq.com> escribió:
  
When I build a cube in kylin1.6, I get error in step #2: Redistribute Flat Hive Table 
 
Please help! Thank you very much!
 
env: kylin1.6 is in a independent server, and have 2 other server cluster: clusterA(hive only)
and clusterB(hbase only).
Error is:
 
2016-12-19 10:28:00,641 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Compute
row count of flat hive table, cmd: 
2016-12-19 10:28:00,642 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : hive -e
"USE boco;
SET dfs.replication=2;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=100000000;
SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
SET mapreduce.job.split.metainfo.maxsize=-1;
SET mapreduce.job.queuename=young;
SET tez.queue.name=young;
 
set hive.exec.compress.output=false;
 
set hive.exec.compress.output=false;
INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count'
SELECT count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_4b54_bf2b_182e4c86214a;
 
"
2016-12-19 10:28:03,277 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : WARNING:
Use "yarn jar" to launch YARN applications.
2016-12-19 10:28:04,444 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:04,445 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Logging
initialized using configuration in file:/etc/hive/conf/hive-log4j.properties
2016-12-19 10:28:14,700 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:14,703 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken:
0.935 seconds
2016-12-19 10:28:15,559 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Query ID
= young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
2016-12-19 10:28:15,560 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Total jobs
= 1
2016-12-19 10:28:15,575 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Launching
Job 1 out of 1
2016-12-19 10:28:22,842 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:22,842 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Status:
Running (Executing on YARN cluster with App id application_1473415773736_1063281)
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : 
2016-12-19 10:28:23,104 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: -/-
Reducer 2: 0/1 
2016-12-19 10:28:23,307 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2
Reducer 2: 0/1 
2016-12-19 10:28:26,363 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0/2
Reducer 2: 0/1 
2016-12-19 10:28:26,567 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+1)/2
Reducer 2: 0/1 
2016-12-19 10:28:26,596 INFO  [pool-7-thread-1] threadpool.DefaultScheduler:118 : Job Fetcher:
1 should running, 1 actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
2016-12-19 10:28:26,769 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2
Reducer 2: 0/1 
2016-12-19 10:28:29,810 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 0(+2)/2
Reducer 2: 0/1 
2016-12-19 10:28:30,217 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 1(+1)/2
Reducer 2: 0(+1)/1 
2016-12-19 10:28:30,826 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2
Reducer 2: 0(+1)/1 
2016-12-19 10:28:31,232 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Map 1: 2/2
Reducer 2: 1/1 
2016-12-19 10:28:31,319 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Moving data
to: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count
2016-12-19 10:28:31,406 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : OK
2016-12-19 10:28:31,454 INFO  [pool-8-thread-7] execution.AbstractExecutable:36 : Time taken:
16.701 seconds
2016-12-19 10:28:35,074 ERROR [pool-8-thread-7] execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
execute finished with exception
java.io.FileNotFoundException: File does not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
 
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
 at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
 at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
 at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1242)
 at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1227)
 at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1215)
 at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:303)
 at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:269)
 at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261)
 at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
 at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:303)
 at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
 at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:299)
 at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
 at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.readRowCountFromFile(HiveMRInput.java:294)
 at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:337)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
 at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
 at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
 at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does
not exist: /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
 at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1820)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1791)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1704)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
 
 at org.apache.hadoop.ipc.Client.call(Client.java:1427)
 at org.apache.hadoop.ipc.Client.call(Client.java:1358)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
 at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
 at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
 at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
 at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1240)
 ... 20 more
2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output,
id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7] hbase.HBaseResourceStore:262 : Update row
/execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01 from oldTs: 1482114480638, to newTs:
1482114515078, operation result: true
2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 : updating job output,
id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
Mime
View raw message