kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alberto Ramón <a.ramonporto...@gmail.com>
Subject Re: Error when #2 Step: Redistribute Flat Hive Table - File does not exist
Date Tue, 20 Dec 2016 07:23:32 GMT
other idea:
Can be a problem with permissions?: the user that execute Kylin can't read
data generated by YARN
check if Kylin user can read your folder  /young/kylin_test/
Which Hadoop user are executing Kylin?

(no more ideas, Good Luck)

2016-12-20 7:51 GMT+01:00 雨日听风 <491245229@qq.com>:

> Thank you!
> We checked the yarn and hard disk. But not found any error. Hard disk
> space and memory and so on is working well.
> Last time its error code was "unknownhost clusterB",now in new server env
> it cant find clusterB(hbase only). but cant find rowCount file.
> ===
> the follow command runs ok:
> hdfs dfs -mkdir /young/kylin_test/kylin_metadata_nokia/
> kylin-678c15ba-5375-4f80-831e-1ae0af8ed576/row_count/tmp
> And "ls" cant find file "000000_0"  which it said "file does not exist".
>
> ------------------ 原始邮件 ------------------
> *发件人:* "Alberto Ramón";<a.ramonportoles@gmail.com>;
> *发送时间:* 2016年12月19日(星期一) 晚上9:13
> *收件人:* "user"<user@kylin.apache.org>;
> *主题:* Re: Error when #2 Step: Redistribute Flat Hive Table - File does
> not exist
>
> i think i had this error last nigth  :)
> (go to yarn to find detailed error & find on internet)
> in my case was free space less than 10% of hard disk. Check this please
>
> El 19/12/2016 11:35, "雨日听风" <491245229@qq.com> escribió:
>
>> When I build a cube in kylin1.6, I get error in step #2: Redistribute
>> Flat Hive Table
>>
>> Please help! Thank you very much!
>>
>> env: kylin1.6 is in a independent server, and have 2 other server
>> cluster: clusterA(hive only) and clusterB(hbase only).
>> Error is:
>>
>> 2016-12-19 10:28:00,641 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Compute row count of flat hive table,
>> cmd:
>> 2016-12-19 10:28:00,642 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : hive -e "USE boco;
>> SET dfs.replication=2;
>> SET hive.exec.compress.output=true;
>> SET hive.auto.convert.join.noconditionaltask=true;
>> SET hive.auto.convert.join.noconditionaltask.size=100000000;
>> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
>> SET mapreduce.job.split.metainfo.maxsize=-1;
>> SET mapreduce.job.queuename=young;
>> SET tez.queue.name=young;
>>
>> set hive.exec.compress.output=false;
>>
>> set hive.exec.compress.output=false;
>> INSERT OVERWRITE DIRECTORY '/young/kylin_test/kylin_metad
>> ata_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count' SELECT
>> count(*) FROM kylin_intermediate_hbase_in_testCluster_CUBE_f9468805_eabf_
>> 4b54_bf2b_182e4c86214a;
>>
>> "
>> 2016-12-19 10:28:03,277 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : WARNING: Use "yarn jar" to launch YARN
>> applications.
>> 2016-12-19 10:28:04,444 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:04,445 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Logging initialized using
>> configuration in file:/etc/hive/conf/hive-log4j.properties
>> 2016-12-19 10:28:14,700 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : OK
>> 2016-12-19 10:28:14,703 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Time taken: 0.935 seconds
>> 2016-12-19 10:28:15,559 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Query ID =
>> young_20161219102814_a7104fd4-ba83-47fc-ac0b-0c9bef4e1969
>> 2016-12-19 10:28:15,560 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Total jobs = 1
>> 2016-12-19 10:28:15,575 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Launching Job 1 out of 1
>> 2016-12-19 10:28:22,842 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:22,842 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:23,104 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Status: Running (Executing on YARN
>> cluster with App id application_1473415773736_1063281)
>> 2016-12-19 10:28:23,104 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 :
>> 2016-12-19 10:28:23,104 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: -/- Reducer 2: 0/1
>> 2016-12-19 10:28:23,307 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,363 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,567 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+1)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:26,596 INFO  [pool-7-thread-1]
>> threadpool.DefaultScheduler:118 : Job Fetcher: 1 should running, 1
>> actual running, 0 ready, 0 already succeed, 3 error, 1 discarded, 0 others
>> 2016-12-19 10:28:26,769 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:29,810 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 0(+2)/2 Reducer 2: 0/1
>> 2016-12-19 10:28:30,217 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 1(+1)/2 Reducer 2: 0(+1)/1
>> 2016-12-19 10:28:30,826 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 0(+1)/1
>> 2016-12-19 10:28:31,232 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Map 1: 2/2 Reducer 2: 1/1
>> 2016-12-19 10:28:31,319 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Moving data to:
>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-
>> 48b4-bdb5-6e578320375a/row_count
>> 2016-12-19 10:28:31,406 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : OK
>> 2016-12-19 10:28:31,454 INFO  [pool-8-thread-7]
>> execution.AbstractExecutable:36 : Time taken: 16.701 seconds
>> 2016-12-19 10:28:35,074 ERROR [pool-8-thread-7]
>> execution.AbstractExecutable:357 : job:678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> execute finished with exception
>> java.io.FileNotFoundException: File does not exist:
>> /young/kylin_test/kylin_metadata_test/kylin-678266c0-ba0e-
>> 48b4-bdb5-6e578320375a/row_count/000000_0
>>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:71)
>>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:61)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> LocationsInt(FSNamesystem.java:1820)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1791)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1704)
>>  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>> BlockLocations(NameNodeRpcServer.java:587)
>>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>> erverSideTranslatorPB.java:365)
>>  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:616)
>>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>>  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1657)
>>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>
>>  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>  at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native
>> ConstructorAccessorImpl.java:57)
>>  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De
>> legatingConstructorAccessorImpl.java:45)
>>  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>  at org.apache.hadoop.ipc.RemoteException.instantiateException(R
>> emoteException.java:106)
>>  at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
>> RemoteException.java:73)
>>  at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>> ient.java:1242)
>>  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>> java:1227)
>>  at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.
>> java:1215)
>>  at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndG
>> etLastBlockLength(DFSInputStream.java:303)
>>  at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStrea
>> m.java:269)
>>  at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:261)
>>  at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1540)
>>  at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(
>> DistributedFileSystem.java:303)
>>  at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(
>> DistributedFileSystem.java:299)
>>  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>> emLinkResolver.java:81)
>>  at org.apache.hadoop.hdfs.DistributedFileSystem.open(Distribute
>> dFileSystem.java:299)
>>  at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767)
>>  at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>> eTableStep.readRowCountFromFile(HiveMRInput.java:294)
>>  at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiv
>> eTableStep.doWork(HiveMRInput.java:337)
>>  at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:113)
>>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>> rk(DefaultChainedExecutable.java:57)
>>  at org.apache.kylin.job.execution.AbstractExecutable.execute(
>> AbstractExecutable.java:113)
>>  at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>> ner.run(DefaultScheduler.java:136)
>>  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
>> File does not exist: /young/kylin_test/kylin_metada
>> ta_test/kylin-678266c0-ba0e-48b4-bdb5-6e578320375a/row_count/000000_0
>>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:71)
>>  at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INo
>> deFile.java:61)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> LocationsInt(FSNamesystem.java:1820)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1791)
>>  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlock
>> Locations(FSNamesystem.java:1704)
>>  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.get
>> BlockLocations(NameNodeRpcServer.java:587)
>>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolS
>> erverSideTranslatorPB.java:365)
>>  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:616)
>>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
>>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2081)
>>  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2077)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>>  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1657)
>>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2075)
>>
>>  at org.apache.hadoop.ipc.Client.call(Client.java:1427)
>>  at org.apache.hadoop.ipc.Client.call(Client.java:1358)
>>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>> ProtobufRpcEngine.java:229)
>>  at com.sun.proxy.$Proxy25.getBlockLocations(Unknown Source)
>>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>> slatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>> ssorImpl.java:57)
>>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>>  at java.lang.reflect.Method.invoke(Method.java:606)
>>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>> od(RetryInvocationHandler.java:252)
>>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>> ryInvocationHandler.java:104)
>>  at com.sun.proxy.$Proxy26.getBlockLocations(Unknown Source)
>>  at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSCl
>> ient.java:1240)
>>  ... 20 more
>> 2016-12-19 10:28:35,078 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> 2016-12-19 10:28:35,082 DEBUG [pool-8-thread-7]
>> hbase.HBaseResourceStore:262 : Update row /execute_output/678266c0-ba0e-48b4-bdb5-6e578320375a-01
>> from oldTs: 1482114480638, to newTs: 1482114515078, operation result: true
>> 2016-12-19 10:28:35,085 DEBUG [pool-8-thread-7] dao.ExecutableDao:210 :
>> updating job output, id: 678266c0-ba0e-48b4-bdb5-6e578320375a-01
>>
>

Mime
View raw message