kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: Error in kylin with standalone HBase cluster
Date Tue, 10 Jan 2017 02:35:18 GMT
Hi Nan,

This error indicates that the hadoop configuration of the HBase cluster has
overwriten the configuration of the default hadoop cluster. As you know,
Kylin uses the output of "hbase classpath" as the classpath to startup; so
it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml"
are on the classpath's starting position. Please locate them, take backup,
and then replace them with the hive/mapreduce cluster's config file; only
left "hbase-site.xml" with the configurations to the dedicated hbase
cluster.

After doing that, restart Kylin, discard the error job, and resubmit a
build.



2017-01-09 23:12 GMT+08:00 柯南 <zzm_bupt@qq.com>:

> hi,all:
>      I want to deploy apache kylin with standalone HBase cluster,and also
> refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone-
> hbase-cluster) to update the config kylin.hbase.cluster.fs in
> kylin.properties, but when I build new cube in step2(Redistribute Flat Hive
> Table), it is using hadoop client whch hbase cluster is dependent on, but
> not main cluster's. Thus error that file in hdfs couldn't be find  occur.
> The program get hadoop configuration using HadoopUtil.getCurrentConfiguration()
> in CreateFlatHiveTableStep class.
>     The version of Kylin upgade 1.6.0 is the same.
>
>
>
>
> 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3]
> hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0-615b1ceab3bc-01
> execute finished with exception
> java.io.FileNotFoundException: File does not exist:
> /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0-
> 615b1ceab3bc/row_count/000000_0
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:65)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:55)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsInt(FSNamesystem.java:1820)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1800)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1772)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> getBlockLocations(NameNodeRpcServer.java:527)
>         at org.apache.hadoop.hdfs.server.namenode.
> AuthorizationProviderProxyClientProtocol.getBlockLocations(
> AuthorizationProviderProxyClientProtocol.java:85)
>         at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolServerSideTranslatorPB.java:356)
>         at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1642)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>
>
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at org.apache.hadoop.ipc.RemoteException.instantiateException(
> RemoteException.java:106)
>         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
> RemoteException.java:73)
>         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> DFSClient.java:1171)
>         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> DFSClient.java:1159)
>         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> DFSClient.java:1149)
>         at org.apache.hadoop.hdfs.DFSInputStream.
> fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
>         at org.apache.hadoop.hdfs.DFSInputStream.openInfo(
> DFSInputStream.java:237)
>         at org.apache.hadoop.hdfs.DFSInputStream.<init>(
> DFSInputStream.java:230)
>         at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> doCall(DistributedFileSystem.java:301)
>         at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> doCall(DistributedFileSystem.java:297)
>         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> FileSystemLinkResolver.java:81)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.open(
> DistributedFileSystem.java:297)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
>         at org.apache.kylin.source.hive.CreateFlatHiveTableStep.
> readRowCountFromFile(CreateFlatHiveTableStep.java:51)
>         at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(
> CreateFlatHiveTableStep.java:103)
>         at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:112)
>         at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
>         at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:112)
>         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:127)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException):
> File does not exist: /user/kylin/kylin_metadata/
> kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:65)
>         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> INodeFile.java:55)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocationsInt(FSNamesystem.java:1820)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1800)
>         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> getBlockLocations(FSNamesystem.java:1772)
>         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> getBlockLocations(NameNodeRpcServer.java:527)
>         at org.apache.hadoop.hdfs.server.namenode.
> AuthorizationProviderProxyClientProtocol.getBlockLocations(
> AuthorizationProviderProxyClientProtocol.java:85)
>         at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolServerSideTranslatorPB.java:356)
>         at org.apache.hadoop.hdfs.protocol.proto.
> ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(
> ClientNamenodeProtocolProtos.java)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1642)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
>
>
>         at org.apache.hadoop.ipc.Client.call(Client.java:1411)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
>         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> invoke(ProtobufRpcEngine.java:206)
>         at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source)
>         at org.apache.hadoop.hdfs.protocolPB.
> ClientNamenodeProtocolTranslatorPB.getBlockLocations(
> ClientNamenodeProtocolTranslatorPB.java:246)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
> RetryInvocationHandler.java:187)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> RetryInvocationHandler.java:102)
>         at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> DFSClient.java:1169)
>         ... 20 more




-- 
Best regards,

Shaofeng Shi 史少锋

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message