kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "柯南" <zzm_b...@qq.com>
Subject 回复: Error in kylin with standalone HBase cluster
Date Wed, 11 Jan 2017 10:01:42 GMT
Thank you! It works,and the file "yarn-site.xml" and "mapred-site.xml" also should be replaced.




------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<shaofengshi@apache.org>;
发送时间: 2017年1月10日(星期二) 晚上6:05
收件人: "dev"<dev@kylin.apache.org>; 

主题: Re: Error in kylin with standalone HBase cluster



Hi,

Firstly you need install Kylin in a dedicated machine (pure client machine
to run Kylin, not a hadoop job node nor HBase RS), so that replace the
"core-site.xml" or "hdfs-site.xml" will not impact on other services.

Secondly, Kylin's communication with HBase all goes to HBase API/RPC, that
means Kylin only need the "hbase-site.xml". So replace the "core-site.xml"
or "hdfs-site.xml" in HBase client folder (only this machine) doesn't
impact.

Thirdly, configure "kylin.hbase.cluster.fs" in kylin.properties is telling
Kylin that HBase is using another HDFS (not the default one) ; so after the
cube be calculated in the default cluster, Kylin will write the HFile to
this HDFS.

After all these done, Kylin will use HBase's API to do bulkload; HBase
region server will load the files from their own HDFS directly, which are
already in place.

Hope this helps.

2017-01-10 14:10 GMT+08:00 柯南 <zzm_bupt@qq.com>:

> hi Shao feng,
>
> Thank you for your advice.
> As you said, if I replace the "core-site.xml" and "hdfs-site.xml" with the
> hive/mapreduce cluster's config file, it means that the hbase and
> hive/mapreduce cluster's are all dependent on the same hadoop client, but
> not deploying with standalone HBase cluster. When they dependent on the
> same hadoop client, I know it works well.
> On the other hand, the property "kylin.hbase.cluster.fs" in
> kylin.properties,when to use for?
> If I need to dependent hbase and hive/mapreduce cluster's on diffent hdfs,
> counld kylin support that? Or may be in the future?
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<shaofengshi@apache.org>;
> 发送时间: 2017年1月10日(星期二) 上午10:35
> 收件人: "dev"<dev@kylin.apache.org>;
>
> 主题: Re: Error in kylin with standalone HBase cluster
>
>
>
> Hi Nan,
>
> This error indicates that the hadoop configuration of the HBase cluster has
> overwriten the configuration of the default hadoop cluster. As you know,
> Kylin uses the output of "hbase classpath" as the classpath to startup; so
> it is possible that the hbase cluster's "core-site.xml", "hdfs-site.xml"
> are on the classpath's starting position. Please locate them, take backup,
> and then replace them with the hive/mapreduce cluster's config file; only
> left "hbase-site.xml" with the configurations to the dedicated hbase
> cluster.
>
> After doing that, restart Kylin, discard the error job, and resubmit a
> build.
>
>
>
> 2017-01-09 23:12 GMT+08:00 柯南 <zzm_bupt@qq.com>:
>
> > hi,all:
> >      I want to deploy apache kylin with standalone HBase cluster,and also
> > refer offical doc(http://kylin.apache.org/blog/2016/06/10/standalone-
> > hbase-cluster) to update the config kylin.hbase.cluster.fs in
> > kylin.properties, but when I build new cube in step2(Redistribute Flat
> Hive
> > Table), it is using hadoop client whch hbase cluster is dependent on, but
> > not main cluster's. Thus error that file in hdfs couldn't be find  occur.
> > The program get hadoop configuration using HadoopUtil.
> getCurrentConfiguration()
> > in CreateFlatHiveTableStep class.
> >     The version of Kylin upgade 1.6.0 is the same.
> >
> >
> >
> >
> > 2017-01-04 18:42:43,459 ERROR [pool-7-thread-3]
> > hive.CreateFlatHiveTableStep:114 : job:3889515b-0054-4b71-9db0-
> 615b1ceab3bc-01
> > execute finished with exception
> > java.io.FileNotFoundException: File does not exist:
> > /user/kylin/kylin_metadata/kylin-3889515b-0054-4b71-9db0-
> > 615b1ceab3bc/row_count/000000_0
> >         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> > INodeFile.java:65)
> >         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> > INodeFile.java:55)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocationsInt(FSNamesystem.java:1820)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocations(FSNamesystem.java:1800)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocations(FSNamesystem.java:1772)
> >         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> > getBlockLocations(NameNodeRpcServer.java:527)
> >         at org.apache.hadoop.hdfs.server.namenode.
> > AuthorizationProviderProxyClientProtocol.getBlockLocations(
> > AuthorizationProviderProxyClientProtocol.java:85)
> >         at org.apache.hadoop.hdfs.protocolPB.
> > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> > ClientNamenodeProtocolServerSideTranslatorPB.java:356)
> >         at org.apache.hadoop.hdfs.protocol.proto.
> > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.
> callBlockingMethod(
> > ClientNamenodeProtocolProtos.java)
> >         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >         at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1642)
> >         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> >
> >
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> >         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> > NativeConstructorAccessorImpl.java:57)
> >         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> > DelegatingConstructorAccessorImpl.java:45)
> >         at java.lang.reflect.Constructor.newInstance(Constructor.java:
> 526)
> >         at org.apache.hadoop.ipc.RemoteException.instantiateException(
> > RemoteException.java:106)
> >         at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
> > RemoteException.java:73)
> >         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> > DFSClient.java:1171)
> >         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> > DFSClient.java:1159)
> >         at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(
> > DFSClient.java:1149)
> >         at org.apache.hadoop.hdfs.DFSInputStream.
> > fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
> >         at org.apache.hadoop.hdfs.DFSInputStream.openInfo(
> > DFSInputStream.java:237)
> >         at org.apache.hadoop.hdfs.DFSInputStream.<init>(
> > DFSInputStream.java:230)
> >         at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1448)
> >         at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> > doCall(DistributedFileSystem.java:301)
> >         at org.apache.hadoop.hdfs.DistributedFileSystem$3.
> > doCall(DistributedFileSystem.java:297)
> >         at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
> > FileSystemLinkResolver.java:81)
> >         at org.apache.hadoop.hdfs.DistributedFileSystem.open(
> > DistributedFileSystem.java:297)
> >         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
> >         at org.apache.kylin.source.hive.CreateFlatHiveTableStep.
> > readRowCountFromFile(CreateFlatHiveTableStep.java:51)
> >         at org.apache.kylin.source.hive.CreateFlatHiveTableStep.doWork(
> > CreateFlatHiveTableStep.java:103)
> >         at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:112)
> >         at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(
> > DefaultChainedExecutable.java:57)
> >         at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:112)
> >         at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > JobRunner.run(DefaultScheduler.java:127)
> >         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1145)
> >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:615)
> >         at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.
> FileNotFoundException):
> > File does not exist: /user/kylin/kylin_metadata/
> > kylin-3889515b-0054-4b71-9db0-615b1ceab3bc/row_count/000000_0
> >         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> > INodeFile.java:65)
> >         at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(
> > INodeFile.java:55)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocationsUpdateTimes(FSNamesystem.java:1879)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocationsInt(FSNamesystem.java:1820)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocations(FSNamesystem.java:1800)
> >         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
> > getBlockLocations(FSNamesystem.java:1772)
> >         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.
> > getBlockLocations(NameNodeRpcServer.java:527)
> >         at org.apache.hadoop.hdfs.server.namenode.
> > AuthorizationProviderProxyClientProtocol.getBlockLocations(
> > AuthorizationProviderProxyClientProtocol.java:85)
> >         at org.apache.hadoop.hdfs.protocolPB.
> > ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(
> > ClientNamenodeProtocolServerSideTranslatorPB.java:356)
> >         at org.apache.hadoop.hdfs.protocol.proto.
> > ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.
> callBlockingMethod(
> > ClientNamenodeProtocolProtos.java)
> >         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
> > ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> >         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> >         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >         at org.apache.hadoop.security.UserGroupInformation.doAs(
> > UserGroupInformation.java:1642)
> >         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> >
> >
> >         at org.apache.hadoop.ipc.Client.call(Client.java:1411)
> >         at org.apache.hadoop.ipc.Client.call(Client.java:1364)
> >         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.
> > invoke(ProtobufRpcEngine.java:206)
> >         at com.sun.proxy.$Proxy23.getBlockLocations(Unknown Source)
> >         at org.apache.hadoop.hdfs.protocolPB.
> > ClientNamenodeProtocolTranslatorPB.getBlockLocations(
> > ClientNamenodeProtocolTranslatorPB.java:246)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:57)
> >         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:606)
> >         at org.apache.hadoop.io.retry.RetryInvocationHandler.
> invokeMethod(
> > RetryInvocationHandler.java:187)
> >         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
> > RetryInvocationHandler.java:102)
> >         at com.sun.proxy.$Proxy24.getBlockLocations(Unknown Source)
> >         at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(
> > DFSClient.java:1169)
> >         ... 20 more
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>



-- 
Best regards,

Shaofeng Shi 史少锋
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message