kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ShaoFeng Shi <shaofeng...@apache.org>
Subject Re: 回复:RE: Apache Kylin - Hbase Cube Name Space
Date Mon, 27 Nov 2017 08:19:20 GMT
Hi Manoj,

Only replace a jar may not work; You need to build a new binary package.

What's your Kylin version and Hadoop/HBase release/version?

2017-11-27 16:10 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:

> Can someone help on this? We created new jar using below patch -  see:
> https://issues.apache.org/jira/browse/KYLIN-2846
>
>
>
> but still its getting same Error – Its failing at *CubeHTableUtil* Java
> class file – Method call CreateHTable() – As per patch file, This file is
> not changed & its not reading Namespace properties to create the Hbase
> Table on that Namespace? Can you pls. advise on this – we may need to add
> the Name space code here as well??
>
>
>
>   public static void createHTable(CubeSegment cubeSegment, byte[][]
> splitKeys) throws IOException {
>
>         String tableName = cubeSegment.getStorageLocationIdentifier();
>
>         CubeInstance cubeInstance = cubeSegment.getCubeInstance();
>
>         CubeDesc cubeDesc = cubeInstance.getDescriptor();
>
>         KylinConfig kylinConfig = cubeDesc.getConfig();
>
>
>
>
>
> 1448 Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.
> apache.hadoop.hbase.security.AccessDeniedException):
> org.apache.hadoop.hbase.security.AccessDeniedException: Insuff     icient
> permissions (user=a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM, scope=default,
> params=[namespace=default,table=default:KYLIN_
> CC2EO7R2NO,family=F1],action=CREATE)
>
> 1436         at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.
> createHTable(CubeHTableUtil.java:105)
>
> 1437         at org.apache.kylin.storage.hbase.steps.CreateHTableJob.
> run(CreateHTableJob.java:103)
>
> 1449         at org.apache.hadoop.hbase.security.access.AccessController.
> requireNamespacePermission(AccessController.java:633)
>
> 1450         at org.apache.hadoop.hbase.security.access.AccessController.
> preCreateTable(AccessController.java:1002)
>
> 1451         at org.apache.hadoop.hbase.master.MasterCoprocessorHost$
> 11.call(MasterCoprocessorHost.java:216)
>
> 1452         at org.apache.hadoop.hbase.master.MasterCoprocessorHost.
> execOperation(MasterCoprocessorHost.java:1142)
>
> 1453         at org.apache.hadoop.hbase.master.MasterCoprocessorHost.
> preCreateTable(MasterCoprocessorHost.java:212)
>
> 1454         at org.apache.hadoop.hbase.master.HMaster.createTable(
> HMaster.java:1521)
>
> 1455         at org.apache.hadoop.hbase.master.MasterRpcServices.
> createTable(MasterRpcServices.java:469)
>
> 1456         at org.apache.hadoop.hbase.protobuf.generated.
> MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
>
> 1457         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:
> 2170)
>
> 1458         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.
> java:109)
>
> 1459         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
> RpcExecutor.java:185)
>
> 1460         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(
> RpcExecutor.java:165)
>
> 1462         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(
> RpcClientImpl.java:1269)
>
> 1463         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.
> callBlockingMethod(AbstractRpcClient.java:227)
>
> 1464         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$
> BlockingRpcChannelImplementation.callBlockingMethod(
> AbstractRpcClient.java:336)
>
> 1465         at org.apache.hadoop.hbase.protobuf.generated.
> MasterProtos$MasterService$BlockingStub.createTable(
> MasterProtos.java:58551)
>
> 1466         at org.apache.hadoop.hbase.client.ConnectionManager$
> HConnectionImplementation$5.createTable(ConnectionManager.java:1968)
>
> 1467         at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(
> HBaseAdmin.java:756)
>
> 1468         at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(
> HBaseAdmin.java:751)
>
> 1469         at org.apache.hadoop.hbase.client.RpcRetryingCaller.
> callWithRetries(RpcRetryingCaller.java:126)
>
>
>
> 1436         at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.
> createHTable(CubeHTableUtil.java:105)
>
> 1437         at org.apache.kylin.storage.hbase.steps.CreateHTableJob.
> run(CreateHTableJob.java:103)
>
> Regards,
>
> Manoj
>
>
>
> *From:* Kumar, Manoj H
> *Sent:* Friday, November 24, 2017 9:46 AM
> *To:* user@kylin.apache.org
> *Subject:* RE: 回复:RE: Apache Kylin - Hbase Cube Name Space
>
>
>
> Can you pls. upload the jar to github? I cant build it locally as external
> jar needs to be registered first.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Wei Li [mailto:lw_data@qq.com]
> *Sent:* Friday, November 24, 2017 9:44 AM
> *To:* user
> *Subject:* 回复:RE: Apache Kylin - Hbase Cube Name Space
>
>
>
> Hi Manoj,
>
> I also need hbase namespace feature, and I build the package successfully.
>
>
>
>
>
> ------------------ 原始邮件 ------------------
>
> *发件人**:* "Kumar, Manoj H";<manoj.h.kumar@jpmorgan.com>;
>
> *发送时间**:* 2017年11月24日(星期五) 中午11:56
>
> *收件人**:* "user@kylin.apache.org"<user@kylin.apache.org>;
>
> *主题**:* RE: Apache Kylin - Hbase Cube Name Space
>
>
>
> Can someone upload Beta Jar file to Github so that other folks can use it?
> I am sure many folks have done applied the patch at their locally but it
> will be good to use the Updated Jar file instead of building it again &
> again. Can you pls. someone help on this? I need this issue to be fixed
> asap.
>
>
>
> Appreciate your faster response in this regard.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* Kumar, Manoj H
> *Sent:* Thursday, November 23, 2017 8:08 PM
> *To:* user@kylin.apache.org
> *Subject:* RE: Apache Kylin - Hbase Cube Name Space
>
>
>
> How to do the patch.. I cant built it locally as This project is not added
> to JPMC Maven Repo yet. Can you send the binary & I can do the update?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Thursday, November 23, 2017 8:03 PM
> *To:* user
> *Subject:* Re: Apache Kylin - Hbase Cube Name Space
>
>
>
> Hi Manoj,
>
>
>
> This feature is not released yet, see: https://issues.apache.org/
> jira/browse/KYLIN-2846
>
> But the patch is available; you can build a new Kylin binary package by
> your own with the patches.
>
>
>
> 2017-11-23 21:49 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> Its pointing to Default name – How we can change the Habse Namespace so
> that we can build the code successfully. Pls. advise. It has failed at
> Create HT Table step.
>
>
>
> org.apache.hadoop.hbase.security.AccessDeniedException:
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions (user=a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM, scope=default,
> params=[namespace=*default*,table=default:KYLIN_
> S1W12SO8AC,family=F1],action=CREATE)
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Wednesday, November 22, 2017 8:53 AM
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> The intermediate hive table will be cleaned in the last step "Hive
> Cleanup" automatically.
>
>
>
> If your job is failed in the between, then the last step won't be executed
> by Kylin. In this case, you can manually clean them with the the storage
> <https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html> clean
> up cli <https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html>.
>
>
>
> 2017-11-22 11:13 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> Np Mind, I got confused with Cube’s Partition column. Once Cube is build
> out, Can we clean Hive Intermediate flat table? Also, How can we know which
> flat belongs particular cube? How to schedule the cube build process –
> taking incremental data from hive source table
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Wednesday, November 22, 2017 6:52 AM
>
>
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> Hi Manoj,
>
>
>
> The intermediate flat table is not partitioned, always; it has all files
> in one folder; I'm wondering what's the purpose to make it partitioned?
>
>
>
> 2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> Thanks… one quick question – How to make intermediate flat table
> partitioned one?? Do we have any parameter in kylin.properties?
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 8:02 PM
>
>
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> This document can help:
>
>
>
> https://kylin.apache.org/docs21/howto/howto_optimize_build.html
>
>
>
>
>
> 2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> Thanks for the inputs.. It make sense..
>
>
>
> As now I am running on Dev Cluster & need to know optimize parameter for
> Kylin so that Cube building process runs fast. Can you pls. tell what are
> properties I need to change in kylin.properties file – To optimize the cube
> build process.. I have 180 millions of data on fact table.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Tuesday, November 21, 2017 3:23 PM
> *To:* user
> *Subject:* Re: How to Update the Cube Structure
>
>
>
> To be safe, you can clone the existing to a new cube, make changes on the
> new cube, and then rebuild it.
>
>
>
> 2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> In case of Data Source table has DDL changes – either new Column added or
> Existing Column data type changed. How Can we do Cube data model update &
> reflecting new column data to Cube? Pls. advise.
>
>
>
> Regards,
>
> Manoj
>
>
>
> *From:* ShaoFeng Shi [mailto:shaofengshi@apache.org]
> *Sent:* Monday, November 20, 2017 7:03 PM
> *To:* user
> *Subject:* Re: Apache Kyine - kylin.env.hdfs-working-dir
>
>
>
> It seems that the script still try to create "/kylin" on HDFS, not in the
> configured path.
>
>
>
> Please double check and make sure this line is not commented:
>
>
>
> kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin
>
>
>
> 2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com>:
>
> I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have
> given 777 permission on kylin Folder but still giving error. Can you pls.
> help on this? Whats missing here?
>
>
>
> -bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
>
> -bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin
>
>
>
> Kyline properties.conf - #kylin.env.hdfs-working-dir=/
> tenants/rft/rcmo/kylin
>
>
>
> -bash-4.1$ ./check-env.sh
>
> Retrieving hadoop conf dir...
>
> KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/
> apache-kylin-2.1.0-bin-cdh57
>
> mkdir: Permission denied: user=a_rcmo_nd, access=WRITE,
> inode="/":hdfs:supergroup:drwxr-xr-x
>
> Failed to create /kylin. Please make sure the user has right to access
> /kylin
>
>
>
> Regards,
>
> Manoj
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
>
>
>
>
> --
>
> Best regards,
>
>
>
> Shaofeng Shi 史少锋
>
>
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at: http://
> www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not
> the intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>



-- 
Best regards,

Shaofeng Shi 史少锋

Mime
View raw message