kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Manoj H" <manoj.h.ku...@jpmorgan.com>
Subject RE: 回复:RE: Apache Kylin - Hbase Cube Name Space
Date Mon, 27 Nov 2017 08:10:23 GMT
Can someone help on this? We created new jar using below patch -  see: https://issues.apache.org/jira/browse/KYLIN-2846

but still its getting same Error C Its failing at CubeHTableUtil Java class file C Method
call CreateHTable() C As per patch file, This file is not changed & its not reading Namespace
properties to create the Hbase Table on that Namespace? Can you pls. advise on this C we
may need to add the Name space code here as well??

  public static void createHTable(CubeSegment cubeSegment, byte[][] splitKeys) throws IOException
{
        String tableName = cubeSegment.getStorageLocationIdentifier();
        CubeInstance cubeInstance = cubeSegment.getCubeInstance();
        CubeDesc cubeDesc = cubeInstance.getDescriptor();
        KylinConfig kylinConfig = cubeDesc.getConfig();


1448 Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException):
org.apache.hadoop.hbase.security.AccessDeniedException: Insuff     icient permissions (user=a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM,
scope=default, params=[namespace=default,table=default:KYLIN_CC2EO7R2NO,family=F1],action=CREATE)
1436         at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:105)
1437         at org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:103)
1449         at org.apache.hadoop.hbase.security.access.AccessController.requireNamespacePermission(AccessController.java:633)
1450         at org.apache.hadoop.hbase.security.access.AccessController.preCreateTable(AccessController.java:1002)
1451         at org.apache.hadoop.hbase.master.MasterCoprocessorHost$11.call(MasterCoprocessorHost.java:216)
1452         at org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1142)
1453         at org.apache.hadoop.hbase.master.MasterCoprocessorHost.preCreateTable(MasterCoprocessorHost.java:212)
1454         at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1521)
1455         at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:469)
1456         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55682)
1457         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
1458         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
1459         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
1460         at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
1462         at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
1463         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
1464         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
1465         at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.createTable(MasterProtos.java:58551)
1466         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$5.createTable(ConnectionManager.java:1968)
1467         at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(HBaseAdmin.java:756)
1468         at org.apache.hadoop.hbase.client.HBaseAdmin$5.call(HBaseAdmin.java:751)
1469         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)

1436         at org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:105)
1437         at org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:103)
Regards,
Manoj

From: Kumar, Manoj H
Sent: Friday, November 24, 2017 9:46 AM
To: user@kylin.apache.org
Subject: RE: ظRE: Apache Kylin - Hbase Cube Name Space

Can you pls. upload the jar to github? I cant build it locally as external jar needs to be
registered first.

Regards,
Manoj

From: Wei Li [mailto:lw_data@qq.com<mailto:lw_data@qq.com>]
Sent: Friday, November 24, 2017 9:44 AM
To: user
Subject: ظRE: Apache Kylin - Hbase Cube Name Space

Hi Manoj,
I also need hbase namespace feature, and I build the package successfully.


------------------ ԭʼʼ ------------------
: "Kumar, Manoj H";<manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>;
ʱ: 20171124() 11:56
ռ: "user@kylin.apache.org<mailto:user@kylin.apache.org>"<user@kylin.apache.org<mailto:user@kylin.apache.org>>;
: RE: Apache Kylin - Hbase Cube Name Space

Can someone upload Beta Jar file to Github so that other folks can use it? I am sure many
folks have done applied the patch at their locally but it will be good to use the Updated
Jar file instead of building it again & again. Can you pls. someone help on this? I need
this issue to be fixed asap.

Appreciate your faster response in this regard.

Regards,
Manoj

From: Kumar, Manoj H
Sent: Thursday, November 23, 2017 8:08 PM
To: user@kylin.apache.org<mailto:user@kylin.apache.org>
Subject: RE: Apache Kylin - Hbase Cube Name Space

How to do the patch.. I cant built it locally as This project is not added to JPMC Maven Repo
yet. Can you send the binary & I can do the update?

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Thursday, November 23, 2017 8:03 PM
To: user
Subject: Re: Apache Kylin - Hbase Cube Name Space

Hi Manoj,

This feature is not released yet, see: https://issues.apache.org/jira/browse/KYLIN-2846
But the patch is available; you can build a new Kylin binary package by your own with the
patches.

2017-11-23 21:49 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
Its pointing to Default name C How we can change the Habse Namespace so that we can build
the code successfully. Pls. advise. It has failed at Create HT Table step.

org.apache.hadoop.hbase.security.AccessDeniedException: org.apache.hadoop.hbase.security.AccessDeniedException:
Insufficient permissions (user=a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM<mailto:a_rcmo_nd@NAEAST.AD.JPMORGANCHASE.COM>,
scope=default, params=[namespace=default,table=default:KYLIN_S1W12SO8AC,family=F1],action=CREATE)

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Wednesday, November 22, 2017 8:53 AM
To: user
Subject: Re: How to Update the Cube Structure

The intermediate hive table will be cleaned in the last step "Hive Cleanup" automatically.

If your job is failed in the between, then the last step won't be executed by Kylin. In this
case, you can manually clean them with the the storage<https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html>
clean up cli<https://kylin.apache.org/docs21/howto/howto_cleanup_storage.html>.

2017-11-22 11:13 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
Np Mind, I got confused with Cubes Partition column. Once Cube is build out, Can we clean
Hive Intermediate flat table? Also, How can we know which flat belongs particular cube? How
to schedule the cube build process C taking incremental data from hive source table

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Wednesday, November 22, 2017 6:52 AM

To: user
Subject: Re: How to Update the Cube Structure

Hi Manoj,

The intermediate flat table is not partitioned, always; it has all files in one folder; I'm
wondering what's the purpose to make it partitioned?

2017-11-22 1:12 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
Thanks one quick question C How to make intermediate flat table partitioned one?? Do we
have any parameter in kylin.properties?

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Tuesday, November 21, 2017 8:02 PM

To: user
Subject: Re: How to Update the Cube Structure

This document can help:

https://kylin.apache.org/docs21/howto/howto_optimize_build.html


2017-11-21 21:36 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
Thanks for the inputs.. It make sense..

As now I am running on Dev Cluster & need to know optimize parameter for Kylin so that
Cube building process runs fast. Can you pls. tell what are properties I need to change in
kylin.properties file C To optimize the cube build process.. I have 180 millions of data
on fact table.

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Tuesday, November 21, 2017 3:23 PM
To: user
Subject: Re: How to Update the Cube Structure

To be safe, you can clone the existing to a new cube, make changes on the new cube, and then
rebuild it.

2017-11-21 14:53 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
In case of Data Source table has DDL changes C either new Column added or Existing Column
data type changed. How Can we do Cube data model update & reflecting new column data to
Cube? Pls. advise.

Regards,
Manoj

From: ShaoFeng Shi [mailto:shaofengshi@apache.org<mailto:shaofengshi@apache.org>]
Sent: Monday, November 20, 2017 7:03 PM
To: user
Subject: Re: Apache Kyine - kylin.env.hdfs-working-dir

It seems that the script still try to create "/kylin" on HDFS, not in the configured path.

Please double check and make sure this line is not commented:

kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin

2017-11-20 14:00 GMT+08:00 Kumar, Manoj H <manoj.h.kumar@jpmorgan.com<mailto:manoj.h.kumar@jpmorgan.com>>:
I am installing kyline 2.10 on Dev Cluster(Cloudera) . Although I have given 777 permission
on kylin Folder but still giving error. Can you pls. help on this? Whats missing here?

-bash-4.1$ hdfs dfs -mkdir /tenants/rft/rcmo/kylin
-bash-4.1$ hdfs dfs -chmod 777 /tenants/rft/rcmo/kylin

Kyline properties.conf - #kylin.env.hdfs-working-dir=/tenants/rft/rcmo/kylin

-bash-4.1$ ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /apps/rft/rcmo/apps/kylin/apache-kylin-2.1.0-bin-cdh57
mkdir: Permission denied: user=a_rcmo_nd, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin

Regards,
Manoj


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.



--
Best regards,

Shaofeng Shi ʷٷ


This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.

This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.

This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.

This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer
including on confidentiality, legal privilege, viruses and monitoring of electronic messages.
If you are not the intended recipient, please delete this message and notify the sender immediately.
Any unauthorized use is strictly prohibited.
Mime
View raw message