ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhan Neethiraj <mad...@apache.org>
Subject Re: Review Request 68881: RANGER-2203: Review and update database schema for ranger policies to minimize database queries/updates
Date Sun, 07 Oct 2018 22:28:13 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68881/#review209275
-----------------------------------------------------------




agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java
Lines 43 (patched)
<https://reviews.apache.org/r/68881/#comment293564>

    Consider not calling setPrettyPrinting() - to make serialization faster/smaller; doing
so in Atlas significantly reduced the size of the serialized json. Given with this enhancement
stores serialized json in database, this could make a sizable difference. Please review and
update other such usage as well.



agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java
Line 44 (original), 44 (patched)
<https://reviews.apache.org/r/68881/#comment293563>

    This file has whitespace only changes. Please consider excluding this file from the patch.



agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java
Lines 87 (patched)
<https://reviews.apache.org/r/68881/#comment293587>

    updateTagsInServiceResource(resourceId) method seems to be for migration use. Having this
in TagStore interface may not be appropriate (and confussing). Consider moving this to migration
module.



security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java
Line 305 (original), 259 (patched)
<https://reviews.apache.org/r/68881/#comment293590>

    findTaggedResourcesInServiceId_after_porting()? Is earlier version,  findTaggedResourcesInServiceId(),
needed any more? If not remove the earlier version and remove suffix "_after_porting" from
method name.
    
    Review other such use of suffix _after_porting.



security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java
Line 347 (original), 285 (patched)
<https://reviews.apache.org/r/68881/#comment293591>

    >> StringUtils.isNotEmpty(xServiceResource.getTags()
    This condition will endup skiping service-resource with no-tags. Is this the intention?



security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Line 1608 (original), 1592 (patched)
<https://reviews.apache.org/r/68881/#comment293588>

    Please avoid whiltespace only changes like #1592, #1595, #1605 in this file.



security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
Line 1965 (original), 1904 (patched)
<https://reviews.apache.org/r/68881/#comment293589>

    intentation



security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java
Lines 63 (patched)
<https://reviews.apache.org/r/68881/#comment293594>

    This method, findUpdatedAccessDefNames(), doesn't seem to be used. Please review and remove
if unused.



security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java
Lines 64 (patched)
<https://reviews.apache.org/r/68881/#comment293593>

    This method, findUpdatedDataMaskDefNames(), doesn't seem to be used. Please review and
remove if unused.



security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java
Lines 79 (patched)
<https://reviews.apache.org/r/68881/#comment293592>

    Method findUpdatedUserNamesByService() doesn't belong here, in XXDataMaskTypeDefDao.java.
Please review and move/remove.



security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java
Lines 102 (patched)
<https://reviews.apache.org/r/68881/#comment293595>

    This method, findUpdatedResourceDefNames(), doesn't seem to be used. Please review and
remove if unused.



security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java
Lines 136 (patched)
<https://reviews.apache.org/r/68881/#comment293596>

    UpdateRangerTagsTablesWithTagsJson() => updateRangerTagsTablesWithTagsJson()


- Madhan Neethiraj


On Oct. 5, 2018, 2:47 a.m., Abhay Kulkarni wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68881/
> -----------------------------------------------------------
> 
> (Updated Oct. 5, 2018, 2:47 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal,
and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2186, RANGER-2195 and RANGER-2219
>     https://issues.apache.org/jira/browse/RANGER-2186
>     https://issues.apache.org/jira/browse/RANGER-2195
>     https://issues.apache.org/jira/browse/RANGER-2219
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Notes:
> 
> This is a composite patch for updating database schema for Ranger policy and tag related
objects.
> 
> Notes:
> 
> This patch is more complete than the one posted earlier (https://reviews.apache.org/r/68876/).
Database scripts to drop foreign-keys are missing for some database types. They need to be
added.
> 
> This patch also subsumes back-ports of RANGER-2186 and RANGER-2195 from master to ranger-0.7
branch. Porting those JIRA separately involved fixing conflicts in many common files.
> 
> Description:
> 
> Currently, ranger policies are fully normalized and stored in a multiple Relational database
tables. There is a performance overhead incurred when retrieving a ranger policy, as multiple
database accesses are required to fully reconstruct it. This is significant when there are
large ranger policies (that is, the number of resources addressed by the policy is large),
and/or when there is a large number of ranger policies in an installation.
> 
> This Jira tracks alternate design of database schema, where a policy is stored in a de-normalized
way, in its entirely, in one database table (preferably as a JSON string).
> 
> 
> Diffs
> -----
> 
>   agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java PRE-CREATION

>   agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java 743b02825

>   agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
a75ca59e8 
>   agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java 5918b1292

>   embeddedwebserver/scripts/ranger-admin-services.sh 19dab2a9d 
>   security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
>   security-admin/db/mysql/patches/040-denormalize-tag-tables.sql PRE-CREATION 
>   security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
>   security-admin/db/oracle/patches/040-denormalize-tag-tables.sql PRE-CREATION 
>   security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql PRE-CREATION

>   security-admin/db/postgres/patches/040-denormalize-tag-tables.sql PRE-CREATION 
>   security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql PRE-CREATION

>   security-admin/db/sqlanywhere/patches/040-denormalize-tag-tables.sql PRE-CREATION 
>   security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql PRE-CREATION

>   security-admin/db/sqlserver/patches/040-denormalize-tag-tables.sql PRE-CREATION 
>   security-admin/scripts/db_setup.py d4f37ed6f 
>   security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 354ab1cba

>   security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 53683ec04

>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ceee8cef2 
>   security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3234be6a2 
>   security-admin/src/main/java/org/apache/ranger/common/db/RangerTransactionSynchronizationAdapter.java
2a62fb408 
>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManager.java e3b878b3e 
>   security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java d718441ee

>   security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java 8f3a50687

>   security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java f6e1aff0b

>   security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 19e2e11f3 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java 21afcac89

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java de37e10cd

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 11596ef4d

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java
a8418c626 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java 8c05699b2

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java
4618e7dc7 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java 40a0da13b

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 4b04b9664

>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java 70657370a

>   security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java aee56e833 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java f87c0ae59

>   security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java
72fe2140b 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java
e726d047c 
>   security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 0098bff23

>   security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java e361b33bd

>   security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 145399f1c

>   security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java 77428dbf9 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java dfd2fcb5d 
>   security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java bbcd54618

>   security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 7af1bf96a 
>   security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4 
>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java 69d28bb54 
>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java
PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java PRE-CREATION

>   security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c

>   security-admin/src/main/java/org/apache/ranger/entity/XXTag.java 9155385ec 
>   security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10017.java
PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java
PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 6685e1a58 
>   security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java 7223f109c

>   security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java
0195c72a7 
>   security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java
d7256802a 
>   security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java
6af682a81 
>   security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6

>   security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa

>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4a7055dee 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 44523fb22

>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 68908b205

>   security-admin/src/test/java/org/apache/ranger/service/TestRangerServiceDefService.java
d65b0d6ee 
>   security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java
2ca0fd2fb 
> 
> 
> Diff: https://reviews.apache.org/r/68881/diff/8/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message