ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Kulkarni <akulka...@hortonworks.com>
Subject Re: Review Request 68881: RANGER-2203: Review and update database schema for ranger policies to minimize database queries/updates
Date Thu, 04 Oct 2018 15:20:28 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68881/
-----------------------------------------------------------

(Updated Oct. 4, 2018, 3:20 p.m.)


Review request for ranger, Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, and
Velmurugan Periasamy.


Summary (updated)
-----------------

RANGER-2203: Review and update database schema for ranger policies to minimize database queries/updates


Bugs: RANGER-2186, RANGER-2195 and RANGER-2219
    https://issues.apache.org/jira/browse/RANGER-2186
    https://issues.apache.org/jira/browse/RANGER-2195
    https://issues.apache.org/jira/browse/RANGER-2219


Repository: ranger


Description
-------

Notes:

This is a composite patch for updating database schema for Ranger policy and tag related objects.

Notes:

This patch is more complete than the one posted earlier (https://reviews.apache.org/r/68876/).
Database scripts to drop foreign-keys are missing for some database types. They need to be
added.

This patch also subsumes back-ports of RANGER-2186 and RANGER-2195 from master to ranger-0.7
branch. Porting those JIRA separately involved fixing conflicts in many common files.

Description:

Currently, ranger policies are fully normalized and stored in a multiple Relational database
tables. There is a performance overhead incurred when retrieving a ranger policy, as multiple
database accesses are required to fully reconstruct it. This is significant when there are
large ranger policies (that is, the number of resources addressed by the policy is large),
and/or when there is a large number of ranger policies in an installation.

This Jira tracks alternate design of database schema, where a policy is stored in a de-normalized
way, in its entirely, in one database table (preferably as a JSON string).


Diffs
-----

  agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java PRE-CREATION

  agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java 743b02825 
  agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java a75ca59e8

  agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java 5918b1292 
  embeddedwebserver/scripts/ranger-admin-services.sh 19dab2a9d 
  security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/mysql/patches/040-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/oracle/patches/040-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/postgres/patches/040-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/sqlanywhere/patches/040-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/sqlserver/patches/040-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/scripts/db_setup.py d4f37ed6f 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java 354ab1cba

  security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 53683ec04 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java ceee8cef2 
  security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java 3234be6a2 
  security-admin/src/main/java/org/apache/ranger/common/db/RangerTransactionSynchronizationAdapter.java
2a62fb408 
  security-admin/src/main/java/org/apache/ranger/db/RangerDaoManager.java e3b878b3e 
  security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java d718441ee 
  security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java 8f3a50687 
  security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java f6e1aff0b 
  security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 19e2e11f3 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java 21afcac89

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java de37e10cd 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 11596ef4d

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java a8418c626

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java 8c05699b2

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java 4618e7dc7

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java 40a0da13b

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java 4b04b9664 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java 70657370a

  security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java aee56e833 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java f87c0ae59 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java 72fe2140b

  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java
e726d047c 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceVersionInfoDao.java 0098bff23

  security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java e361b33bd 
  security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 145399f1c 
  security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java 77428dbf9 
  security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java dfd2fcb5d 
  security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java bbcd54618 
  security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 7af1bf96a 
  security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java 69d28bb54 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c 
  security-admin/src/main/java/org/apache/ranger/entity/XXTag.java 9155385ec 
  security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10017.java
PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java
PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 6685e1a58 
  security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java 7223f109c

  security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 0195c72a7

  security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java
d7256802a 
  security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java
6af682a81 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6

  security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 4a7055dee 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 44523fb22 
  security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 68908b205 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerServiceDefService.java
d65b0d6ee 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 2ca0fd2fb



Diff: https://reviews.apache.org/r/68881/diff/7/


Testing
-------


Thanks,

Abhay Kulkarni


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message