ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abhay Kulkarni <akulka...@hortonworks.com>
Subject Re: Review Request 69072: RANGER-2203 : Review and update database schema for ranger policies to minimize database queries/updates, RANGER-2219: De-normalize schema for storing tags and related objects
Date Thu, 18 Oct 2018 22:01:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69072/
-----------------------------------------------------------

(Updated Oct. 18, 2018, 10:01 p.m.)


Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy.


Changes
-------

Addressed review comments on a related patch for ranger-0.7 (https://reviews.apache.org/r/68881/)


Bugs: RANGER-2203 and RANGER-2219
    https://issues.apache.org/jira/browse/RANGER-2203
    https://issues.apache.org/jira/browse/RANGER-2219


Repository: ranger


Description
-------

Currently, ranger policies are fully normalized and stored in a multiple Relational database
tables. There is a performance overhead incurred when retrieving a ranger policy, as multiple
database accesses are required to fully reconstruct it. This is significant when there are
large ranger policies (that is, the number of resources addressed by the policy is large),
and/or when there is a large number of ranger policies in an installation.

This Jira tracks alternate design of database schema, where a policy is stored in a de-normalized
way, in its entirely, in one database table (preferably as a JSON string).

Currently, tag-definitions, tags and service-resources are stored in database using a normalized
form. When constructing resource->tag mappings, this schema design may lead to a large
number of database accesses, thereby causing a major performance bottleneck when the number
of resource->tag associations is large.

Denormalized schema will reduce the number of database accesses, and improve overall performance
significantly.


Diffs (updated)
-----

  agents-common/pom.xml 09a32fa8b 
  agents-common/src/main/java/org/apache/ranger/authorization/utils/JsonUtils.java bc4a8b523

  agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceDef.java adafb9985

  agents-common/src/main/java/org/apache/ranger/plugin/store/TagStore.java 5918b1292 
  embeddedwebserver/scripts/ranger-admin-services.sh b8ca6c7d5 
  security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 32cf6db3b 
  security-admin/db/mysql/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/mysql/patches/036-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 2e577f3bd 
  security-admin/db/oracle/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/oracle/patches/036-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql bad32efbc 
  security-admin/db/postgres/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/postgres/patches/036-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 9482992e7

  security-admin/db/sqlanywhere/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/sqlanywhere/patches/036-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 85f328556 
  security-admin/db/sqlserver/patches/035-update-schema-for-x-policy.sql PRE-CREATION 
  security-admin/db/sqlserver/patches/036-denormalize-tag-tables.sql PRE-CREATION 
  security-admin/scripts/db_setup.py 02701c726 
  security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java c26f0a576

  security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 7875bc2a8 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 6c699cad9 
  security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java d29df930c 
  security-admin/src/main/java/org/apache/ranger/common/RangerServicePoliciesCache.java 0d5689a03

  security-admin/src/main/java/org/apache/ranger/db/RangerDaoManagerBase.java 5dffc0ef2 
  security-admin/src/main/java/org/apache/ranger/db/XXAccessTypeDefDao.java e04280bc8 
  security-admin/src/main/java/org/apache/ranger/db/XXDataMaskTypeDefDao.java 3dd4376e4 
  security-admin/src/main/java/org/apache/ranger/db/XXGroupDao.java 9a87b4cfd 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyConditionDefDao.java a12140aba

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemAccessDao.java 9be38decd 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemConditionDao.java 9b11545a0

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemDataMaskInfoDao.java 67c7e9961

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemGroupPermDao.java a6fd8c65b

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemRowFilterInfoDao.java 01a36a50d

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyItemUserPermDao.java 667265485

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefAccessTypeDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefConditionDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefDataMaskTypeDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefGroupDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefResourceDao.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/db/XXPolicyRefUserDao.java PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceDao.java a7157deb4 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyResourceMapDao.java 55d8c50e4

  security-admin/src/main/java/org/apache/ranger/db/XXResourceDefDao.java b2e311fef 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java ee0e40043 
  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java c9a1c2132

  security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java
364af6d0f 
  security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 40c3a887b 
  security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 129f3c13c 
  security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java e14f836e7 
  security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java c0dd88371 
  security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java f9e041a3f 
  security-admin/src/main/java/org/apache/ranger/db/XXUserDao.java 58bf4d891 
  security-admin/src/main/java/org/apache/ranger/entity/XXDBBase.java 8405eb3e4 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyBase.java 584a103ff 
  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefAccessType.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefCondition.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefDataMaskType.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefGroup.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefResource.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXPolicyRefUser.java PRE-CREATION

  security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c 
  security-admin/src/main/java/org/apache/ranger/entity/XXTag.java d26a0b079 
  security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java
PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java
PRE-CREATION 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 97b8d7192 
  security-admin/src/main/java/org/apache/ranger/service/RangerAuditFields.java e6a519a17

  security-admin/src/main/java/org/apache/ranger/service/RangerPolicyServiceBase.java 8d42165f7

  security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java
5cbe47ad3 
  security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java
d7256802a 
  security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java
6af682a81 
  security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6

  security-admin/src/main/java/org/apache/ranger/service/RangerTagDefServiceBase.java 408358c48

  security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa 
  security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java c96878d21

  security-admin/src/main/resources/META-INF/jpa_named_queries.xml cdf6ba655 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java c9db90a0e 
  security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 15344ea30 
  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 8f19ffd78

  security-admin/src/test/java/org/apache/ranger/service/TestRangerTagService.java 1175989e2



Diff: https://reviews.apache.org/r/69072/diff/2/

Changes: https://reviews.apache.org/r/69072/diff/1-2/


Testing
-------

Tested with local VM for migration, policy CRUD, policy browsing and policy enforcement in
Hive plugin.


Thanks,

Abhay Kulkarni


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message