ranger-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qiang Zhang <zhangqia...@zte.com.cn>
Subject Re: Review Request 68128: RANGER-2170:Ranger supports plugin to enable, monitor and manage Elasticsearch
Date Fri, 07 Sep 2018 02:48:11 GMT


> On 九月 6, 2018, 6:30 p.m., Ramesh Mani wrote:
> > ranger-elasticsearch-plugin-shim/conf/plugin-descriptor.properties
> > Lines 46 (patched)
> > <https://reviews.apache.org/r/68128/diff/2/?file=2072368#file2072368line46>
> >
> >     Why is -shim- is having the conf and classes which are core to the plugin it
should part of /plugin-elasticseach/conf folder.
> >     -shim is to hold the ranger-classloader.
> >     Please refer the existing plugin and see if that can be done.

Because unlike other Hadoop components, Elasticseach is designed to be pluggable.
To implement a new extension function, 
we need to organize code and configuration files according to the requirements of Elasticseach.
Some classes in ranger-elasticsearch-plugin-shim is necessary to mount on Elasticseach.
Other classes is due to the fact that Elasticseach itself does not support user authentication,
so this plugin should work with other Elasticsearch plugin to authenticate users,
such as Basic Authentication, Kerberos, LDAP, etc.
Or, in the future, we can realize user authentication in ranger-elasticsearch-plugin-shim.
So these classes can't sink into plugin-elasticseach.
And plugin-descriptor.properties is for Elasticseach to start the Ranger Elasticseach plugin.
But the files in the plugin-elasticseach/conf directory are for Ranger Elasticseach plugin.

The related directory structure after plugin installed is as follows:
1.ranger-elasticsearch-plugin-shim/conf/?
```
[elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ pwd
/home/elasticsearch/elasticsearch-6.2.2/plugins/ranger-elasticsearch-plugin
[elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ ll
-rwxrwxrwx. 1 elasticsearch hadoop 588337 4?  25 2017 commons-collections-3.2.2.jar
-rwxrwxrwx. 1 elasticsearch hadoop 284220 4?  25 2017 commons-lang-2.6.jar
-rwxrwxrwx. 1 elasticsearch hadoop   2547 6?  26 09:41 plugin-descriptor.properties
-rwxrwxrwx. 1 elasticsearch hadoop   1754 6?  26 09:27 plugin-security.policy
drwxrwxrwx. 2 elasticsearch hadoop   4096 7?  13 09:40 ranger-elasticsearch-plugin-impl
-rwxrwxrwx. 1 elasticsearch hadoop  20627 6?  26 09:36 ranger-elasticsearch-plugin-shim-1.1.0-SNAPSHOT.jar
-rwxrwxrwx. 1 elasticsearch hadoop  16799 6?  26 09:35 ranger-plugin-classloader-1.1.0-SNAPSHOT.jar
-rwxrwxrwx. 1 elasticsearch hadoop  26084 4?  25 2017 slf4j-api-1.7.5.jar
-rwxrwxrwx. 1 elasticsearch hadoop   8866 6?  26 15:30 slf4j-log4j12-1.7.10.jar
```

2.plugin-elasticseach/conf?
```
[elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ pwd
/home/elasticsearch/elasticsearch-6.2.2/config/ranger-elasticsearch-plugin
[elasticsearch@zdh-11 ranger-elasticsearch-plugin]$ ll
-rwxrwxrwx. 1 elasticsearch hadoop 9548 6?  26 14:15 ranger-elasticsearch-audit.xml
-rwxrwxrwx. 1 elasticsearch hadoop 2773 6?  26 14:15 ranger-elasticsearch-security.xml
-rwxrwxrwx. 1 elasticsearch hadoop 1917 6?  26 14:15 ranger-policymgr-ssl.xml
-rwxrwxrwx. 1 elasticsearch hadoop   83 6?  26 14:15 ranger-security.xml
```

In addition, I have developed 2 plugins: Ranger Kylin Plugin and Ranger Sqoop2 Plugin.
In order to meet the requirements of Elasticsearch plugins 
and take into account the general design principles of Ranger plugins,
I think the implementation of Ranger Elasticsearch plugin is reasonable at present.
Try install this plugin , and you can get a better understanding of the implementation of
it.


- Qiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68128/#review208418
-----------------------------------------------------------


On 八月 14, 2018, 8:46 a.m., Qiang Zhang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68128/
> -----------------------------------------------------------
> 
> (Updated 八月 14, 2018, 8:46 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O hEigeartaigh, Gautam
Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, Selvamohan Neethiraj, sam  rome, Venkat
Ranganathan, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2170
>     https://issues.apache.org/jira/browse/RANGER-2170
> 
> 
> Repository: ranger
> 
> 
> Description
> -------
> 
> Elasticsearch is a distributed, RESTful search and analytics engine capable of solving
a growing number of use cases. 
> Like Apache Solr, it is also an index server based on Lucence.
> Ranger supports plugin to enable, monitor and manage Elasticsearch,
> to control index security of Elasticsearch.
> 
> As there is X-Pack plugin for the Elasticsearch, but it is not free.
> X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting,

> and graph capabilities into one easy-to-install package.
> We refer to the Indices Privileges design of X-Pack,
> by keeping the permissions consistent,
> to make user use ranger Elasticsearch plugin easily.
> Reference X-Pack Indices Privileges:
> https://www.elastic.co/guide/en/x-pack/current/security-privileges.html
> 
> Here we develop Ranger Elasticsearch plugin, based on Elasticsearch version 6.2.2.
> Elasticsearch 6.2.2 was released in February 20, 2018, reference release-notes:
> https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.2.html
> Not like other system, Elasticsearch has no basic authentication, 
> it uses X-pack plugin to support basic authentication, 
> role-based access control, SSL/TLS encryption, LDAP and so on.
> Not like X-pack, our Ranger Elasticsearch plugin is designed to do authorization,
> it is to control index of Elasticsearch without authentication,
> this plugin should work with other Elasticsearch plugin to authenticate users.
> 
> 
> Diffs
> -----
> 
>   agents-common/scripts/enable-agent.sh ce0dc8c 
>   agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java e654f2b

>   agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
5e74da8 
>   agents-common/src/main/resources/service-defs/ranger-servicedef-elasticsearch.json
PRE-CREATION 
>   plugin-elasticsearch/.gitignore PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-audit-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-audit.xml PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-security-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-elasticsearch-security.xml PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION 
>   plugin-elasticsearch/conf/ranger-policymgr-ssl.xml PRE-CREATION 
>   plugin-elasticsearch/pom.xml PRE-CREATION 
>   plugin-elasticsearch/scripts/install.properties PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/RangerServiceElasticsearch.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchClient.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchResourceMgr.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilege.java
PRE-CREATION 
>   plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/privilege/IndexPrivilegeUtils.java
PRE-CREATION 
>   pom.xml 8d7ea13 
>   ranger-elasticsearch-plugin-shim/.gitignore PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/conf/plugin-descriptor.properties PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/conf/plugin-security.policy PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/pom.xml PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAccessControl.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticsearchPlugin.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/authc/user/UsernamePasswordToken.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/rest/filter/RangerSecurityRestFilter.java
PRE-CREATION 
>   ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/utils/RequestUtils.java
PRE-CREATION 
>   src/main/assembly/admin-web.xml d0f3545 
>   src/main/assembly/plugin-elasticsearch.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/68128/diff/2/
> 
> 
> Testing
> -------
> 
> #Test Steps:
> 
> 1.Intall
> Ranger Elasticsearch Plugin Installation Guide	
> https://cwiki.apache.org/confluence/display/RANGER/Elasticsearch+Plugin
> Include install Elasticsearch and Ranger Elasticsearch Plugin,
> and verify install result.
> 
> 2.Create policy in Ranger Admin
> User "elasticsearch" has all permissions on all indices.
> User "yuwen" has permission "read" on index "twitter".
> 
> 3.Test permission
> 
> 3.1 successful:
> curl -u elasticsearch:xxx -X GET "localhost:9200/twitter/_stats?pretty"
> curl -u elasticsearch:xxx -X GET "localhost:9200/twitter2/_stats?pretty"
> curl -u yuwen:xxx -X GET "localhost:9200/twitter/_stats?pretty"
> 
> 3.2 failed:
> curl -X GET "localhost:9200/twitter/_stats?pretty"
> {
>   "error" : {
>     "root_cause" : [
>       {
>         "type" : "status_exception",
>         "reason" : "Error: User is null, the request requires user authentication."
>       }
>     ],
>     "type" : "status_exception",
>     "reason" : "Error: User is null, the request requires user authentication."
>   },
>   "status" : 401
> }
> 
> curl -u yuwen:xxx -X GET "localhost:9200/twitter2/_stats?pretty"
> {
>   "error" : {
>     "root_cause" : [
>       {
>         "type" : "status_exception",
>         "reason" : "Error: User[yuwen] could not do action[indices:monitor/stats] on
index[twitter2]"
>       }
>     ],
>     "type" : "status_exception",
>     "reason" : "Error: User[yuwen] could not do action[indices:monitor/stats] on index[twitter2]"
>   },
>   "status" : 403
> }
> 
> 
> Thanks,
> 
> Qiang Zhang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message