hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-2742) Provide strong authentication with a secure RPC engine
Date Thu, 17 Nov 2011 07:28:54 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151882#comment-13151882
] 

jiraposter@reviews.apache.org commented on HBASE-2742:
------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1991/
-----------------------------------------------------------

(Updated 2011-11-17 07:27:20.090955)


Review request for hbase.


Changes
-------

Updated patch addressing Ted's comments.


Summary
-------

This patch creates a new secure RPC engine for HBase, which provides Kerberos based authentication
of clients, and a token-based authentication mechanism for mapreduce jobs.  Primary components
of the patch are:

- a new maven profile for secure Hadoop/HBase: hadoop-0.20S
  - Secure Hadoop dependent classes are separated under a pseudo-module in the security/ directory.
 These source and test directories are only including if building the secure Hadoop profile
  - Currently the security classes get packaged with the regular HBase build artifacts.  We
need a way to at least override project.version, so we can append something like a "-security"
suffix indicating the additional security components.
  - The pseudo-module here is really a half-step forward.  It enables the security code to
be optionally included in the build for now, and sets up the structure for a security module.
 But we still will want to pursue full modularization (see HBASE-4336), which will allow packing
the security code in a separate build artifact.

- a new RPC engine providing kerberos and token-based authentication: org.apache.hadoop.hbase.ipc.SecureRpcEngine
  - implementation under security/src/main/java/org/apache/hadoop/hbase/ipc/
  - The implementation classes extend the existing HBaseClient and HBaseServer to share as
much of the RPC code as possible.  The main override is of the connection classes to allow
control over the SASL negotiation of secure connections

- existing RPC changes
  - The existing HBaseClient and HBaseServer have been modified to make subclassing possible
  - All references to Hadoop UserGroupInformation have been replaced with org.apache.hadoop.hbase.security.User
to insulate from future dependencies on specific Hadoop versions

- a coprocessor endpoint for obtaining new authentication tokens: TokenProvider, and supporting
classes for token generation and synchronization (incorporating HBASE-3615)
  - implementation is under security/src/main/java/org/apache/hadoop/hbase/security/token/
  - Secret keys for token generation and verification are synchronized throughout the cluster
in zookeeper, under /hbase/tokenauth/keys


To enable secure RPC, add the following configuration to hbase-site.xml:

  <property>
   <name>hadoop.security.authorization</name>
   <value>true</value>
  </property>
  <property>
   <name>hadoop.security.authentication</name>
   <value>kerberos</value>
  </property>
  <property>
   <name>hbase.rpc.engine</name>
   <value>org.apache.hadoop.hbase.ipc.SecureRpcEngine</value>
  </property>
  <property>
   <name>hbase.coprocessor.region.classes</name>
   <value>org.apache.hadoop.hbase.security.token.TokenProvider</value>
  </property>

In addition, the master and regionserver processes must be configured for kerberos authentication
using the properties:

 * hbase.(master|regionserver).keytab.file
 * hbase.(master|regionserver).kerberos.principal
 * hbase.(master|regionserver).kerberos.https.principal


This addresses bug HBASE-2742.
    https://issues.apache.org/jira/browse/HBASE-2742


Diffs (updated)
-----

  conf/hbase-policy.xml PRE-CREATION 
  pom.xml 2847416 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureClient.java PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureConnectionHeader.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/ipc/SecureServer.java PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java
PRE-CREATION 
  security/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION

  security/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION

  security/src/test/java/org/apache/hadoop/hbase/security/token/TestTokenAuthentication.java
PRE-CREATION 
  security/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java PRE-CREATION

  security/src/test/resources/hbase-site.xml PRE-CREATION 
  src/assembly/all.xml 3ad8ab3 
  src/main/java/org/apache/hadoop/hbase/HServerAddress.java f28240c 
  src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java cba7bd1 
  src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 904c107 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 4086829 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 054b92b 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 337da78 
  src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d9ba0ea 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java 9e0535e 
  src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 0dc5bea 
  src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java e8b2c9e 
  src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 1b5629a 
  src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java c8e030c 
  src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java db07ed1 
  src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java ad88b76 
  src/main/java/org/apache/hadoop/hbase/master/HMaster.java 19ed5b8 
  src/main/java/org/apache/hadoop/hbase/security/KerberosInfo.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/security/TokenInfo.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/security/User.java 00bd06d 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKLeaderManager.java PRE-CREATION 
  src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 4c5e844 
  src/main/resources/hbase-default.xml 6f98f5d 
  src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java c0634f4 
  src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java bb9ea45 
  src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKLeaderManager.java PRE-CREATION 

Diff: https://reviews.apache.org/r/1991/diff


Testing
-------

A full test suite run showed failures in TestMasterFailover (timeout) and TestAvroServer.
 I'll look into those.

Cluster testing (1 master + 3 slaves in ec2) with Kerberos authentication and map reduce token
authentication, using YCSB, RowCounter, PerformanceEvaluation.


Thanks,

Gary


                
> Provide strong authentication with a secure RPC engine
> ------------------------------------------------------
>
>                 Key: HBASE-2742
>                 URL: https://issues.apache.org/jira/browse/HBASE-2742
>             Project: HBase
>          Issue Type: Improvement
>          Components: ipc
>            Reporter: Gary Helmling
>            Priority: Critical
>             Fix For: 0.92.0
>
>
> The HBase RPC code (org.apache.hadoop.hbase.ipc.*) was originally forked off of Hadoop
RPC classes, with some performance tweaks added.  Those optimizations have come at a cost
in keeping up with Hadoop RPC changes however, both bug fixes and improvements/new features.
 
> In particular, this impacts how we implement security features in HBase (see HBASE-1697
and HBASE-2016).  The secure Hadoop implementation (HADOOP-4487) relies heavily on RPC changes
to support client authentication via kerberos and securing and mutual authentication of client/server
connections via SASL.  Making use of the built-in Hadoop RPC classes will gain us these pieces
for free in a secure HBase.
> So, I'm proposing that we drop the HBase forked version of RPC and convert to direct
use of Hadoop RPC, while working to contribute important fixes back upstream to Hadoop core.
 Based on a review of the HBase RPC changes, the key divergences seem to be:
> HBaseClient:
>  - added use of TCP keepalive (HBASE-1754)
>  - made connection retries and sleep configurable (HBASE-1815)
>  - prevent NPE if socket == null due to creation failure (HBASE-2443)
> HBaseRPC:
>  - mapping of method names <-> codes (removed in HBASE-2219)
> HBaseServer:
>  - use of TCP keep alives (HBASE-1754)
>  - OOME in server does not trigger abort (HBASE-1198)
> HbaseObjectWritable:
>  - allows List<> serialization
>  - includes it's own class <-> code mapping (HBASE-328)
> Proposed process is:
> 1. open issues with patches on Hadoop core for important fixes/adjustments from HBase
RPC (HBASE-1198, HBASE-1815, HBASE-1754, HBASE-2443, plus a pluggable ObjectWritable implementation
in RPC.Invocation to allow use of HbaseObjectWritable).
> 2. ship a Hadoop version with RPC patches applied -- ideally we should avoid another
copy-n-paste code fork, subject to ability to isolate changes from impacting Hadoop internal
RPC wire formats
> 3. if all Hadoop core patches are applied we can drop back to a plain vanilla Hadoop
version
> I realize there are many different opinions on how to proceed with HBase RPC, so I'm
hoping this issue will kick off a discussion on what the best approach might be.  My own motivation
is maximizing re-use of the authentication and connection security work that's already gone
into Hadoop core.  I'll put together a set of patches around #1 and #2, but obviously we need
some consensus around this to move forward.  If I'm missing other differences between HBase
and Hadoop RPC, please list as well.  Discuss!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message