hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Helmling" <ghelml...@gmail.com>
Subject Re: Review Request: HBASE-2742, HBASE-2016: Port of secure Hadoop RPC changes and integration with HBase RPC protocols
Date Mon, 09 Aug 2010 21:15:33 GMT

> On 2010-08-04 22:40:35, stack wrote:
> > conf/hadoop-policy.xml, line 1
> > <http://review.cloudera.org/r/406/diff/2/?file=3872#file3872line1>
> >
> >     Should it be named hbase-policy.xml or is it a case of the name of the file
being hard-coded (as it is for metrics config -- see hadoop-metrics.xml in the hbase/conf

The default is down in org.apache.hadoop.security.authorize.ServiceAuthorizationManager, but
looks like we can override it with "hadoop.policy.file" in config.  I'll do that and update.

> On 2010-08-04 22:40:35, stack wrote:
> > pom.xml, line 480
> > <http://review.cloudera.org/r/406/diff/2/?file=3873#file3873line480>
> >
> >     How much of this patch do you want to commit to TRUNK Gary? Would hbase 0.90.0
have a dependency on secure hadoop and if so are there implications (performance?) or are
you thinking it secure hbase a separate offering, somehow?

At the moment, none of this is really eligible for trunk, due to dependency on secure hadoop.
 I'm looking at enabling pluggable RPC in HBASE-2321 as a way to introduce secure RPC without
the hard dependency.  Don't know how messy that will get, but I should have a good idea in
the next few days.

The idea would be to make secure RPC a conditional build target and enable optional configuration
of it as the RPC implementation, so that it doesn't block the ability of HBase to run on Hadoop
0.20 versions without the security features.

As long as that proves out, I would use HBASE-2321 as the framework for pluggable RPC, with
a new JIRA for the secure RPC plugin.

> On 2010-08-04 22:40:35, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java, line 85
> > <http://review.cloudera.org/r/406/diff/2/?file=3888#file3888line85>
> >
> >     What changed in here?

Looks like just a whitespace change, I'll remove.  In one of the intermediate changes I had
wrapped this in UserGroupInformation.doAs() to enable execution of RS's as different users
in LocalHBaseCluster (since some tests require this).  But I would up moving out to MiniHBaseCluster
to keep it isolated to test cases.

> On 2010-08-04 22:40:35, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java, line 116
> > <http://review.cloudera.org/r/406/diff/2/?file=3875#file3875line116>
> >
> >     Man, I hate this final stuff (I know its not you Gary.... you're just bringing
it over... thats fine... but I hate it anyways).

Understood and I don't disagree. :)

Most of this RPC code is not very extensible and sometimes serves a common case, but sometimes
frustrates it.  Maybe HBASE-2321 can help towards trying alternatives.  

> On 2010-08-04 22:40:35, stack wrote:
> > src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java, line 1
> > <http://review.cloudera.org/r/406/diff/2/?file=3874#file3874line1>
> >
> >     If we are running in insecure mode, does this thing send the only single-byte
or whatever it was identifying the rpc?  Or, to ask in another way, does insecure rpc have
same character with this patch as it does w/o it?

This does change the over the wire format for both secure and insecure cases in an incompatible

Existing format is:
version (3)
UGI ticket
  group names length
    each group name

However the UGI ticket always seems to be null so none of that is passed.

New format is:

version (4)
auth method (single byte code)
short class name (HMasterInterface, HRegionInterface, HMasterRegionInterface)
username present? (true|false)
+ if true
  real username present? (true|false)
  + if true
    real username

- Gary

This is an automatically generated e-mail. To reply, visit:

On 2010-07-29 12:40:06, Gary Helmling wrote:
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> http://review.cloudera.org/r/406/
> -----------------------------------------------------------
> (Updated 2010-07-29 12:40:06)
> Review request for hbase.
> Summary
> -------
> This patch ports over the secure Hadoop RPC changes from the latest Yahoo 0.20 based
branch (yahoo-hadoop-0.20.104).  This patch is produced against HBase trunk, but is targeted
as the first step in a "security" feature branch for a full role-based access control implementation
> RPC Changes
> --------------------
> The primary changes are updates from the classes:
> org.apache.hadoop.ipc.Client -> org.apache.hadoop.hbase.ipc.HBaseClient
> org.apache.hadoop.ipc.RPC -> org.apache.hadoop.hbase.ipc.HBaseRPC
> org.apache.hadoop.ipc.Server -> org.apache.hadoop.hbase.ipc.HBaseServer
> The new classes were also ported:
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient
> org.apache.hadoop.hbase.security.HBaseSaslRpcServer
> Due to type dependencies on the Hadoop RPC classes, the original Hadoop SaslRpc* classes
could not be used.
> The RPC port provides client authentication via Kerberos, and SASL negotiation of client
server connections for mutual authentication and optionally encryption, so it also provides
the authentication functionality for HBASE-2016.  The ported RPC code contains dependencies
on other classes in secure Hadoop/Hadoop trunk, preventing it from currently running on 0.20
branches missing the security changes.
> Process Authentication
> ---------------------------
> The HMaster and HRegionServer processes have been updated to allow configuration of the
Kerberos principals used to run the processes.  The new configuration parameters are:
> * hbase.master.keytab.file - Path to the keytab file containing the master principal's
> * hbase.master.kerberos.principal - Kerberos principal name used to login the HMaster
> * hbase.master.kerberos.https.principal - Kerberos principal name used to login the HMaster
info server
> * hbase.regionserver.keytab.file - Path to the keytab file containing the region server's
> * hbase.regionserver.kerberos.principal - Kerberos principal name used to login the HRegionServer
> * hbase.regionserver.kerberos.https.principal - Kerberos principal name used to login
the HRegionServer info server
> The new class org.apache.hadoop.hbase.security.HBasePolicyProvider and new file conf/hadoop-policy.xml
allow restriction of the users and groups permitting to utilize each of the RPC protocol interfaces
(HMasterInterface, HMasterRegionInterface, HRegionInterface).
> Testing Updates
> --------------------
> Parts of the test code (org.apache.hadoop.hbase.HBaseTestingUtility and org.apache.hadoop.hbase.MiniHBaseCluster)
were directly using the internal Hadoop UnixUserGroupInformation class to manipulate process
ownership for testing.  These have been updated to use UserGroupInformation.doAs() instead.
> This addresses bugs HBASE-2016 and HBASE-2742.
>     http://issues.apache.org/jira/browse/HBASE-2016
>     http://issues.apache.org/jira/browse/HBASE-2742
> Diffs
> -----
>   conf/hadoop-policy.xml PRE-CREATION 
>   pom.xml 2d3d75a 
>   src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 2b5eeb6 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 9873172 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java d88c12d 
>   src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java d3c6c21 
>   src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java bd48a4b 
>   src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java 71a0447 
>   src/main/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java 1157fe1 
>   src/main/java/org/apache/hadoop/hbase/ipc/Status.java PRE-CREATION 
>   src/main/java/org/apache/hadoop/hbase/master/HMaster.java e4bd30d 
>   src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6a54736 
>   src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION

>   src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java 280b91d 
>   src/main/resources/hbase-default.xml e3a9669 
>   src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 4d09fe9 
>   src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 9c49e36 
>   src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java 0b47975 
>   src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java c982662 
> Diff: http://review.cloudera.org/r/406/diff
> Testing
> -------
> Thanks,
> Gary

View raw message