accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser" <josh.el...@gmail.com>
Subject Re: Review Request 30817: ACCUMULO-3513 Delegation token support
Date Tue, 10 Feb 2015 18:22:32 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30817/
-----------------------------------------------------------

(Updated Feb. 10, 2015, 6:22 p.m.)


Review request for accumulo and Christopher Tubbs.


Changes
-------

Fixed a bug where the default principal used from MapReduce wasn't the Kerberos principal
from the UGI


Bugs: ACCUMULO-3513
    https://issues.apache.org/jira/browse/ACCUMULO-3513


Repository: accumulo


Description
-------

DelegationTokens are a way to enable MapReduce jobs (or any distributed task which does not
have direct access to the user's kerberos credentials) to identify themself and communicate
with Accumulo. This is implemented in the same style of Hadoop which does this to provide
HDFS and YARN access in YARN jobs, as well as HBase (HBase also served as the base when I
started coding).

A DelegationToken (nothing more than a short-lived, on-demand password generated by Accumulo)
is provided to users who request one through the SecurityOperations API. Secret keys are created,
distributed and expired within Accumulo itself (Master creates and expires keys, distributes
via ZK to tservers), which allow the master to create delegation tokens upon request and tabletservers
to validate delegation tokens provided as the authentication mechanism by users. Tokens are
valid for 7 days by default and new secret keys are rolled every day (configurable).

RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI support we already
have for "normal" kerberos -- thrift makes this really easy). YARN provides a secure way to
pass delegation tokens from the client JVM to each YARN task (mapper/reducer) to prevent unwanted
parties from stealing the token (password) and using it for their own (nefarious) purposes.
The feature is transparent for users as long as they as using our MapReduce JCommander options
classes.

User documentation has been added to the kerberos chapter.


Diffs (updated)
-----

  core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e 
  core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d 
  core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java 0cf081f

  core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java 7719e92

  core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java 4b3b7ed 
  core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java efeafc0

  core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java 8470da4 
  core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java feb1ee7

  core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java 072724b

  core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java b83a024

  core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java f877ec6

  core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java 5c7b780

  core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java 5e0aa73

  core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java
b2b5150 
  core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java
5405ac0 
  core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java db88cfb

  core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73 
  core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java f5cfdd5

  core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java e067e23 
  core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3 
  core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java b998179 
  core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationKey.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationTokenIdentifier.java
PRE-CREATION 
  core/src/main/java/org/apache/accumulo/core/security/thrift/TDelegationToken.java PRE-CREATION

  core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java PRE-CREATION 
  core/src/main/thrift/master.thrift d89e381 
  core/src/main/thrift/security.thrift 66235a8 
  core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java 2723273

  core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java
PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java
PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java 3910f34 
  core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java
PRE-CREATION 
  core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java PRE-CREATION

  docs/src/main/asciidoc/chapters/kerberos.txt 05d7384 
  fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java 610b1bd 
  fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java 5706cf3 
  proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be 
  server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java 84c3853

  server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java
bf56a7a 
  server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java
f85505d 
  server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java f1f8963 
  server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java ab106a6

  server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java
cc7a7cd 
  server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java 7adb46e

  server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java 51d50a1

  server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java
PRE-CREATION 
  server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java
08fa55b 
  server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java 49a60a6

  server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java
PRE-CREATION 
  server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java
PRE-CREATION 
  server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java da0b07c 
  server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java 1d7f90f

  server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java 6fcdd37 
  server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java
120692a 
  server/master/src/main/java/org/apache/accumulo/master/Master.java be476de 
  server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java 72cba26

  server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java a5675dc 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java a2687bb

  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java dba6ac9 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java 4b5c3e7 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java PRE-CREATION

  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java 73048f6 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java f68377a 
  test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 60f8ec2 
  test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java 06b4303 
  test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java 3d48657 

Diff: https://reviews.apache.org/r/30817/diff/


Testing
-------

Many new unit tests. Additional integration tests. Standalone mapreduce jobs (continuous verify
and terasort ingest) on a fully secure cluster.


Thanks,

Josh Elser


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message