Return-Path: X-Original-To: apmail-accumulo-dev-archive@www.apache.org Delivered-To: apmail-accumulo-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B6D0610B9C for ; Tue, 10 Feb 2015 18:22:38 +0000 (UTC) Received: (qmail 88911 invoked by uid 500); 10 Feb 2015 18:22:33 -0000 Delivered-To: apmail-accumulo-dev-archive@accumulo.apache.org Received: (qmail 88867 invoked by uid 500); 10 Feb 2015 18:22:33 -0000 Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list dev@accumulo.apache.org Received: (qmail 88845 invoked by uid 99); 10 Feb 2015 18:22:33 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Feb 2015 18:22:33 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id D4A8F1CDE76; Tue, 10 Feb 2015 18:22:32 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============6054451359879394422==" MIME-Version: 1.0 Subject: Re: Review Request 30817: ACCUMULO-3513 Delegation token support From: "Josh Elser" To: "Christopher Tubbs" Cc: "accumulo" , "Josh Elser" Date: Tue, 10 Feb 2015 18:22:32 -0000 Message-ID: <20150210182232.29075.19716@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Josh Elser" X-ReviewGroup: accumulo X-ReviewRequest-URL: https://reviews.apache.org/r/30817/ X-Sender: "Josh Elser" References: <20150210043253.29075.4226@reviews.apache.org> In-Reply-To: <20150210043253.29075.4226@reviews.apache.org> Reply-To: "Josh Elser" X-ReviewRequest-Repository: accumulo --===============6054451359879394422== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30817/ ----------------------------------------------------------- (Updated Feb. 10, 2015, 6:22 p.m.) Review request for accumulo and Christopher Tubbs. Changes ------- Fixed a bug where the default principal used from MapReduce wasn't the Kerberos principal from the UGI Bugs: ACCUMULO-3513 https://issues.apache.org/jira/browse/ACCUMULO-3513 Repository: accumulo Description ------- DelegationTokens are a way to enable MapReduce jobs (or any distributed task which does not have direct access to the user's kerberos credentials) to identify themself and communicate with Accumulo. This is implemented in the same style of Hadoop which does this to provide HDFS and YARN access in YARN jobs, as well as HBase (HBase also served as the base when I started coding). A DelegationToken (nothing more than a short-lived, on-demand password generated by Accumulo) is provided to users who request one through the SecurityOperations API. Secret keys are created, distributed and expired within Accumulo itself (Master creates and expires keys, distributes via ZK to tservers), which allow the master to create delegation tokens upon request and tabletservers to validate delegation tokens provided as the authentication mechanism by users. Tokens are valid for 7 days by default and new secret keys are rolled every day (configurable). RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI support we already have for "normal" kerberos -- thrift makes this really easy). YARN provides a secure way to pass delegation tokens from the client JVM to each YARN task (mapper/reducer) to prevent unwanted parties from stealing the token (password) and using it for their own (nefarious) purposes. The feature is transparent for users as long as they as using our MapReduce JCommander options classes. User documentation has been added to the kerberos chapter. Diffs (updated) ----- core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java 0cf081f core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java 7719e92 core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java 4b3b7ed core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java efeafc0 core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java 8470da4 core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java feb1ee7 core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java 072724b core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java b83a024 core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java f877ec6 core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java 5c7b780 core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java 5e0aa73 core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java b2b5150 core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java 5405ac0 core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java db88cfb core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73 core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java f5cfdd5 core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java e067e23 core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3 core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java b998179 core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationKey.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationTokenIdentifier.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/security/thrift/TDelegationToken.java PRE-CREATION core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java PRE-CREATION core/src/main/thrift/master.thrift d89e381 core/src/main/thrift/security.thrift 66235a8 core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java 2723273 core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java PRE-CREATION core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java PRE-CREATION core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java 3910f34 core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java PRE-CREATION core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java PRE-CREATION docs/src/main/asciidoc/chapters/kerberos.txt 05d7384 fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java 610b1bd fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java 5706cf3 proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java 84c3853 server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java bf56a7a server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java f85505d server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java f1f8963 server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java ab106a6 server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java cc7a7cd server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java 7adb46e server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java 51d50a1 server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java PRE-CREATION server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java 08fa55b server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java 49a60a6 server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java PRE-CREATION server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java PRE-CREATION server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java da0b07c server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java 1d7f90f server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java 6fcdd37 server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java 120692a server/master/src/main/java/org/apache/accumulo/master/Master.java be476de server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java 72cba26 server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java a5675dc test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java a2687bb test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java dba6ac9 test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java 4b5c3e7 test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java PRE-CREATION test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java 73048f6 test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java f68377a test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 60f8ec2 test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java 06b4303 test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java 3d48657 Diff: https://reviews.apache.org/r/30817/diff/ Testing ------- Many new unit tests. Additional integration tests. Standalone mapreduce jobs (continuous verify and terasort ingest) on a fully secure cluster. Thanks, Josh Elser --===============6054451359879394422==--