Return-Path: X-Original-To: apmail-accumulo-dev-archive@www.apache.org Delivered-To: apmail-accumulo-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1C4C710BCE for ; Tue, 10 Feb 2015 18:27:00 +0000 (UTC) Received: (qmail 4410 invoked by uid 500); 10 Feb 2015 18:27:00 -0000 Delivered-To: apmail-accumulo-dev-archive@accumulo.apache.org Received: (qmail 4369 invoked by uid 500); 10 Feb 2015 18:26:59 -0000 Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list dev@accumulo.apache.org Received: (qmail 4346 invoked by uid 99); 10 Feb 2015 18:26:59 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Feb 2015 18:26:59 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 229D51CDE76; Tue, 10 Feb 2015 18:26:59 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============0350163214818871791==" MIME-Version: 1.0 Subject: Re: Review Request 30817: ACCUMULO-3513 Delegation token support From: "Josh Elser" To: "Christopher Tubbs" Cc: "accumulo" , "Josh Elser" Date: Tue, 10 Feb 2015 18:26:59 -0000 Message-ID: <20150210182659.29076.46468@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Josh Elser" X-ReviewGroup: accumulo X-ReviewRequest-URL: https://reviews.apache.org/r/30817/ X-Sender: "Josh Elser" References: <20150210180712.29076.68334@reviews.apache.org> In-Reply-To: <20150210180712.29076.68334@reviews.apache.org> Reply-To: "Josh Elser" X-ReviewRequest-Repository: accumulo --===============0350163214818871791== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java, line 173 > > > > > > Why does OutputFormat depend on utils for InputFormat? Should probably push this method down to ConfiguratorBase. Good point. Done. > On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java, line 88 > > > > > > It's not just for PasswordToken. It's for any serialized token, which might be provided by an add-on package. Do we care about any hypothetical tokens that may or may not exist? My main concern is noting that DelegationTokens (and KerberosTokens, although they're not relevant at all really) are not serialized in the Configuration. Recommendations on how you would prefer this to read? > On Feb. 10, 2015, 6:07 p.m., Christopher Tubbs wrote: > > core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java, line 87 > > > > > > Would be helpful if review did not include thrift generated classes (easier to evaluate). Uhh, I'll see if I can rip that out of the patch.. - Josh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30817/#review71810 ----------------------------------------------------------- On Feb. 10, 2015, 6:22 p.m., Josh Elser wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30817/ > ----------------------------------------------------------- > > (Updated Feb. 10, 2015, 6:22 p.m.) > > > Review request for accumulo and Christopher Tubbs. > > > Bugs: ACCUMULO-3513 > https://issues.apache.org/jira/browse/ACCUMULO-3513 > > > Repository: accumulo > > > Description > ------- > > DelegationTokens are a way to enable MapReduce jobs (or any distributed task which does not have direct access to the user's kerberos credentials) to identify themself and communicate with Accumulo. This is implemented in the same style of Hadoop which does this to provide HDFS and YARN access in YARN jobs, as well as HBase (HBase also served as the base when I started coding). > > A DelegationToken (nothing more than a short-lived, on-demand password generated by Accumulo) is provided to users who request one through the SecurityOperations API. Secret keys are created, distributed and expired within Accumulo itself (Master creates and expires keys, distributes via ZK to tservers), which allow the master to create delegation tokens upon request and tabletservers to validate delegation tokens provided as the authentication mechanism by users. Tokens are valid for 7 days by default and new secret keys are rolled every day (configurable). > > RPC is implemented SASL's DIGEST-MD5 mechanism (in addition to the GSSAPI support we already have for "normal" kerberos -- thrift makes this really easy). YARN provides a secure way to pass delegation tokens from the client JVM to each YARN task (mapper/reducer) to prevent unwanted parties from stealing the token (password) and using it for their own (nefarious) purposes. The feature is transparent for users as long as they as using our MapReduce JCommander options classes. > > User documentation has been added to the kerberos chapter. > > > Diffs > ----- > > core/src/main/java/org/apache/accumulo/core/Constants.java 0229d4e > core/src/main/java/org/apache/accumulo/core/cli/ClientOpts.java 216f32d > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnDefaultTable.java 0cf081f > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOnRequiredTable.java 7719e92 > core/src/main/java/org/apache/accumulo/core/cli/MapReduceClientOpts.java 4b3b7ed > core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java efeafc0 > core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java 8470da4 > core/src/main/java/org/apache/accumulo/core/client/impl/SecurityOperationsImpl.java feb1ee7 > core/src/main/java/org/apache/accumulo/core/client/impl/ThriftTransportKey.java 072724b > core/src/main/java/org/apache/accumulo/core/client/mapred/AbstractInputFormat.java b83a024 > core/src/main/java/org/apache/accumulo/core/client/mapred/AccumuloOutputFormat.java f877ec6 > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java 5c7b780 > core/src/main/java/org/apache/accumulo/core/client/mapreduce/AccumuloOutputFormat.java 5e0aa73 > core/src/main/java/org/apache/accumulo/core/client/mapreduce/impl/DelegationTokenStub.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java b2b5150 > core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/InputConfigurator.java 5405ac0 > core/src/main/java/org/apache/accumulo/core/client/mock/MockSecurityOperations.java db88cfb > core/src/main/java/org/apache/accumulo/core/client/security/tokens/DelegationToken.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/conf/Property.java 68fac73 > core/src/main/java/org/apache/accumulo/core/master/thrift/MasterClientService.java f5cfdd5 > core/src/main/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandler.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/SaslConnectionParams.java e067e23 > core/src/main/java/org/apache/accumulo/core/rpc/SaslDigestCallbackHandler.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/rpc/ThriftUtil.java d880fb3 > core/src/main/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifier.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/security/SystemPermission.java b998179 > core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationKey.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/security/thrift/TAuthenticationTokenIdentifier.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/security/thrift/TDelegationToken.java PRE-CREATION > core/src/main/java/org/apache/accumulo/core/util/ThriftMessageUtil.java PRE-CREATION > core/src/main/thrift/master.thrift d89e381 > core/src/main/thrift/security.thrift 66235a8 > core/src/test/java/org/apache/accumulo/core/client/impl/ThriftTransportKeyTest.java 2723273 > core/src/test/java/org/apache/accumulo/core/client/security/tokens/DelegationTokenTest.java PRE-CREATION > core/src/test/java/org/apache/accumulo/core/rpc/SaslClientDigestCallbackHandlerTest.java PRE-CREATION > core/src/test/java/org/apache/accumulo/core/rpc/SaslConnectionParamsTest.java 3910f34 > core/src/test/java/org/apache/accumulo/core/security/AuthenticationTokenIdentifierTest.java PRE-CREATION > core/src/test/java/org/apache/accumulo/core/util/ThriftMessageUtilTest.java PRE-CREATION > docs/src/main/asciidoc/chapters/kerberos.txt 05d7384 > fate/src/main/java/org/apache/accumulo/fate/zookeeper/IZooReader.java 610b1bd > fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooReader.java 5706cf3 > proxy/src/main/java/org/apache/accumulo/proxy/Proxy.java f9039be > server/base/src/main/java/org/apache/accumulo/server/AccumuloServerContext.java 84c3853 > server/base/src/main/java/org/apache/accumulo/server/master/state/MetaDataStateStore.java bf56a7a > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerConnectionParams.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/rpc/SaslServerDigestCallbackHandler.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/rpc/TCredentialsUpdatingInvocationHandler.java f85505d > server/base/src/main/java/org/apache/accumulo/server/rpc/TServerUtils.java f1f8963 > server/base/src/main/java/org/apache/accumulo/server/rpc/UGIAssumingProcessor.java ab106a6 > server/base/src/main/java/org/apache/accumulo/server/security/AuditedSecurityOperation.java cc7a7cd > server/base/src/main/java/org/apache/accumulo/server/security/SecurityOperation.java 7adb46e > server/base/src/main/java/org/apache/accumulo/server/security/SystemCredentials.java 51d50a1 > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationKey.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManager.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManager.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributor.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcher.java PRE-CREATION > server/base/src/main/java/org/apache/accumulo/server/security/handler/KerberosAuthenticator.java 08fa55b > server/base/src/test/java/org/apache/accumulo/server/AccumuloServerContextTest.java 49a60a6 > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslDigestCallbackHandlerTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/rpc/SaslServerConnectionParamsTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationKeyTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenKeyManagerTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/security/delegation/AuthenticationTokenSecretManagerTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyDistributorTest.java PRE-CREATION > server/base/src/test/java/org/apache/accumulo/server/security/delegation/ZooAuthenticationKeyWatcherTest.java PRE-CREATION > server/gc/src/main/java/org/apache/accumulo/gc/SimpleGarbageCollector.java da0b07c > server/gc/src/test/java/org/apache/accumulo/gc/GarbageCollectWriteAheadLogsTest.java 1d7f90f > server/gc/src/test/java/org/apache/accumulo/gc/SimpleGarbageCollectorTest.java 6fcdd37 > server/gc/src/test/java/org/apache/accumulo/gc/replication/CloseWriteAheadLogReferencesTest.java 120692a > server/master/src/main/java/org/apache/accumulo/master/Master.java be476de > server/master/src/main/java/org/apache/accumulo/master/MasterClientServiceHandler.java 72cba26 > server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java a5675dc > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousBatchWalker.java a2687bb > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousIngest.java dba6ac9 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousMoru.java 4b5c3e7 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousOpts.java PRE-CREATION > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousQuery.java 73048f6 > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousScanner.java f68377a > test/src/main/java/org/apache/accumulo/test/continuous/ContinuousWalk.java 60f8ec2 > test/src/test/java/org/apache/accumulo/harness/MiniClusterHarness.java 06b4303 > test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java 3d48657 > > Diff: https://reviews.apache.org/r/30817/diff/ > > > Testing > ------- > > Many new unit tests. Additional integration tests. Standalone mapreduce jobs (continuous verify and terasort ingest) on a fully secure cluster. > > > Thanks, > > Josh Elser > > --===============0350163214818871791==--