accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Hughes <jn...@virginia.edu>
Subject Re: kerberos auth, getDelegationToken
Date Sun, 07 Jun 2015 20:51:57 GMT
Thanks.  In terms of knobs for Kerberos, etc. would that be a small enough
change to make sense in the 1.7.x series?  If so, and if we end up needing
it, I'd definitely be interested in contributing it back.

On Sun, Jun 7, 2015 at 4:36 PM, Josh Elser <josh.elser@gmail.com> wrote:

> Check out the accumulo-maven-plugin:
>
> https://accumulo.apache.org/release_notes/1.6.0.html#maven-plugin
>
> This will make it pretty simple to start a single MAC, run your tests and
> then stop it. A word of caution, I don't think it's heavily used, nor do I
> think it contains support for configuration all the knobs you could using
> MAC in your code (e.g. Kerberos, for one).
>
> A full pom example can be seen at
> https://github.com/apache/accumulo/blob/1.7.0/maven-plugin/src/it/plugin-test/pom.xml
>
> James Hughes wrote:
>
>> Josh,
>>
>> Thanks.  That's more or less what I expected.
>>
>> As we work to transition from Mock to MiniAccumulo, I'd want to change
>> from spinning up lots of MockInstances to one MiniAccumulo.  To
>> understand that pattern, do I basically just need to read through test
>> sub-module and the test/pom.xml?  Are there any other resources I should
>> be checking out?
>>
>> Cheers,
>>
>> Jim
>>
>> On Sun, Jun 7, 2015 at 1:37 PM, Josh Elser <josh.elser@gmail.com
>> <mailto:josh.elser@gmail.com>> wrote:
>>
>>     MiniAccumulo, yes. MockAccumulo, no. In general, we've near
>>     completely moved away from MockAccumulo. I wouldn't be surprised if
>>     it gets deprecated and removed soon.
>>
>>
>> https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java
>>
>>     Apache Directory provides a MiniKdc that can be used easily w/
>>     MiniAccumulo. Many of the integration tests have already been
>>     altered to support running w/ or w/o kerberos.
>>
>>     James Hughes wrote:
>>
>>         Hi all,
>>
>>         For GeoMesa, stats writing is quite secondary and optional, so
>>         we can
>>         sort that out as a follow-on to seeing GeoMesa work with
>>         Accumulo 1.7.
>>
>>         I haven't had a chance to read in details yet, so forgive me if
>>         this is
>>         covered in the docs.  Does either Mock or MiniAccumulo provide
>>         enough
>>         hooks to test out Kerberos integration effectively?  I suppose I'm
>>         really asking what kind of testing environment a project like
>>         GeoMesa
>>         would need to use to test out Accumulo 1.7.
>>
>>         Even though MockAccumulo has a number of limitations, it is rather
>>         effective in unit tests which can be part of a quick  build.
>>
>>         Thanks,
>>
>>         Jim
>>
>>         On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen
>>         <xchenum@gmail.com <mailto:xchenum@gmail.com>
>>         <mailto:xchenum@gmail.com <mailto:xchenum@gmail.com>>> wrote:
>>
>>              Nope, I am running the example as what the readme file
>>         suggested:
>>
>>              java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
>>              org.geomesa.QuickStart -instanceId somecloud -zookeepers
>>         "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
>>              -tableName sometable
>>
>>              I'll raise this question with the geomesa folks, but you're
>>         right that
>>              I can ignore it for now...
>>
>>              Thanks!
>>              -Simon
>>
>>
>>              On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser
>>         <josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>>         <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
>> wrote:
>>          > Are you running it via `mvn exec:java` by chance or netbeans?
>>          >
>>          >
>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
>>          >
>>          > If that's just a background thread writing in Stats, it might
>>              just be a
>>          > factor of how you're invoking the program and you can ignore
>> it.
>>              I don't
>>          > know enough about the inner-workings of GeoMesa to say one way
>> or
>>              the other.
>>          >
>>          >
>>          > Xu (Simon) Chen wrote:
>>          >>
>>          >> Josh,
>>          >>
>>          >> Everything works well, except for one thing :-)
>>          >>
>>          >> I am running geomesa-quickstart program that ingest some data
>>              and then
>>          >> perform a simple query:
>>          >> https://github.com/geomesa/geomesa-quickstart
>>          >>
>>          >> For some reason, the following error is emitted consistently
>>         at the
>>          >> end of the execution, after outputting the correct result:
>>          >> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error,
>> will
>>              retry
>>          >> java.lang.InterruptedException
>>          >>          at java.lang.Object.wait(Native Method)
>>          >>          at java.lang.Object.wait(Object.java:503)
>>          >>          at
>>          >>
>>
>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
>>          >>          at
>>              org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
>>          >>          at
>>          >>
>>
>> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
>>          >>          at
>>          >>
>>
>> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
>>          >>          at
>>          >>
>>         org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
>>          >>          at
>>          >>
>>         org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
>>          >>          at
>>          >>
>>
>>
>> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
>>          >>          at
>>          >>
>>
>>
>> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
>>          >>          at
>>          >>
>>
>>
>> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
>>          >>          at
>>          >>
>>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
>>          >>          at
>>          >>
>>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
>>          >>          at
>>              scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>>          >>          at
>>          >>
>>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
>>          >>          at
>>          >>
>>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
>>          >>          at
>>          >>
>>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
>>          >>          at
>>          >>
>>
>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>          >>          at
>>          >>
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>          >>          at
>>          >>
>>
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>          >>          at
>>          >>
>>
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>          >>          at
>>          >>
>>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>          >>          at
>>          >>
>>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>          >>          at java.lang.Thread.run(Thread.java:745)
>>          >>
>>          >> This is more annoying than a real problem. I am new to both
>>         accumulo
>>          >> and geomesa, but I am curious what the problem might be.
>>          >>
>>          >> Thanks!
>>          >> -Simon
>>          >>
>>          >>
>>          >> On Sat, Jun 6, 2015 at 8:01 PM, Josh
>>         Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>>         <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
>> wrote:
>>         > >>
>>         > >> Great! Glad to hear it. Please let us know how it works out!
>>         > >>
>>         > >>
>>         > >> Xu (Simon) Chen wrote:
>>         > >>>
>>         > >>> Josh,
>>         > >>>
>>         > >>> You're right again.. Thanks!
>>         > >>>
>>         > >>> My ansible play actually pushed client.conf to all the
>> server
>>              config
>>         > >>> directories, but didn't do anything for the clients, and
>>         that's my
>>         > >>> problem. Now kerberos is working great for me.
>>         > >>>
>>         > >>> Thanks again!
>>         > >>> -Simon
>>         > >>>
>>         > >>> On Sat, Jun 6, 2015 at 5:04 PM, Josh
>>              Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>>         <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
>>
>>
>>          >>>> wrote:
>>          >>>>>
>>          >>>>> Simon,
>>          >>>>>
>>          >>>>> Did you create a client configuration file
>>         (~/.accumulo/config or
>>          >>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure
>>              Accumulo clients
>>          >>>>> to
>>          >>>>> actually use SASL when you're trying to use Kerberos
>>              authentication.
>>          >>>>> Your
>>          >>>>> server is expecting that, but I would venture a guess
that
>>              your client
>>          >>>>> isn't.
>>          >>>>>
>>          >>>>> See
>>          >>>>>
>>          >>>>>
>>
>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
>>          >>>>>
>>          >>>>>
>>          >>>>> Xu (Simon) Chen wrote:
>>          >>>>>>
>>          >>>>>> Josh,
>>          >>>>>>
>>          >>>>>> Thanks. It makes sense...
>>          >>>>>>
>>          >>>>>> I used a KerberosToken, but my program got stuck
when
>>              running the
>>          >>>>>> following:
>>          >>>>>> new ZooKeeperInstance(instance,
>>         zookeepers).getConnector(user,
>>          >>>>>> krbToken)
>>          >>>>>>
>>          >>>>>> It looks like my client is stuck here:
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>          >>>>>> failing in the receive part of
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
>>          >>>>>>
>>          >>>>>> On my tservers, I see the following:
>>          >>>>>>
>>          >>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
>>         ERROR: Error
>>          >>>>>> occurred during processing of message.
>>          >>>>>> java.lang.RuntimeException:
>>          >>>>>> org.apache.thrift.transport.TTransportException:
>>          >>>>>> java.net <http://java.net>.SocketTimeoutException:
Read
>>
>>         timed out
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>          >>>>>>            at
>>         java.security.AccessController.doPrivileged(Native
>>          >>>>>> Method)
>>          >>>>>>            at
>>         javax.security.auth.Subject.doAs(Subject.java:356)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>          >>>>>>            at java.lang.Thread.run(Thread.java:745)
>>          >>>>>> Caused by:
>> org.apache.thrift.transport.TTransportException:
>>          >>>>>> java.net <http://java.net>.SocketTimeoutException:
Read
>>         timed out
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>          >>>>>>            at
>>          >>>>>>
>>
>>         org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>          >>>>>>            ... 11 more
>>          >>>>>> Caused by: java.net
>>         <http://java.net>.SocketTimeoutException: Read timed out
>>          >>>>>>            at
>> java.net.SocketInputStream.socketRead0(Native
>>              Method)
>>          >>>>>>            at
>>          >>>>>>
>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>          >>>>>>            at
>>          >>>>>>
>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>          >>>>>>            at
>>          >>>>>>
>>         java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>          >>>>>>            at
>>          >>>>>>
>>         java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>          >>>>>>            at
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>          >>>>>>            ... 17 more
>>          >>>>>>
>>          >>>>>> Any ideas why?
>>          >>>>>>
>>          >>>>>> Thanks.
>>          >>>>>> -Simon
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>          >>>>>>
>>          >>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh
>>              Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>
>>         <mailto:josh.elser@gmail.com <mailto:josh.elser@gmail.com>>>
>>
>>
>>          >>>>>> wrote:
>>          >>>>>>>
>>          >>>>>>> Make sure you read the JavaDoc on DelegationToken:
>>          >>>>>>>
>>          >>>>>>> <snip>
>>          >>>>>>> Obtain a delegation token by calling {@link
>>          >>>>>>>
>>          >>>>>>>
>>          >>>>>>>
>>          >>>>>>>
>>
>>
>> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
>>          >>>>>>> </snip>
>>          >>>>>>>
>>          >>>>>>> You cannot create a usable DelegationToken
as the client
>>              itself.
>>          >>>>>>>
>>          >>>>>>> Anyways, DelegationTokens are only relevant
in cases
>> where
>>              the client
>>          >>>>>>> Kerberos credentials are unavailable. The most
common
>> case
>>              is running
>>          >>>>>>> MapReduce jobs. If you are just interacting
with Accumulo
>>              through the
>>          >>>>>>> Java
>>          >>>>>>> API, the KerberosToken is all you need to use.
>>          >>>>>>>
>>          >>>>>>> The user-manual likely just needs to be updated.
I
>>         believe the
>>          >>>>>>> DelegationTokenConfig was added after I wrote
the initial
>>          >>>>>>> documentation.
>>          >>>>>>>
>>          >>>>>>>
>>          >>>>>>> Xu (Simon) Chen wrote:
>>          >>>>>>>>
>>          >>>>>>>> Hi folks,
>>          >>>>>>>>
>>          >>>>>>>> The latest kerberos doc seems to indicate
that
>>              getDelegationToken
>>          >>>>>>>> can
>>          >>>>>>>> be
>>          >>>>>>>> called without any parameters:
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>
>> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
>>          >>>>>>>>
>>          >>>>>>>> Yet the source code indicates a DelegationTokenConfig
>>              object must be
>>          >>>>>>>> passed in:
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>
>> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
>>          >>>>>>>>
>>          >>>>>>>> Any ideas on how I should construct the
>>         DelegationTokenConfig
>>          >>>>>>>> object?
>>          >>>>>>>>
>>          >>>>>>>> For context, I've been trying to get geomesa
to work on
>> my
>>              accumulo
>>          >>>>>>>> 1.7
>>          >>>>>>>> with kerberos turned on. Right now, the
code is somewhat
>>              tied to
>>          >>>>>>>> password auth:
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>          >>>>>>>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>          >>>>>>>> My thought is that I should get a KerberosToken
first,
>> and
>>              then try
>>          >>>>>>>> generate a DelegationToken, which is passed
back for
>> later
>>          >>>>>>>> interactions
>>          >>>>>>>> between geomesa and accumulo.
>>          >>>>>>>>
>>          >>>>>>>> Thanks.
>>          >>>>>>>> -Simon
>>
>>
>>
>>

Mime
View raw message