accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: kerberos auth, getDelegationToken
Date Sun, 07 Jun 2015 17:37:30 GMT
MiniAccumulo, yes. MockAccumulo, no. In general, we've near completely 
moved away from MockAccumulo. I wouldn't be surprised if it gets 
deprecated and removed soon.

https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java

Apache Directory provides a MiniKdc that can be used easily w/ 
MiniAccumulo. Many of the integration tests have already been altered to 
support running w/ or w/o kerberos.

James Hughes wrote:
> Hi all,
>
> For GeoMesa, stats writing is quite secondary and optional, so we can
> sort that out as a follow-on to seeing GeoMesa work with Accumulo 1.7.
>
> I haven't had a chance to read in details yet, so forgive me if this is
> covered in the docs.  Does either Mock or MiniAccumulo provide enough
> hooks to test out Kerberos integration effectively?  I suppose I'm
> really asking what kind of testing environment a project like GeoMesa
> would need to use to test out Accumulo 1.7.
>
> Even though MockAccumulo has a number of limitations, it is rather
> effective in unit tests which can be part of a quick  build.
>
> Thanks,
>
> Jim
>
> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <xchenum@gmail.com
> <mailto:xchenum@gmail.com>> wrote:
>
>     Nope, I am running the example as what the readme file suggested:
>
>     java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
>     org.geomesa.QuickStart -instanceId somecloud -zookeepers
>     "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
>     -tableName sometable
>
>     I'll raise this question with the geomesa folks, but you're right that
>     I can ignore it for now...
>
>     Thanks!
>     -Simon
>
>
>     On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <josh.elser@gmail.com
>     <mailto:josh.elser@gmail.com>> wrote:
>      > Are you running it via `mvn exec:java` by chance or netbeans?
>      >
>      >
>     http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
>      >
>      > If that's just a background thread writing in Stats, it might
>     just be a
>      > factor of how you're invoking the program and you can ignore it.
>     I don't
>      > know enough about the inner-workings of GeoMesa to say one way or
>     the other.
>      >
>      >
>      > Xu (Simon) Chen wrote:
>      >>
>      >> Josh,
>      >>
>      >> Everything works well, except for one thing :-)
>      >>
>      >> I am running geomesa-quickstart program that ingest some data
>     and then
>      >> perform a simple query:
>      >> https://github.com/geomesa/geomesa-quickstart
>      >>
>      >> For some reason, the following error is emitted consistently at the
>      >> end of the execution, after outputting the correct result:
>      >> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will
>     retry
>      >> java.lang.InterruptedException
>      >>          at java.lang.Object.wait(Native Method)
>      >>          at java.lang.Object.wait(Object.java:503)
>      >>          at
>      >> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
>      >>          at
>     org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
>      >>          at
>      >> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
>      >>          at
>      >> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
>      >>          at
>      >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
>      >>          at
>      >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
>      >>          at
>      >>
>     org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
>      >>          at
>      >>
>     org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
>      >>          at
>      >>
>     org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
>      >>          at
>      >>
>     org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
>      >>          at
>      >>
>     org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
>      >>          at
>     scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>      >>          at
>      >>
>     org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
>      >>          at
>      >>
>     org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
>      >>          at
>      >>
>     org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
>      >>          at
>      >>
>     java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>      >>          at
>      >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>      >>          at
>      >>
>     java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>      >>          at
>      >>
>     java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>      >>          at
>      >>
>     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>      >>          at
>      >>
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>      >>          at java.lang.Thread.run(Thread.java:745)
>      >>
>      >> This is more annoying than a real problem. I am new to both accumulo
>      >> and geomesa, but I am curious what the problem might be.
>      >>
>      >> Thanks!
>      >> -Simon
>      >>
>      >>
>      >> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<josh.elser@gmail.com
>     <mailto:josh.elser@gmail.com>>  wrote:
>      >>>
>      >>> Great! Glad to hear it. Please let us know how it works out!
>      >>>
>      >>>
>      >>> Xu (Simon) Chen wrote:
>      >>>>
>      >>>> Josh,
>      >>>>
>      >>>> You're right again.. Thanks!
>      >>>>
>      >>>> My ansible play actually pushed client.conf to all the server
>     config
>      >>>> directories, but didn't do anything for the clients, and that's
my
>      >>>> problem. Now kerberos is working great for me.
>      >>>>
>      >>>> Thanks again!
>      >>>> -Simon
>      >>>>
>      >>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh
>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>      >>>> wrote:
>      >>>>>
>      >>>>> Simon,
>      >>>>>
>      >>>>> Did you create a client configuration file (~/.accumulo/config
or
>      >>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure
>     Accumulo clients
>      >>>>> to
>      >>>>> actually use SASL when you're trying to use Kerberos
>     authentication.
>      >>>>> Your
>      >>>>> server is expecting that, but I would venture a guess that
>     your client
>      >>>>> isn't.
>      >>>>>
>      >>>>> See
>      >>>>>
>      >>>>>
>     http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
>      >>>>>
>      >>>>>
>      >>>>> Xu (Simon) Chen wrote:
>      >>>>>>
>      >>>>>> Josh,
>      >>>>>>
>      >>>>>> Thanks. It makes sense...
>      >>>>>>
>      >>>>>> I used a KerberosToken, but my program got stuck when
>     running the
>      >>>>>> following:
>      >>>>>> new ZooKeeperInstance(instance, zookeepers).getConnector(user,
>      >>>>>> krbToken)
>      >>>>>>
>      >>>>>> It looks like my client is stuck here:
>      >>>>>>
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>      >>>>>> failing in the receive part of
>      >>>>>>
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
>      >>>>>>
>      >>>>>> On my tservers, I see the following:
>      >>>>>>
>      >>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR:
Error
>      >>>>>> occurred during processing of message.
>      >>>>>> java.lang.RuntimeException:
>      >>>>>> org.apache.thrift.transport.TTransportException:
>      >>>>>> java.net.SocketTimeoutException: Read timed out
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>      >>>>>>            at java.security.AccessController.doPrivileged(Native
>      >>>>>> Method)
>      >>>>>>            at javax.security.auth.Subject.doAs(Subject.java:356)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>      >>>>>>            at java.lang.Thread.run(Thread.java:745)
>      >>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>      >>>>>> java.net.SocketTimeoutException: Read timed out
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>      >>>>>>            at
>      >>>>>>
>     org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>      >>>>>>            ... 11 more
>      >>>>>> Caused by: java.net.SocketTimeoutException: Read timed
out
>      >>>>>>            at java.net.SocketInputStream.socketRead0(Native
>     Method)
>      >>>>>>            at
>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>      >>>>>>            at
>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>      >>>>>>            at
>      >>>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>      >>>>>>            at
>      >>>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>      >>>>>>            at
>      >>>>>>
>      >>>>>>
>      >>>>>>
>     org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>      >>>>>>            ... 17 more
>      >>>>>>
>      >>>>>> Any ideas why?
>      >>>>>>
>      >>>>>> Thanks.
>      >>>>>> -Simon
>      >>>>>>
>      >>>>>>
>      >>>>>>
>      >>>>>>
>      >>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh
>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>      >>>>>> wrote:
>      >>>>>>>
>      >>>>>>> Make sure you read the JavaDoc on DelegationToken:
>      >>>>>>>
>      >>>>>>> <snip>
>      >>>>>>> Obtain a delegation token by calling {@link
>      >>>>>>>
>      >>>>>>>
>      >>>>>>>
>      >>>>>>>
>     SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
>      >>>>>>> </snip>
>      >>>>>>>
>      >>>>>>> You cannot create a usable DelegationToken as the client
>     itself.
>      >>>>>>>
>      >>>>>>> Anyways, DelegationTokens are only relevant in cases
where
>     the client
>      >>>>>>> Kerberos credentials are unavailable. The most common
case
>     is running
>      >>>>>>> MapReduce jobs. If you are just interacting with Accumulo
>     through the
>      >>>>>>> Java
>      >>>>>>> API, the KerberosToken is all you need to use.
>      >>>>>>>
>      >>>>>>> The user-manual likely just needs to be updated. I
believe the
>      >>>>>>> DelegationTokenConfig was added after I wrote the initial
>      >>>>>>> documentation.
>      >>>>>>>
>      >>>>>>>
>      >>>>>>> Xu (Simon) Chen wrote:
>      >>>>>>>>
>      >>>>>>>> Hi folks,
>      >>>>>>>>
>      >>>>>>>> The latest kerberos doc seems to indicate that
>     getDelegationToken
>      >>>>>>>> can
>      >>>>>>>> be
>      >>>>>>>> called without any parameters:
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>     https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
>      >>>>>>>>
>      >>>>>>>> Yet the source code indicates a DelegationTokenConfig
>     object must be
>      >>>>>>>> passed in:
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>     https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
>      >>>>>>>>
>      >>>>>>>> Any ideas on how I should construct the DelegationTokenConfig
>      >>>>>>>> object?
>      >>>>>>>>
>      >>>>>>>> For context, I've been trying to get geomesa to
work on my
>     accumulo
>      >>>>>>>> 1.7
>      >>>>>>>> with kerberos turned on. Right now, the code is
somewhat
>     tied to
>      >>>>>>>> password auth:
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>      >>>>>>>>
>     https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>      >>>>>>>> My thought is that I should get a KerberosToken
first, and
>     then try
>      >>>>>>>> generate a DelegationToken, which is passed back
for later
>      >>>>>>>> interactions
>      >>>>>>>> between geomesa and accumulo.
>      >>>>>>>>
>      >>>>>>>> Thanks.
>      >>>>>>>> -Simon
>
>

Mime
View raw message