accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Hughes <jn...@virginia.edu>
Subject Re: kerberos auth, getDelegationToken
Date Sun, 07 Jun 2015 17:20:06 GMT
Hi all,

For GeoMesa, stats writing is quite secondary and optional, so we can sort
that out as a follow-on to seeing GeoMesa work with Accumulo 1.7.

I haven't had a chance to read in details yet, so forgive me if this is
covered in the docs.  Does either Mock or MiniAccumulo provide enough hooks
to test out Kerberos integration effectively?  I suppose I'm really asking
what kind of testing environment a project like GeoMesa would need to use
to test out Accumulo 1.7.

Even though MockAccumulo has a number of limitations, it is rather
effective in unit tests which can be part of a quick  build.

Thanks,

Jim

On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <xchenum@gmail.com> wrote:

> Nope, I am running the example as what the readme file suggested:
>
> java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
> org.geomesa.QuickStart -instanceId somecloud -zookeepers
> "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
> -tableName sometable
>
> I'll raise this question with the geomesa folks, but you're right that
> I can ignore it for now...
>
> Thanks!
> -Simon
>
>
> On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <josh.elser@gmail.com> wrote:
> > Are you running it via `mvn exec:java` by chance or netbeans?
> >
> >
> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
> >
> > If that's just a background thread writing in Stats, it might just be a
> > factor of how you're invoking the program and you can ignore it. I don't
> > know enough about the inner-workings of GeoMesa to say one way or the
> other.
> >
> >
> > Xu (Simon) Chen wrote:
> >>
> >> Josh,
> >>
> >> Everything works well, except for one thing :-)
> >>
> >> I am running geomesa-quickstart program that ingest some data and then
> >> perform a simple query:
> >> https://github.com/geomesa/geomesa-quickstart
> >>
> >> For some reason, the following error is emitted consistently at the
> >> end of the execution, after outputting the correct result:
> >> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will retry
> >> java.lang.InterruptedException
> >>          at java.lang.Object.wait(Native Method)
> >>          at java.lang.Object.wait(Object.java:503)
> >>          at
> >> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
> >>          at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> >>          at
> >> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
> >>          at
> >> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
> >>          at
> >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
> >>          at
> >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
> >>          at
> >>
> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
> >>          at
> >>
> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
> >>          at
> >>
> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
> >>          at
> >>
> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
> >>          at
> >>
> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
> >>          at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> >>          at
> >>
> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
> >>          at
> >>
> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
> >>          at
> >>
> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
> >>          at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>          at
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> >>          at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >>          at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >>          at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>          at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>          at java.lang.Thread.run(Thread.java:745)
> >>
> >> This is more annoying than a real problem. I am new to both accumulo
> >> and geomesa, but I am curious what the problem might be.
> >>
> >> Thanks!
> >> -Simon
> >>
> >>
> >> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<josh.elser@gmail.com>
> wrote:
> >>>
> >>> Great! Glad to hear it. Please let us know how it works out!
> >>>
> >>>
> >>> Xu (Simon) Chen wrote:
> >>>>
> >>>> Josh,
> >>>>
> >>>> You're right again.. Thanks!
> >>>>
> >>>> My ansible play actually pushed client.conf to all the server config
> >>>> directories, but didn't do anything for the clients, and that's my
> >>>> problem. Now kerberos is working great for me.
> >>>>
> >>>> Thanks again!
> >>>> -Simon
> >>>>
> >>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh Elser<josh.elser@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Simon,
> >>>>>
> >>>>> Did you create a client configuration file (~/.accumulo/config or
> >>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure Accumulo
> clients
> >>>>> to
> >>>>> actually use SASL when you're trying to use Kerberos authentication.
> >>>>> Your
> >>>>> server is expecting that, but I would venture a guess that your
> client
> >>>>> isn't.
> >>>>>
> >>>>> See
> >>>>>
> >>>>>
> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
> >>>>>
> >>>>>
> >>>>> Xu (Simon) Chen wrote:
> >>>>>>
> >>>>>> Josh,
> >>>>>>
> >>>>>> Thanks. It makes sense...
> >>>>>>
> >>>>>> I used a KerberosToken, but my program got stuck when running
the
> >>>>>> following:
> >>>>>> new ZooKeeperInstance(instance, zookeepers).getConnector(user,
> >>>>>> krbToken)
> >>>>>>
> >>>>>> It looks like my client is stuck here:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
> >>>>>> failing in the receive part of
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
> >>>>>>
> >>>>>> On my tservers, I see the following:
> >>>>>>
> >>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer] ERROR: Error
> >>>>>> occurred during processing of message.
> >>>>>> java.lang.RuntimeException:
> >>>>>> org.apache.thrift.transport.TTransportException:
> >>>>>> java.net.SocketTimeoutException: Read timed out
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> >>>>>>            at java.security.AccessController.doPrivileged(Native
> >>>>>> Method)
> >>>>>>            at javax.security.auth.Subject.doAs(Subject.java:356)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> >>>>>>            at java.lang.Thread.run(Thread.java:745)
> >>>>>> Caused by: org.apache.thrift.transport.TTransportException:
> >>>>>> java.net.SocketTimeoutException: Read timed out
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> >>>>>>            at
> >>>>>> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> >>>>>>            at
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> >>>>>>            ... 11 more
> >>>>>> Caused by: java.net.SocketTimeoutException: Read timed out
> >>>>>>            at java.net.SocketInputStream.socketRead0(Native
Method)
> >>>>>>            at
> >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
> >>>>>>            at
> >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
> >>>>>>            at
> >>>>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> >>>>>>            at
> >>>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> >>>>>>            at
> >>>>>>
> >>>>>>
> >>>>>>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> >>>>>>            ... 17 more
> >>>>>>
> >>>>>> Any ideas why?
> >>>>>>
> >>>>>> Thanks.
> >>>>>> -Simon
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh Elser<josh.elser@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Make sure you read the JavaDoc on DelegationToken:
> >>>>>>>
> >>>>>>> <snip>
> >>>>>>> Obtain a delegation token by calling {@link
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
> >>>>>>> </snip>
> >>>>>>>
> >>>>>>> You cannot create a usable DelegationToken as the client
itself.
> >>>>>>>
> >>>>>>> Anyways, DelegationTokens are only relevant in cases where
the
> client
> >>>>>>> Kerberos credentials are unavailable. The most common case
is
> running
> >>>>>>> MapReduce jobs. If you are just interacting with Accumulo
through
> the
> >>>>>>> Java
> >>>>>>> API, the KerberosToken is all you need to use.
> >>>>>>>
> >>>>>>> The user-manual likely just needs to be updated. I believe
the
> >>>>>>> DelegationTokenConfig was added after I wrote the initial
> >>>>>>> documentation.
> >>>>>>>
> >>>>>>>
> >>>>>>> Xu (Simon) Chen wrote:
> >>>>>>>>
> >>>>>>>> Hi folks,
> >>>>>>>>
> >>>>>>>> The latest kerberos doc seems to indicate that getDelegationToken
> >>>>>>>> can
> >>>>>>>> be
> >>>>>>>> called without any parameters:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
> >>>>>>>>
> >>>>>>>> Yet the source code indicates a DelegationTokenConfig
object must
> be
> >>>>>>>> passed in:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
> >>>>>>>>
> >>>>>>>> Any ideas on how I should construct the DelegationTokenConfig
> >>>>>>>> object?
> >>>>>>>>
> >>>>>>>> For context, I've been trying to get geomesa to work
on my
> accumulo
> >>>>>>>> 1.7
> >>>>>>>> with kerberos turned on. Right now, the code is somewhat
tied to
> >>>>>>>> password auth:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> >>>>>>>> My thought is that I should get a KerberosToken first,
and then
> try
> >>>>>>>> generate a DelegationToken, which is passed back for
later
> >>>>>>>> interactions
> >>>>>>>> between geomesa and accumulo.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>> -Simon
>

Mime
View raw message