accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Busbey <bus...@cloudera.com>
Subject Deprecating Mock Accumulo (was Re: kerberos auth, getDelegationToken)
Date Tue, 09 Jun 2015 02:10:17 GMT
Josh's comment below made me realize we still haven't formally deprecated
MockAccumulo.

What do folks think about doing it soon-ish with an aim of removing it in
Accumulo 3.0? (that's version three, so that it can remain deprecated for
all of version 2).


-Sean

On Sun, Jun 7, 2015 at 12:37 PM, Josh Elser <josh.elser@gmail.com> wrote:

> MiniAccumulo, yes. MockAccumulo, no. In general, we've near completely
> moved away from MockAccumulo. I wouldn't be surprised if it gets deprecated
> and removed soon.
>
>
> https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java
>
> Apache Directory provides a MiniKdc that can be used easily w/
> MiniAccumulo. Many of the integration tests have already been altered to
> support running w/ or w/o kerberos.
>
> James Hughes wrote:
>
>> Hi all,
>>
>> For GeoMesa, stats writing is quite secondary and optional, so we can
>> sort that out as a follow-on to seeing GeoMesa work with Accumulo 1.7.
>>
>> I haven't had a chance to read in details yet, so forgive me if this is
>> covered in the docs.  Does either Mock or MiniAccumulo provide enough
>> hooks to test out Kerberos integration effectively?  I suppose I'm
>> really asking what kind of testing environment a project like GeoMesa
>> would need to use to test out Accumulo 1.7.
>>
>> Even though MockAccumulo has a number of limitations, it is rather
>> effective in unit tests which can be part of a quick  build.
>>
>> Thanks,
>>
>> Jim
>>
>> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <xchenum@gmail.com
>> <mailto:xchenum@gmail.com>> wrote:
>>
>>     Nope, I am running the example as what the readme file suggested:
>>
>>     java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
>>     org.geomesa.QuickStart -instanceId somecloud -zookeepers
>>     "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
>>     -tableName sometable
>>
>>     I'll raise this question with the geomesa folks, but you're right that
>>     I can ignore it for now...
>>
>>     Thanks!
>>     -Simon
>>
>>
>>     On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <josh.elser@gmail.com
>>     <mailto:josh.elser@gmail.com>> wrote:
>>      > Are you running it via `mvn exec:java` by chance or netbeans?
>>      >
>>      >
>>
>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
>>      >
>>      > If that's just a background thread writing in Stats, it might
>>     just be a
>>      > factor of how you're invoking the program and you can ignore it.
>>     I don't
>>      > know enough about the inner-workings of GeoMesa to say one way or
>>     the other.
>>      >
>>      >
>>      > Xu (Simon) Chen wrote:
>>      >>
>>      >> Josh,
>>      >>
>>      >> Everything works well, except for one thing :-)
>>      >>
>>      >> I am running geomesa-quickstart program that ingest some data
>>     and then
>>      >> perform a simple query:
>>      >> https://github.com/geomesa/geomesa-quickstart
>>      >>
>>      >> For some reason, the following error is emitted consistently at
>> the
>>      >> end of the execution, after outputting the correct result:
>>      >> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will
>>     retry
>>      >> java.lang.InterruptedException
>>      >>          at java.lang.Object.wait(Native Method)
>>      >>          at java.lang.Object.wait(Object.java:503)
>>      >>          at
>>      >>
>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
>>      >>          at
>>     org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
>>      >>          at
>>      >>
>> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
>>      >>          at
>>      >>
>> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
>>      >>          at
>>      >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
>>      >>          at
>>      >> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
>>      >>          at
>>      >>
>>
>> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
>>      >>          at
>>      >>
>>
>> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
>>      >>          at
>>      >>
>>
>> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
>>      >>          at
>>      >>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
>>      >>          at
>>      >>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
>>      >>          at
>>     scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>>      >>          at
>>      >>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
>>      >>          at
>>      >>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
>>      >>          at
>>      >>
>>
>> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
>>      >>          at
>>      >>
>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>      >>          at
>>      >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>      >>          at
>>      >>
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>      >>          at
>>      >>
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>      >>          at
>>      >>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>      >>          at
>>      >>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>      >>          at java.lang.Thread.run(Thread.java:745)
>>      >>
>>      >> This is more annoying than a real problem. I am new to both
>> accumulo
>>      >> and geomesa, but I am curious what the problem might be.
>>      >>
>>      >> Thanks!
>>      >> -Simon
>>      >>
>>      >>
>>      >> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<josh.elser@gmail.com
>>     <mailto:josh.elser@gmail.com>>  wrote:
>>      >>>
>>      >>> Great! Glad to hear it. Please let us know how it works out!
>>      >>>
>>      >>>
>>      >>> Xu (Simon) Chen wrote:
>>      >>>>
>>      >>>> Josh,
>>      >>>>
>>      >>>> You're right again.. Thanks!
>>      >>>>
>>      >>>> My ansible play actually pushed client.conf to all the server
>>     config
>>      >>>> directories, but didn't do anything for the clients, and that's
>> my
>>      >>>> problem. Now kerberos is working great for me.
>>      >>>>
>>      >>>> Thanks again!
>>      >>>> -Simon
>>      >>>>
>>      >>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh
>>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>>
>>      >>>> wrote:
>>      >>>>>
>>      >>>>> Simon,
>>      >>>>>
>>      >>>>> Did you create a client configuration file (~/.accumulo/config
>> or
>>      >>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure
>>     Accumulo clients
>>      >>>>> to
>>      >>>>> actually use SASL when you're trying to use Kerberos
>>     authentication.
>>      >>>>> Your
>>      >>>>> server is expecting that, but I would venture a guess that
>>     your client
>>      >>>>> isn't.
>>      >>>>>
>>      >>>>> See
>>      >>>>>
>>      >>>>>
>>
>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
>>      >>>>>
>>      >>>>>
>>      >>>>> Xu (Simon) Chen wrote:
>>      >>>>>>
>>      >>>>>> Josh,
>>      >>>>>>
>>      >>>>>> Thanks. It makes sense...
>>      >>>>>>
>>      >>>>>> I used a KerberosToken, but my program got stuck when
>>     running the
>>      >>>>>> following:
>>      >>>>>> new ZooKeeperInstance(instance, zookeepers).getConnector(user,
>>      >>>>>> krbToken)
>>      >>>>>>
>>      >>>>>> It looks like my client is stuck here:
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>      >>>>>> failing in the receive part of
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
>>      >>>>>>
>>      >>>>>> On my tservers, I see the following:
>>      >>>>>>
>>      >>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
ERROR:
>> Error
>>      >>>>>> occurred during processing of message.
>>      >>>>>> java.lang.RuntimeException:
>>      >>>>>> org.apache.thrift.transport.TTransportException:
>>      >>>>>> java.net.SocketTimeoutException: Read timed out
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>      >>>>>>            at
>> java.security.AccessController.doPrivileged(Native
>>      >>>>>> Method)
>>      >>>>>>            at
>> javax.security.auth.Subject.doAs(Subject.java:356)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>      >>>>>>            at java.lang.Thread.run(Thread.java:745)
>>      >>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>      >>>>>> java.net.SocketTimeoutException: Read timed out
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>      >>>>>>            at
>>      >>>>>>
>>     org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>      >>>>>>            ... 11 more
>>      >>>>>> Caused by: java.net.SocketTimeoutException: Read timed
out
>>      >>>>>>            at java.net.SocketInputStream.socketRead0(Native
>>     Method)
>>      >>>>>>            at
>>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>      >>>>>>            at
>>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>      >>>>>>            at
>>      >>>>>>
>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>      >>>>>>            at
>>      >>>>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>      >>>>>>            at
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>
>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>      >>>>>>            ... 17 more
>>      >>>>>>
>>      >>>>>> Any ideas why?
>>      >>>>>>
>>      >>>>>> Thanks.
>>      >>>>>> -Simon
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>      >>>>>>
>>      >>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh
>>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>>
>>      >>>>>> wrote:
>>      >>>>>>>
>>      >>>>>>> Make sure you read the JavaDoc on DelegationToken:
>>      >>>>>>>
>>      >>>>>>> <snip>
>>      >>>>>>> Obtain a delegation token by calling {@link
>>      >>>>>>>
>>      >>>>>>>
>>      >>>>>>>
>>      >>>>>>>
>>
>> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
>>      >>>>>>> </snip>
>>      >>>>>>>
>>      >>>>>>> You cannot create a usable DelegationToken as the
client
>>     itself.
>>      >>>>>>>
>>      >>>>>>> Anyways, DelegationTokens are only relevant in
cases where
>>     the client
>>      >>>>>>> Kerberos credentials are unavailable. The most
common case
>>     is running
>>      >>>>>>> MapReduce jobs. If you are just interacting with
Accumulo
>>     through the
>>      >>>>>>> Java
>>      >>>>>>> API, the KerberosToken is all you need to use.
>>      >>>>>>>
>>      >>>>>>> The user-manual likely just needs to be updated.
I believe
>> the
>>      >>>>>>> DelegationTokenConfig was added after I wrote the
initial
>>      >>>>>>> documentation.
>>      >>>>>>>
>>      >>>>>>>
>>      >>>>>>> Xu (Simon) Chen wrote:
>>      >>>>>>>>
>>      >>>>>>>> Hi folks,
>>      >>>>>>>>
>>      >>>>>>>> The latest kerberos doc seems to indicate that
>>     getDelegationToken
>>      >>>>>>>> can
>>      >>>>>>>> be
>>      >>>>>>>> called without any parameters:
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>
>> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
>>      >>>>>>>>
>>      >>>>>>>> Yet the source code indicates a DelegationTokenConfig
>>     object must be
>>      >>>>>>>> passed in:
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>
>> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
>>      >>>>>>>>
>>      >>>>>>>> Any ideas on how I should construct the
>> DelegationTokenConfig
>>      >>>>>>>> object?
>>      >>>>>>>>
>>      >>>>>>>> For context, I've been trying to get geomesa
to work on my
>>     accumulo
>>      >>>>>>>> 1.7
>>      >>>>>>>> with kerberos turned on. Right now, the code
is somewhat
>>     tied to
>>      >>>>>>>> password auth:
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>      >>>>>>>>
>>
>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>      >>>>>>>> My thought is that I should get a KerberosToken
first, and
>>     then try
>>      >>>>>>>> generate a DelegationToken, which is passed
back for later
>>      >>>>>>>> interactions
>>      >>>>>>>> between geomesa and accumulo.
>>      >>>>>>>>
>>      >>>>>>>> Thanks.
>>      >>>>>>>> -Simon
>>
>>
>>


-- 
Sean

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message