accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Deprecating Mock Accumulo (was Re: kerberos auth, getDelegationToken)
Date Tue, 09 Jun 2015 21:48:42 GMT
On Tue, Jun 9, 2015 at 5:29 PM, Josh Elser <josh.elser@gmail.com> wrote:

> On Jun 9, 2015 2:46 PM, "Keith Turner" <keith@deenlo.com> wrote:
> >
> > On Mon, Jun 8, 2015 at 11:42 PM, Ryan Leary <ryan@bbn.com> wrote:
> >
> > > Just an anecdote from someone who has been bitten by mock more than a
> > > couple times. I would try to deprecate it in 1.8 and remove in 2.0 if
> at
> > > all possible. People really shouldn't write tests against it.
> > >
> >
> > +1
> >
> > It should be deprecated ASAP inorder to clearly communicate its status as
> > unmaintained.   If its deprecated in 1.8, its eligible to be dropped in
> 2.0
> > but does not have to be dropped.    When its dropped can be a separate
> > decision.
> >
>
> Doesn't semver require a major version of deprecation? Still def deprecate
> immediately, remove eventually.
>

No.  Christopher had proposed that more strict requirement, but we
eventually just went w/ unmodified semver.  The following is from the
semver page.


  Before you completely remove the functionality in a new major release
there should be at least one minor release that contains the deprecation so
that users can smoothly transition to the new API.



>
> >
> > >
> > > > On Jun 8, 2015, at 10:10 PM, Sean Busbey <busbey@cloudera.com>
> wrote:
> > > >
> > > > Josh's comment below made me realize we still haven't formally
> deprecated
> > > > MockAccumulo.
> > > >
> > > > What do folks think about doing it soon-ish with an aim of removing
> it in
> > > > Accumulo 3.0? (that's version three, so that it can remain deprecated
> for
> > > > all of version 2).
> > > >
> > > >
> > > > -Sean
> > > >
> > > >> On Sun, Jun 7, 2015 at 12:37 PM, Josh Elser <josh.elser@gmail.com>
> > > wrote:
> > > >>
> > > >> MiniAccumulo, yes. MockAccumulo, no. In general, we've near
> completely
> > > >> moved away from MockAccumulo. I wouldn't be surprised if it gets
> > > deprecated
> > > >> and removed soon.
> > > >>
> > > >>
> > > >>
> > >
>
> https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java
> > > >>
> > > >> Apache Directory provides a MiniKdc that can be used easily w/
> > > >> MiniAccumulo. Many of the integration tests have already been
> altered to
> > > >> support running w/ or w/o kerberos.
> > > >>
> > > >> James Hughes wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> For GeoMesa, stats writing is quite secondary and optional, so
we
> can
> > > >>> sort that out as a follow-on to seeing GeoMesa work with Accumulo
> 1.7.
> > > >>>
> > > >>> I haven't had a chance to read in details yet, so forgive me if
> this is
> > > >>> covered in the docs.  Does either Mock or MiniAccumulo provide
> enough
> > > >>> hooks to test out Kerberos integration effectively?  I suppose
I'm
> > > >>> really asking what kind of testing environment a project like
> GeoMesa
> > > >>> would need to use to test out Accumulo 1.7.
> > > >>>
> > > >>> Even though MockAccumulo has a number of limitations, it is rather
> > > >>> effective in unit tests which can be part of a quick  build.
> > > >>>
> > > >>> Thanks,
> > > >>>
> > > >>> Jim
> > > >>>
> > > >>> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <
> xchenum@gmail.com
> > > >>> <mailto:xchenum@gmail.com>> wrote:
> > > >>>
> > > >>>    Nope, I am running the example as what the readme file
> suggested:
> > > >>>
> > > >>>    java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
> > > >>>    org.geomesa.QuickStart -instanceId somecloud -zookeepers
> > > >>>    "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
> > > >>>    -tableName sometable
> > > >>>
> > > >>>    I'll raise this question with the geomesa folks, but you're
> right
> > > that
> > > >>>    I can ignore it for now...
> > > >>>
> > > >>>    Thanks!
> > > >>>    -Simon
> > > >>>
> > > >>>
> > > >>>    On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <
> josh.elser@gmail.com
> > > >>>    <mailto:josh.elser@gmail.com>> wrote:
> > > >>>> Are you running it via `mvn exec:java` by chance or netbeans?
> > > >>>
> > > >>>
> > >
>
> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
> > > >>>>
> > > >>>> If that's just a background thread writing in Stats, it might
> > > >>>    just be a
> > > >>>> factor of how you're invoking the program and you can ignore
it.
> > > >>>    I don't
> > > >>>> know enough about the inner-workings of GeoMesa to say one
way or
> > > >>>    the other.
> > > >>>>
> > > >>>>
> > > >>>> Xu (Simon) Chen wrote:
> > > >>>>>
> > > >>>>> Josh,
> > > >>>>>
> > > >>>>> Everything works well, except for one thing :-)
> > > >>>>>
> > > >>>>> I am running geomesa-quickstart program that ingest some
data
> > > >>>    and then
> > > >>>>> perform a simple query:
> > > >>>>> https://github.com/geomesa/geomesa-quickstart
> > > >>>>>
> > > >>>>> For some reason, the following error is emitted consistently
at
> > > >>> the
> > > >>>>> end of the execution, after outputting the correct result:
> > > >>>>> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error,
will
> > > >>>    retry
> > > >>>>> java.lang.InterruptedException
> > > >>>>>         at java.lang.Object.wait(Native Method)
> > > >>>>>         at java.lang.Object.wait(Object.java:503)
> > > >>>>>         at
> > > >>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
> > > >>>>>         at
> > > >>>    org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
> > > >>>>>         at
> > > >>>
> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
> > > >>>>>         at
> > > >>>
> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
> > > >>>>>         at
> > > >>>>>
> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
> > > >>>>>         at
> > > >>>>>
> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
> > > >>>>>         at
> > > >>>    scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
> > > >>>>>         at
> > > >>>
> > > >>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > >>>>>         at
> > > >>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >>>>>         at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >>>>>         at java.lang.Thread.run(Thread.java:745)
> > > >>>>>
> > > >>>>> This is more annoying than a real problem. I am new to
both
> > > >>> accumulo
> > > >>>>> and geomesa, but I am curious what the problem might be.
> > > >>>>>
> > > >>>>> Thanks!
> > > >>>>> -Simon
> > > >>>>>
> > > >>>>>
> > > >>>>> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<josh.elser@gmail.com
> > > >>>    <mailto:josh.elser@gmail.com>>  wrote:
> > > >>>>>>
> > > >>>>>> Great! Glad to hear it. Please let us know how it
works out!
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Xu (Simon) Chen wrote:
> > > >>>>>>>
> > > >>>>>>> Josh,
> > > >>>>>>>
> > > >>>>>>> You're right again.. Thanks!
> > > >>>>>>>
> > > >>>>>>> My ansible play actually pushed client.conf to
all the server
> > > >>>    config
> > > >>>>>>> directories, but didn't do anything for the clients,
and that's
> > > >>> my
> > > >>>>>>> problem. Now kerberos is working great for me.
> > > >>>>>>>
> > > >>>>>>> Thanks again!
> > > >>>>>>> -Simon
> > > >>>>>>>
> > > >>>>>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh
> > > >>>    Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
> > > >>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> Simon,
> > > >>>>>>>>
> > > >>>>>>>> Did you create a client configuration file
(~/.accumulo/config
> > > >>> or
> > > >>>>>>>> $ACCUMULO_CONF_DIR/client.conf)? You need
to configure
> > > >>>    Accumulo clients
> > > >>>>>>>> to
> > > >>>>>>>> actually use SASL when you're trying to use
Kerberos
> > > >>>    authentication.
> > > >>>>>>>> Your
> > > >>>>>>>> server is expecting that, but I would venture
a guess that
> > > >>>    your client
> > > >>>>>>>> isn't.
> > > >>>>>>>>
> > > >>>>>>>> See
> > > >>>
> > > >>>
> > >
> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Xu (Simon) Chen wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> Josh,
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks. It makes sense...
> > > >>>>>>>>>
> > > >>>>>>>>> I used a KerberosToken, but my program
got stuck when
> > > >>>    running the
> > > >>>>>>>>> following:
> > > >>>>>>>>> new ZooKeeperInstance(instance,
> zookeepers).getConnector(user,
> > > >>>>>>>>> krbToken)
> > > >>>>>>>>>
> > > >>>>>>>>> It looks like my client is stuck here:
> > > >>>
> > > >>>
> > >
>
> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
> > > >>>>>>>>> failing in the receive part of
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
> > > >>>>>>>>>
> > > >>>>>>>>> On my tservers, I see the following:
> > > >>>>>>>>>
> > > >>>>>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
ERROR:
> > > >>> Error
> > > >>>>>>>>> occurred during processing of message.
> > > >>>>>>>>> java.lang.RuntimeException:
> > > >>>>>>>>> org.apache.thrift.transport.TTransportException:
> > > >>>>>>>>> java.net.SocketTimeoutException: Read
timed out
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
> > > >>>>>>>>>           at
> > > >>> java.security.AccessController.doPrivileged(Native
> > > >>>>>>>>> Method)
> > > >>>>>>>>>           at
> > > >>> javax.security.auth.Subject.doAs(Subject.java:356)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> > > >>>>>>>>>           at java.lang.Thread.run(Thread.java:745)
> > > >>>>>>>>> Caused by: org.apache.thrift.transport.TTransportException:
> > > >>>>>>>>> java.net.SocketTimeoutException: Read
timed out
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> > > >>>>>>>>>           at
> > > >>>
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
> > > >>>>>>>>>           ... 11 more
> > > >>>>>>>>> Caused by: java.net.SocketTimeoutException:
Read timed out
> > > >>>>>>>>>           at java.net.SocketInputStream.socketRead0(Native
> > > >>>    Method)
> > > >>>>>>>>>           at
> > > >>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
> > > >>>>>>>>>           at
> > > >>>>>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
> > > >>>>>>>>>           at
> > > >>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> > > >>>>>>>>>           at
> > > >>>>>>>>>
> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> > > >>>>>>>>>           at
> > > >>>
> > > >>>
> > >
>
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> > > >>>>>>>>>           ... 17 more
> > > >>>>>>>>>
> > > >>>>>>>>> Any ideas why?
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks.
> > > >>>>>>>>> -Simon
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh
> > > >>>    Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
> > > >>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Make sure you read the JavaDoc on
DelegationToken:
> > > >>>>>>>>>>
> > > >>>>>>>>>> <snip>
> > > >>>>>>>>>> Obtain a delegation token by calling
{@link
> > > >>>
> > > >>>
> > >
>
> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
> > > >>>>>>>>>> </snip>
> > > >>>>>>>>>>
> > > >>>>>>>>>> You cannot create a usable DelegationToken
as the client
> > > >>>    itself.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Anyways, DelegationTokens are only
relevant in cases where
> > > >>>    the client
> > > >>>>>>>>>> Kerberos credentials are unavailable.
The most common case
> > > >>>    is running
> > > >>>>>>>>>> MapReduce jobs. If you are just interacting
with Accumulo
> > > >>>    through the
> > > >>>>>>>>>> Java
> > > >>>>>>>>>> API, the KerberosToken is all you
need to use.
> > > >>>>>>>>>>
> > > >>>>>>>>>> The user-manual likely just needs
to be updated. I believe
> > > >>> the
> > > >>>>>>>>>> DelegationTokenConfig was added after
I wrote the initial
> > > >>>>>>>>>> documentation.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Xu (Simon) Chen wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Hi folks,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The latest kerberos doc seems
to indicate that
> > > >>>    getDelegationToken
> > > >>>>>>>>>>> can
> > > >>>>>>>>>>> be
> > > >>>>>>>>>>> called without any parameters:
> > > >>>
> > > >>>
> > >
>
> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Yet the source code indicates
a DelegationTokenConfig
> > > >>>    object must be
> > > >>>>>>>>>>> passed in:
> > > >>>
> > > >>>
> > >
>
> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Any ideas on how I should construct
the
> > > >>> DelegationTokenConfig
> > > >>>>>>>>>>> object?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> For context, I've been trying
to get geomesa to work on my
> > > >>>    accumulo
> > > >>>>>>>>>>> 1.7
> > > >>>>>>>>>>> with kerberos turned on. Right
now, the code is somewhat
> > > >>>    tied to
> > > >>>>>>>>>>> password auth:
> > > >>>
> > > >>>
> > >
>
> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
> > > >>>>>>>>>>> My thought is that I should get
a KerberosToken first, and
> > > >>>    then try
> > > >>>>>>>>>>> generate a DelegationToken, which
is passed back for later
> > > >>>>>>>>>>> interactions
> > > >>>>>>>>>>> between geomesa and accumulo.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks.
> > > >>>>>>>>>>> -Simon
> > > >
> > > >
> > > > --
> > > > Sean
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message