accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xu (Simon) Chen" <xche...@gmail.com>
Subject Re: kerberos auth, getDelegationToken
Date Sun, 07 Jun 2015 22:20:13 GMT
I ran into another problem when trying geomesa+spark:
http://www.geomesa.org/spark/

For some reason, running the geomesa+spark example resulted in the
following error:
scala> queryRdd.count
java.lang.NullPointerException
        at org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.unwrapAuthenticationToken(ConfiguratorBase.java:493)
        at org.apache.accumulo.core.client.mapreduce.AbstractInputFormat.validateOptions(AbstractInputFormat.java:390)
        at org.apache.accumulo.core.client.mapreduce.AbstractInputFormat.getSplits(AbstractInputFormat.java:668)
        at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:95)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1512)
        at org.apache.spark.rdd.RDD.count(RDD.scala:1006)
…

hadoopToken is probably null here:
https://github.com/apache/accumulo/blob/branch-1.7.0/core/src/main/java/org/apache/accumulo/core/client/mapreduce/lib/impl/ConfiguratorBase.java#L493

I suspect it's related to geomesa's way of creating RDD. The following
doesn't seem to be sufficient:
https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-compute/src/main/scala/org/locationtech/geomesa/compute/spark/GeoMesaSpark.scala#L69

Because, accumulo is adding hadoop token here in hadoop.mapreduce.Job,
in addition to the Configuration:
https://github.com/apache/accumulo/blob/branch-1.7.0/core/src/main/java/org/apache/accumulo/core/client/mapreduce/AbstractInputFormat.java#L130

Then in spark, NewAPIHadoopRDD of Spark is only taking in a
hadoop.conf.Configuration object, rather than a job:
https://github.com/apache/spark/blob/v1.3.1/core/src/main/scala/org/apache/spark/SparkContext.scala#L870


This seems to be a more general issue between accumulo and spark. So,
has anyone tried using spark to access data from a kerberized accumulo
cluster?

Thanks.
-Simon

Thanks.
-Simon



On Sun, Jun 7, 2015 at 3:54 PM, James Hughes <jnh5y@virginia.edu> wrote:
> Josh,
>
> Thanks.  That's more or less what I expected.
>
> As we work to transition from Mock to MiniAccumulo, I'd want to change from
> spinning up lots of MockInstances to one MiniAccumulo.  To understand that
> pattern, do I basically just need to read through test sub-module and the
> test/pom.xml?  Are there any other resources I should be checking out?
>
> Cheers,
>
> Jim
>
> On Sun, Jun 7, 2015 at 1:37 PM, Josh Elser <josh.elser@gmail.com> wrote:
>>
>> MiniAccumulo, yes. MockAccumulo, no. In general, we've near completely
>> moved away from MockAccumulo. I wouldn't be surprised if it gets deprecated
>> and removed soon.
>>
>>
>> https://github.com/apache/accumulo/blob/1.7/test/src/test/java/org/apache/accumulo/test/functional/KerberosIT.java
>>
>> Apache Directory provides a MiniKdc that can be used easily w/
>> MiniAccumulo. Many of the integration tests have already been altered to
>> support running w/ or w/o kerberos.
>>
>> James Hughes wrote:
>>>
>>> Hi all,
>>>
>>> For GeoMesa, stats writing is quite secondary and optional, so we can
>>> sort that out as a follow-on to seeing GeoMesa work with Accumulo 1.7.
>>>
>>> I haven't had a chance to read in details yet, so forgive me if this is
>>> covered in the docs.  Does either Mock or MiniAccumulo provide enough
>>> hooks to test out Kerberos integration effectively?  I suppose I'm
>>> really asking what kind of testing environment a project like GeoMesa
>>> would need to use to test out Accumulo 1.7.
>>>
>>> Even though MockAccumulo has a number of limitations, it is rather
>>> effective in unit tests which can be part of a quick  build.
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>> On Sat, Jun 6, 2015 at 11:14 PM, Xu (Simon) Chen <xchenum@gmail.com
>>> <mailto:xchenum@gmail.com>> wrote:
>>>
>>>     Nope, I am running the example as what the readme file suggested:
>>>
>>>     java -cp ./target/geomesa-quickstart-1.0-SNAPSHOT.jar
>>>     org.geomesa.QuickStart -instanceId somecloud -zookeepers
>>>     "zoo1:2181,zoo2:2181,zoo3:2181" -user someuser -password somepwd
>>>     -tableName sometable
>>>
>>>     I'll raise this question with the geomesa folks, but you're right
>>> that
>>>     I can ignore it for now...
>>>
>>>     Thanks!
>>>     -Simon
>>>
>>>
>>>     On Sat, Jun 6, 2015 at 11:00 PM, Josh Elser <josh.elser@gmail.com
>>>     <mailto:josh.elser@gmail.com>> wrote:
>>>      > Are you running it via `mvn exec:java` by chance or netbeans?
>>>      >
>>>      >
>>>
>>> http://mail-archives.apache.org/mod_mbox/accumulo-user/201411.mbox/%3C547A9071.1020704@gmail.com%3E
>>>      >
>>>      > If that's just a background thread writing in Stats, it might
>>>     just be a
>>>      > factor of how you're invoking the program and you can ignore it.
>>>     I don't
>>>      > know enough about the inner-workings of GeoMesa to say one way or
>>>     the other.
>>>      >
>>>      >
>>>      > Xu (Simon) Chen wrote:
>>>      >>
>>>      >> Josh,
>>>      >>
>>>      >> Everything works well, except for one thing :-)
>>>      >>
>>>      >> I am running geomesa-quickstart program that ingest some data
>>>     and then
>>>      >> perform a simple query:
>>>      >> https://github.com/geomesa/geomesa-quickstart
>>>      >>
>>>      >> For some reason, the following error is emitted consistently at
>>> the
>>>      >> end of the execution, after outputting the correct result:
>>>      >> 15/06/07 00:29:22 INFO zookeeper.ZooCache: Zookeeper error, will
>>>     retry
>>>      >> java.lang.InterruptedException
>>>      >>          at java.lang.Object.wait(Native Method)
>>>      >>          at java.lang.Object.wait(Object.java:503)
>>>      >>          at
>>>      >>
>>> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1342)
>>>      >>          at
>>>     org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1036)
>>>      >>          at
>>>      >>
>>> org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
>>>      >>          at
>>>      >>
>>> org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
>>>      >>          at
>>>      >>
>>> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
>>>      >>          at
>>>      >>
>>> org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
>>>      >>          at
>>>      >>
>>>
>>> org.apache.accumulo.core.client.impl.Tables.getTableState(Tables.java:180)
>>>      >>          at
>>>      >>
>>>
>>> org.apache.accumulo.core.client.impl.ConnectorImpl.getTableId(ConnectorImpl.java:82)
>>>      >>          at
>>>      >>
>>>
>>> org.apache.accumulo.core.client.impl.ConnectorImpl.createBatchWriter(ConnectorImpl.java:128)
>>>      >>          at
>>>      >>
>>>
>>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:174)
>>>      >>          at
>>>      >>
>>>
>>> org.locationtech.geomesa.core.stats.StatWriter$$anonfun$write$2.apply(StatWriter.scala:156)
>>>      >>          at
>>>     scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
>>>      >>          at
>>>      >>
>>>
>>> org.locationtech.geomesa.core.stats.StatWriter$.write(StatWriter.scala:156)
>>>      >>          at
>>>      >>
>>>
>>> org.locationtech.geomesa.core.stats.StatWriter$.drainQueue(StatWriter.scala:148)
>>>      >>          at
>>>      >>
>>>
>>> org.locationtech.geomesa.core.stats.StatWriter$.run(StatWriter.scala:116)
>>>      >>          at
>>>      >>
>>>
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>      >>          at
>>>      >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>      >>          at
>>>      >>
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>      >>          at
>>>      >>
>>>
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>      >>          at
>>>      >>
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>      >>          at
>>>      >>
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>      >>          at java.lang.Thread.run(Thread.java:745)
>>>      >>
>>>      >> This is more annoying than a real problem. I am new to both
>>> accumulo
>>>      >> and geomesa, but I am curious what the problem might be.
>>>      >>
>>>      >> Thanks!
>>>      >> -Simon
>>>      >>
>>>      >>
>>>      >> On Sat, Jun 6, 2015 at 8:01 PM, Josh Elser<josh.elser@gmail.com
>>>     <mailto:josh.elser@gmail.com>>  wrote:
>>>      >>>
>>>      >>> Great! Glad to hear it. Please let us know how it works out!
>>>      >>>
>>>      >>>
>>>      >>> Xu (Simon) Chen wrote:
>>>      >>>>
>>>      >>>> Josh,
>>>      >>>>
>>>      >>>> You're right again.. Thanks!
>>>      >>>>
>>>      >>>> My ansible play actually pushed client.conf to all the
server
>>>     config
>>>      >>>> directories, but didn't do anything for the clients, and
that's
>>> my
>>>      >>>> problem. Now kerberos is working great for me.
>>>      >>>>
>>>      >>>> Thanks again!
>>>      >>>> -Simon
>>>      >>>>
>>>      >>>> On Sat, Jun 6, 2015 at 5:04 PM, Josh
>>>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>>>
>>>      >>>> wrote:
>>>      >>>>>
>>>      >>>>> Simon,
>>>      >>>>>
>>>      >>>>> Did you create a client configuration file (~/.accumulo/config
>>> or
>>>      >>>>> $ACCUMULO_CONF_DIR/client.conf)? You need to configure
>>>     Accumulo clients
>>>      >>>>> to
>>>      >>>>> actually use SASL when you're trying to use Kerberos
>>>     authentication.
>>>      >>>>> Your
>>>      >>>>> server is expecting that, but I would venture a guess
that
>>>     your client
>>>      >>>>> isn't.
>>>      >>>>>
>>>      >>>>> See
>>>      >>>>>
>>>      >>>>>
>>>
>>> http://accumulo.apache.org/1.7/accumulo_user_manual.html#_configuration_3
>>>      >>>>>
>>>      >>>>>
>>>      >>>>> Xu (Simon) Chen wrote:
>>>      >>>>>>
>>>      >>>>>> Josh,
>>>      >>>>>>
>>>      >>>>>> Thanks. It makes sense...
>>>      >>>>>>
>>>      >>>>>> I used a KerberosToken, but my program got stuck
when
>>>     running the
>>>      >>>>>> following:
>>>      >>>>>> new ZooKeeperInstance(instance,
>>> zookeepers).getConnector(user,
>>>      >>>>>> krbToken)
>>>      >>>>>>
>>>      >>>>>> It looks like my client is stuck here:
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> https://github.com/apache/accumulo/blob/master/core/src/main/java/org/apache/accumulo/core/client/impl/ConnectorImpl.java#L70
>>>      >>>>>> failing in the receive part of
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.accumulo.core.client.impl.thrift.ClientService.Client.authenticate().
>>>      >>>>>>
>>>      >>>>>> On my tservers, I see the following:
>>>      >>>>>>
>>>      >>>>>> 2015-06-06 18:58:19,616 [server.TThreadPoolServer]
ERROR:
>>> Error
>>>      >>>>>> occurred during processing of message.
>>>      >>>>>> java.lang.RuntimeException:
>>>      >>>>>> org.apache.thrift.transport.TTransportException:
>>>      >>>>>> java.net.SocketTimeoutException: Read timed out
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:51)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory$1.run(UGIAssumingTransportFactory.java:48)
>>>      >>>>>>            at
>>> java.security.AccessController.doPrivileged(Native
>>>      >>>>>> Method)
>>>      >>>>>>            at
>>> javax.security.auth.Subject.doAs(Subject.java:356)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.accumulo.core.rpc.UGIAssumingTransportFactory.getTransport(UGIAssumingTransportFactory.java:48)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:208)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
>>>      >>>>>>            at java.lang.Thread.run(Thread.java:745)
>>>      >>>>>> Caused by: org.apache.thrift.transport.TTransportException:
>>>      >>>>>> java.net.SocketTimeoutException: Read timed out
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
>>>      >>>>>>            at
>>>      >>>>>>
>>>     org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:182)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>>>      >>>>>>            ... 11 more
>>>      >>>>>> Caused by: java.net.SocketTimeoutException: Read
timed out
>>>      >>>>>>            at java.net.SocketInputStream.socketRead0(Native
>>>     Method)
>>>      >>>>>>            at
>>>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:152)
>>>      >>>>>>            at
>>>      >>>>>> java.net.SocketInputStream.read(SocketInputStream.java:122)
>>>      >>>>>>            at
>>>      >>>>>>
>>> java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>      >>>>>>            at
>>>      >>>>>>
>>> java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>>>      >>>>>>            at
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>
>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
>>>      >>>>>>            ... 17 more
>>>      >>>>>>
>>>      >>>>>> Any ideas why?
>>>      >>>>>>
>>>      >>>>>> Thanks.
>>>      >>>>>> -Simon
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>>
>>>      >>>>>> On Sat, Jun 6, 2015 at 2:19 PM, Josh
>>>     Elser<josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
>>>
>>>      >>>>>> wrote:
>>>      >>>>>>>
>>>      >>>>>>> Make sure you read the JavaDoc on DelegationToken:
>>>      >>>>>>>
>>>      >>>>>>> <snip>
>>>      >>>>>>> Obtain a delegation token by calling {@link
>>>      >>>>>>>
>>>      >>>>>>>
>>>      >>>>>>>
>>>      >>>>>>>
>>>
>>> SecurityOperations#getDelegationToken(org.apache.accumulo.core.client.admin.DelegationTokenConfig)}
>>>      >>>>>>> </snip>
>>>      >>>>>>>
>>>      >>>>>>> You cannot create a usable DelegationToken
as the client
>>>     itself.
>>>      >>>>>>>
>>>      >>>>>>> Anyways, DelegationTokens are only relevant
in cases where
>>>     the client
>>>      >>>>>>> Kerberos credentials are unavailable. The most
common case
>>>     is running
>>>      >>>>>>> MapReduce jobs. If you are just interacting
with Accumulo
>>>     through the
>>>      >>>>>>> Java
>>>      >>>>>>> API, the KerberosToken is all you need to use.
>>>      >>>>>>>
>>>      >>>>>>> The user-manual likely just needs to be updated.
I believe
>>> the
>>>      >>>>>>> DelegationTokenConfig was added after I wrote
the initial
>>>      >>>>>>> documentation.
>>>      >>>>>>>
>>>      >>>>>>>
>>>      >>>>>>> Xu (Simon) Chen wrote:
>>>      >>>>>>>>
>>>      >>>>>>>> Hi folks,
>>>      >>>>>>>>
>>>      >>>>>>>> The latest kerberos doc seems to indicate
that
>>>     getDelegationToken
>>>      >>>>>>>> can
>>>      >>>>>>>> be
>>>      >>>>>>>> called without any parameters:
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>
>>> https://github.com/apache/accumulo/blob/1.7/docs/src/main/asciidoc/chapters/kerberos.txt#L410
>>>      >>>>>>>>
>>>      >>>>>>>> Yet the source code indicates a DelegationTokenConfig
>>>     object must be
>>>      >>>>>>>> passed in:
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>
>>> https://github.com/apache/accumulo/blob/1.7/core/src/main/java/org/apache/accumulo/core/client/admin/SecurityOperations.java#L359
>>>      >>>>>>>>
>>>      >>>>>>>> Any ideas on how I should construct the
>>> DelegationTokenConfig
>>>      >>>>>>>> object?
>>>      >>>>>>>>
>>>      >>>>>>>> For context, I've been trying to get geomesa
to work on my
>>>     accumulo
>>>      >>>>>>>> 1.7
>>>      >>>>>>>> with kerberos turned on. Right now, the
code is somewhat
>>>     tied to
>>>      >>>>>>>> password auth:
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>      >>>>>>>>
>>>
>>> https://github.com/locationtech/geomesa/blob/rc7_a1.7_h2.5/geomesa-core/src/main/scala/org/locationtech/geomesa/core/data/AccumuloDataStoreFactory.scala#L177
>>>      >>>>>>>> My thought is that I should get a KerberosToken
first, and
>>>     then try
>>>      >>>>>>>> generate a DelegationToken, which is passed
back for later
>>>      >>>>>>>> interactions
>>>      >>>>>>>> between geomesa and accumulo.
>>>      >>>>>>>>
>>>      >>>>>>>> Thanks.
>>>      >>>>>>>> -Simon
>>>
>>>
>

Mime
View raw message