Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Message-ID: <5920E174.1050802@gmail.com>
Date: Sat, 20 May 2017 20:38:12 -0400
From: Josh Elser <josh.elser@gmail.com>
User-Agent: Postbox 3.0.11 (Macintosh/20140602)
MIME-Version: 1.0
To: user@accumulo.apache.org
Subject: Re: ClientConfiguration using Kerberos & MapReduce
References: <CAGSDCmuaidujPkPu_Jki-U+zrTwbuuRJofhVcCgLDFTcuQ-g+Q@mail.gmail.com> <CAGSDCmsySYOPNbgphmaqod8iEP9U-a-Wa+Sf-XjX_FCMc=6uaQ@mail.gmail.com> <CAGSDCmvB5UgdKvpuOcRAeTXYhTwySn2+1tCkQjzOTJW-g4gMMQ@mail.gmail.com> <591E1006.7040003@gmail.com> <CAGSDCmuZSbRiTyJkT5SkyRUuA0bbHFtgMt9xW8JaU9Yd2SnOGw@mail.gmail.com> <591F5BF7.9080600@gmail.com> <CAGSDCmshGUwvHTtH+hkhzswu+mtSJVNM1vgNYAYNJPZFJooOMA@mail.gmail.com> <CAGSDCmuXu37UfA7uYC6B7fvxneT8ru-Bu_c9VLdMrUsAqVYtCg@mail.gmail.com>
In-Reply-To: <CAGSDCmuXu37UfA7uYC6B7fvxneT8ru-Bu_c9VLdMrUsAqVYtCg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Sun, 21 May 2017 00:38:28 -0000


James Srinivasan wrote:
>>> Delegation tokens are serialized into the Job's "credentials" section and
>>> distributed securely that way.
>> Ah, that's my problem. Will probably have to update the GeoMesa code
>> to wok with Jobs rather than Configurations, so that the Credentials
>> aren't lost.
>
> Hmm, not so easy it seems. My callstack which triggers the exception
> when the credentials are missing from the Job is this:
>
> java.lang.NullPointerException
>    at org.apache.accumulo.core.client.mapreduce.lib.impl.ConfiguratorBase.unwrapAuthenticationToken(ConfiguratorBase.java:493)
>    at org.apache.accumulo.core.client.mapreduce.AbstractInputFormat.validateOptions(AbstractInputFormat.java:390)
>    at org.apache.accumulo.core.client.mapreduce.AbstractInputFormat.getSplits(AbstractInputFormat.java:668)
>    at org.locationtech.geomesa.jobs.mapreduce.GeoMesaAccumuloInputFormat.getSplits(GeoMesaAccumuloInputFormat.scala:174)
>    at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:121)
> ...
>
> Now org.apache.spark.rdd.NewHadoopRDD.getPartitions does this:
>
>    val jobContext = new JobContextImpl(_conf, jobId)
>
> So doesn't seem to support tokens (Jobs) being supplied, just Configurations.
>
> I can't call AccumuloInputFormat.setConnectorInfo again since it has
> already been called, and I presume adding the serialised token to the
> Configuration would be insecure?

Yeah, the configuration can't protect sensitive information. 
MapReduce/YARN has special handling to make sure those tokens serialized 
in the Job's credentials are only readable by you (the job submitter).

The thing I don't entirely follow is how you've gotten into this 
situation to begin with. The adding of the delegation tokens to the 
Job's credentials should be done by Accumulo's MR code on your behalf 
(just like it's obtaining the delegation token, it would automatically 
add it to the job for ya).

Any chance you can provide an end-to-end example? I am also pretty 
Spark-ignorant -- so maybe I just don't understand what is possible and 
what isn't..

> Yours in puzzlement,
>
> James