accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Accumulo 1.7 InputFormat Iterator Question
Date Thu, 18 Aug 2016 15:36:31 GMT
bq. Note that the method isn't idempotent. To clear the iterators on a 
job you can call 
job.getConfiguration.unset("AccumuloInputFormat.ScanOpts.Iterators") 
(but that isn't officially part of the public API)

Hey Russ -- do you have a use case behind why you know the above trick? 
Is there a reason that we might want to formally support removal of 
iterators from a Job's Configuration?

Christopher wrote:
> I'm not sure it's really necessary to add the ability to clear. The way I
> see it, you create the configuration, then execute, create a new
> configuration, then execute, etc. I'm not sure what the use case is for
> clearing.... if you didn't want it there, why did you add it only to remove
> it? But it seems to me that clearing indicates a potential misuse, or at
> least, anti-best-practices. Granted, I don't do a lot of MapReduce these
> days, maybe I'm missing some significant use case.
>
> Russ mentioned the method isn't idempotent. That's true. Additionally, the
> Scanner will throw an exception if the name and/or priority is already in
> use. We don't do any up-front deduplication in the job config.
>
> On Wed, Aug 17, 2016 at 5:45 PM Josh Elser<josh.elser@gmail.com>  wrote:
>
>> Sounds like something we should be addressing (the ability to clear
>> iterators from a Job's configuration)...
>>
>> -------- Original Message --------
>> Subject:        Re: Accumulo 1.7 InputFormat Iterator Question
>> Date:   Wed, 17 Aug 2016 21:31:01 +0000
>> From:   Russ Weeks<rweeks@newbrightidea.com>
>> Reply-To:       user@accumulo.apache.org
>> To:     user@accumulo.apache.org
>>
>>
>>
>> Hi, Jamie,
>>
>> Try the static method AccumuloInputFormat.addIterator(job, new
>> IteratorSetting(...)).
>>
>> Note that the method isn't idempotent. To clear the iterators on a job
>> you can
>> call
>> job.getConfiguration.unset("AccumuloInputFormat.ScanOpts.Iterators") (but
>> that isn't officially part of the public API)
>>
>> -Russ
>>
>> On Wed, Aug 17, 2016 at 2:26 PM Jamie Johnson<jej2003@gmail.com
>> <mailto:jej2003@gmail.com>>  wrote:
>>
>>       I am upgrading from Accumulo 1.6 to 1.7 and I am trying to
>>       understand how iterators are supposed to be set in 1.7 for an input
>>       format.  In my situation, if a particular property is set an
>>       additional iterator needs to be added to do some additional
>>       checking.  Previously I had done this in the
>>       AbstractRecordReader.setupIterators() method but this has been
>>       deprecated.  I had attempted to put them in
>>       AbstractRecordReader.contextIterators(), but this isn't always
>>       called.  This change has made me question if I was ever doing this
>>       according to best practices and now wonder what the correct way to
>>       do this is.  Any pointers would be greatly appreciated.
>>
>>
>

Mime
View raw message