crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From venkatesh kavuluri <venkatesh.kowl...@gmail.com>
Subject Re: call DoFn#increment from helper class
Date Wed, 18 Feb 2015 21:49:57 GMT
Thanks Micah. We are going to use only MRPipeline and passing
the TaskInputOutputContext enabled me to implement counters in the helper
class.

On Thu, Feb 12, 2015 at 4:27 PM, Micah Whitacre <mkwhitacre@gmail.com>
wrote:

> Venkatesh,
>   There isn't really a great way without you exposing your implementation
> of DoFn to the runtime container you are running in.  The DoFn.increment
> method is hiding if the "increment" call is being made to alter a MapReduce
> counter or Spark.  If you know that you will only ever run in a MapReduce
> world then you could pull the TaskInputOutputContext and increment a
> counter using that but that seems like an abstraction leak of a concept
> only known to your selected Pipeline implementation.
>
> Micah
>
> On Thu, Feb 12, 2015 at 6:12 PM, venkatesh kavuluri <
> venkatesh.kowluru@gmail.com> wrote:
>
>> Hi All,
>>
>> I am trying to figure out the best way to call increment method (for
>> counters) from a helper class invoked from DoFn#process.
>>
>> I have something like this.
>>
>> class A extends DoFn<> {
>>
>> Foo foo = nee Foo(); // helper class
>>
>> @Override
>> public void process (S input, Emitter<T> emitter) {
>>     foo.processAndEmitEvents(bar, emitter);
>> }
>>
>> I want to call DoFn#increment method from Foo#processAndEmitEvents. Is
>> there a better way to go about it other than “Foo" extending “A” just to
>> get hold of DoFn#increment.
>>
>> Thanks for the help.
>>
>
>

Mime
View raw message