crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Micah Whitacre <>
Subject Re: call DoFn#increment from helper class
Date Fri, 13 Feb 2015 00:27:20 GMT
  There isn't really a great way without you exposing your implementation
of DoFn to the runtime container you are running in.  The DoFn.increment
method is hiding if the "increment" call is being made to alter a MapReduce
counter or Spark.  If you know that you will only ever run in a MapReduce
world then you could pull the TaskInputOutputContext and increment a
counter using that but that seems like an abstraction leak of a concept
only known to your selected Pipeline implementation.


On Thu, Feb 12, 2015 at 6:12 PM, venkatesh kavuluri <> wrote:

> Hi All,
> I am trying to figure out the best way to call increment method (for
> counters) from a helper class invoked from DoFn#process.
> I have something like this.
> class A extends DoFn<> {
> Foo foo = nee Foo(); // helper class
> @Override
> public void process (S input, Emitter<T> emitter) {
>     foo.processAndEmitEvents(bar, emitter);
> }
> I want to call DoFn#increment method from Foo#processAndEmitEvents. Is
> there a better way to go about it other than “Foo" extending “A” just to
> get hold of DoFn#increment.
> Thanks for the help.

View raw message