Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B05351796E for ; Fri, 13 Feb 2015 00:29:21 +0000 (UTC) Received: (qmail 30977 invoked by uid 500); 13 Feb 2015 00:29:21 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 30927 invoked by uid 500); 13 Feb 2015 00:29:21 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 30917 invoked by uid 99); 13 Feb 2015 00:29:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2015 00:29:21 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mkwhitacre@gmail.com designates 209.85.213.172 as permitted sender) Received: from [209.85.213.172] (HELO mail-ig0-f172.google.com) (209.85.213.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Feb 2015 00:28:52 +0000 Received: by mail-ig0-f172.google.com with SMTP id l13so7453627iga.5 for ; Thu, 12 Feb 2015 16:27:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=qXuuKK/IKawWo29r6vVe0tL/z+16WNl0trX+yhIdqR0=; b=kmFyCqgEWVFwFbgKwO6fhT/Lwer/KSdzwaHeQ/rRRKgh/18CTIiEP77Jy2YYNCTU4s VUsf7nfHbOTCIGyV1u+0G+OXkiuw6cTzAx3jwN87inc63QAxSceoT+d58P03LHS6Jp2Y nLWY8D8DBjOuDeNwE1Hr9w47FpYf4ah6d04NKlYKhQ6ovpNwjRo0i8aqIBq7lDxJpyph 4wjf05CFWDx5grFmqWCity/amAWKrMQZuZwY33kUxeqIzetIsnkesxrKGBFj5xLUwf1W jm7w8j0ofLt4CYLUqQ887IJNm5NF36u/LcYu2ZWRuZtO3DsfxpdkLVve8YriLRnREm7B +3vA== MIME-Version: 1.0 X-Received: by 10.42.78.137 with SMTP id n9mr11915385ick.53.1423787240540; Thu, 12 Feb 2015 16:27:20 -0800 (PST) Received: by 10.36.106.5 with HTTP; Thu, 12 Feb 2015 16:27:20 -0800 (PST) In-Reply-To: References: Date: Thu, 12 Feb 2015 18:27:20 -0600 Message-ID: Subject: Re: call DoFn#increment from helper class From: Micah Whitacre To: user@crunch.apache.org Content-Type: multipart/alternative; boundary=20cf3010e74bb55053050eed4ceb X-Virus-Checked: Checked by ClamAV on apache.org --20cf3010e74bb55053050eed4ceb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Venkatesh, There isn't really a great way without you exposing your implementation of DoFn to the runtime container you are running in. The DoFn.increment method is hiding if the "increment" call is being made to alter a MapReduce counter or Spark. If you know that you will only ever run in a MapReduce world then you could pull the TaskInputOutputContext and increment a counter using that but that seems like an abstraction leak of a concept only known to your selected Pipeline implementation. Micah On Thu, Feb 12, 2015 at 6:12 PM, venkatesh kavuluri < venkatesh.kowluru@gmail.com> wrote: > Hi All, > > I am trying to figure out the best way to call increment method (for > counters) from a helper class invoked from DoFn#process. > > I have something like this. > > class A extends DoFn<> { > > Foo foo =3D nee Foo(); // helper class > > @Override > public void process (S input, Emitter emitter) { > foo.processAndEmitEvents(bar, emitter); > } > > I want to call DoFn#increment method from Foo#processAndEmitEvents. Is > there a better way to go about it other than =E2=80=9CFoo" extending =E2= =80=9CA=E2=80=9D just to > get hold of DoFn#increment. > > Thanks for the help. > --20cf3010e74bb55053050eed4ceb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Venkatesh,
=C2=A0 There isn't really a great way w= ithout you exposing your implementation of DoFn to the runtime container yo= u are running in.=C2=A0 The DoFn.increment method is hiding if the "in= crement" call is being made to alter a MapReduce counter or Spark.=C2= =A0 If you know that you will only ever run in a MapReduce world then you c= ould pull the=C2=A0TaskInputOutputContext and increment a counter using that b= ut that seems like an abstraction leak of a concept only known to your sele= cted Pipeline implementation.

Micah=

On Th= u, Feb 12, 2015 at 6:12 PM, venkatesh kavuluri <venkatesh.kowlur= u@gmail.com> wrote:
Hi All,

I am trying to figure out the best way to call increment met= hod (for counters) from a helper class invoked from DoFn#process.=C2=A0

I have something like this.

class A extends DoFn<> {

Foo foo = =3D nee Foo(); // helper class

@Override
<= div style=3D"font-size:12.8000001907349px">public void process (S input= , Emitter<T> emitter) {
=C2=A0 =C2=A0 foo.processAndEmitEvents(bar, emitter);
}

I want= to call DoFn#increment method from Foo#processAndEmitEvents. Is there a be= tter way to go about it other than =E2=80=9CFoo" extending =E2=80=9CA= =E2=80=9D just to get hold of DoFn#increment.

T= hanks for the help.

--20cf3010e74bb55053050eed4ceb--