Mailing-List: contact user-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@beam.apache.org
MIME-Version: 1.0
References: <CALEBDJR2pZm1Ctb4w6QnJvAgosKgoLs+dZDO=xfasfhKuca4jg@mail.gmail.com>
In-Reply-To: <CALEBDJR2pZm1Ctb4w6QnJvAgosKgoLs+dZDO=xfasfhKuca4jg@mail.gmail.com>
From: Sourabh Bajaj <sourabhbajaj@google.com>
Date: Fri, 21 Jul 2017 21:10:53 +0000
Message-ID: <CAKYUbUs-4XL+0i-ceW_98C_hg-P5kCa+YjrcScqD5Ufmkk6iNw@mail.gmail.com>
Subject: Re: [Python] Replace ParDo's DoFn after both are constructed
To: user@beam.apache.org
Content-Type: multipart/alternative; boundary="f403045e2a2a8e8a9b0554da4d02"
archived-at: Fri, 21 Jul 2017 21:11:17 -0000

--f403045e2a2a8e8a9b0554da4d02
Content-Type: text/plain; charset="UTF-8"

Hi,

Is it possible to create

class ErrorSieve(PTransform):
   def __init__ (dofn):
   def expand():
         return ParDo(modifiedDoFn)

that way your pipeline just looks like p | ErrorSieve(DoFn()) and you don't
expose the ParDo to the user.

Will this work for your usecase?

-Sourabh

On Fri, Jul 21, 2017 at 2:06 PM Dmitry Demeshchuk <dmitry@postmates.com>
wrote:

> Hi list,
>
> I'm trying to make a transformation function (let's call it ErrorSieve)
> that would take a ParDo object as input and modify its underlying DoFn
> object, basically adding extra logic on top of an underlying process()
> method.
>
> Ideally for me, the example usage would be:
>
> ```python
> p | ErrorSieve(beam.ParDo(MyDoFn())
>
> or
>
> p | ErrorSieve(beam.FlatMap(lambda x: x + 1))
> ```
>
> However, this would require me to butcher the internals of ParDo
> mechanisms, especially since ParDo's make_fn() method gets called during
> its transformation. My other thinking was to make it a fair and square DoFn:
>
> ```python
> p | beam.ParDo(ErrorSieve(MyDoFn())
> ```
>
> The only problem with this is that I can't use it with transforms like
> FlatMap, which is a bit unfortunate.
>
> Do you think it's worth investigating how to implement the first approach,
> or should I just instead settle with the second approach, using only custom
> DoFns?
>
> Thank you.
>
>
> --
> Best regards,
> Dmitry Demeshchuk.
>

--f403045e2a2a8e8a9b0554da4d02
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,=C2=A0<div><br></div><div>Is it possible to create=C2=
=A0</div><div><br></div><div>class ErrorSieve(PTransform):</div><div>=C2=A0=
 =C2=A0def __init__ (dofn):</div><div>=C2=A0 =C2=A0def expand():</div><div>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return ParDo(modifiedDoFn)</div><div><br>=
</div><div>that way your pipeline just looks like p | ErrorSieve(DoFn()) an=
d you don&#39;t expose the ParDo to the user.</div><div><br></div><div>Will=
 this work for your usecase?</div><div><br></div><div>-Sourabh</div></div><=
br><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, Jul 21, 2017 at 2:06=
 PM Dmitry Demeshchuk &lt;<a href=3D"mailto:dmitry@postmates.com">dmitry@po=
stmates.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr">Hi list,<div><br></div><div>I&#39;m trying to make a transformatio=
n function (let&#39;s call it ErrorSieve) that would take a ParDo object as=
 input and modify its underlying DoFn object, basically adding extra logic =
on top of an underlying process() method.</div><div><br></div><div>Ideally =
for me, the example usage would be:</div><div><br></div><div></div><div>```=
python</div><div>p | ErrorSieve(beam.ParDo(MyDoFn())</div><div><br></div><d=
iv>or</div><div><br></div><div>p | ErrorSieve(beam.FlatMap(lambda x: x + 1)=
)</div><div>```</div><div><br></div><div>However, this would require me to =
butcher the internals of ParDo mechanisms, especially since ParDo&#39;s mak=
e_fn() method gets called during its transformation. My other thinking was =
to make it a fair and square DoFn:</div><div><br></div><div>```python</div>=
<div>p | beam.ParDo(ErrorSieve(MyDoFn())</div><div>```</div><div><br></div>=
<div>The only problem with this is that I can&#39;t use it with transforms =
like FlatMap, which is a bit unfortunate.</div><div><br></div><div>Do you t=
hink it&#39;s worth investigating how to implement the first approach, or s=
hould I just instead settle with the second approach, using only custom DoF=
ns?</div><div><br></div><div>Thank you.</div></div><div dir=3D"ltr"><div><b=
r clear=3D"all"><div><br></div>-- <br><div class=3D"m_-1118745781410991587g=
mail_signature" data-smartmail=3D"gmail_signature"><div dir=3D"ltr">Best re=
gards,<div>Dmitry Demeshchuk.</div></div></div>
</div></div></blockquote></div>

--f403045e2a2a8e8a9b0554da4d02--