beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Sookocheff <kevin.sookoch...@gmail.com>
Subject Re: [PROPOSAL] Beam recipes (integration new IOs & transforms)
Date Fri, 04 Mar 2016 18:03:23 GMT
We have been using ParDo for now as well. And it works fine. Our use case
revolves around batching data — a typical case is outputting data to a
rate-limited API. We can do that already with existing primitives but it
would be nice to have helper functions and an interface for that. Totally
understand about priority though :).

Kevin

On Fri, 4 Mar 2016 at 11:39 Dan Halperin <dhalperi@google.com.invalid>
wrote:

> Hi Kevin,
>
> Can you say more about your needs for an UnboundedSink API?
>
> Basically, we have found most Sinks are fairly simple and can be
> implemented with just ParDo; so the Unbounded Sink API has generally been
> dropping in priority relative to all the work we've got to do for general
> Beam-readiness and enabling other Runners.
>
> I'd love to have a discussion about what you'd like to see in the Sink API.
>
> Thanks,
> Dan
>
> On Fri, Mar 4, 2016 at 7:59 AM, Jean-Baptiste Onofré <jb@nanthrax.net>
> wrote:
>
> > Hi Kevin,
> >
> > it's part of the area where we are working for the IO, yes.
> >
> > Right now, I'm working on the unbounded source (Java), but next one is
> > unbounded sink (we will discuss with Dan about that).
> >
> > Regards
> > JB
> >
> >
> > On 03/04/2016 03:26 PM, Kevin Sookocheff wrote:
> >
> >> One thing we have been waiting on as Dataflow users is the finalization
> of
> >> the unbounded sink API. I'm not sure if this would fit into the scope
> for
> >> this example proposal though.
> >> On Fri, Mar 4, 2016 at 4:13 AM Jean-Baptiste Onofré <jb@nanthrax.net>
> >> wrote:
> >>
> >> Hi guys,
> >>>
> >>> all good things have an end: I'm back from vacation ;)
> >>>
> >>> This morning, I'm resuming my work on new IOs and transforms, data
> >>> integration oriented. Dan already provided great feedbacks on the JMS
> >>> one, and I'm updating it to use "new" Bounded and Unbounded interfaces.
> >>> Unfortunately, right now, we don't have a lot of samples of unbounded
> IO
> >>> in Java.
> >>>
> >>> Actually, I think it would be great if we provide straight forward
> >>> recipes, showing how to implement pipelines using IOs & transform to
> >>> solve simple but real problems. The WordCount sample is a good one. We
> >>> can provide new one to show IOs/data integration IOs, more complex
> >>> transforms, etc.
> >>>
> >>> Thoughts ?
> >>>
> >>> Regards
> >>> JB
> >>> --
> >>> Jean-Baptiste Onofré
> >>> jbonofre@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message