crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Dolan <>
Subject Re: Google Cloud Dataflow?
Date Sun, 25 Jan 2015 20:53:30 GMT
The promise of being able to port application logic to a different
execution engine by dropping in a different Pipeline implementation was one
of the key reasons for adopting Crunch at Nuna Health.  I'm strongly
supportive of a third Pipeline implementation :)

On Sun, Jan 25, 2015 at 11:01 AM, Josh Wills <> wrote:

> :)
> I never want anyone to have to rewrite code in order to pick up and move
> it to a different execution engine. At the very least, we should write a
> wrapper that lets you run existing o.a.c.DoFn subclasses in Dataflow
> pipelines, and maybe even a DataflowPipeline implementation to port
> existing Crunch pipelines over to Dataflow once it's easier to bring Hadoop
> Input/OutputFormats over.
> I have some thoughts on why I think Dataflow is interesting, esp. to
> developers who are familiar with Crunch/Spark/Scalding, but I'll send them
> out in a different email b/c they're getting kind of long.
> J
> On Sun, Jan 25, 2015 at 10:02 AM, Danny Morgan <>
> wrote:
>> So should I start porting my crunch code over to the Cloud Dataflow sdk?
>> Danny

View raw message