beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lukasz Cwik <>
Subject Re: Donating the Dataflow Worker code to Apache Beam
Date Thu, 13 Sep 2018 21:34:57 GMT
Romain, the code is very similar to the adaptation layer between the shared
libraries part of Apache Beam and any other runner, for example the code
within runners/spark or runners/apex or runners/flink.
If someone wanted to build an emulator of the Dataflow service, they would
be able to re-use them but that is as impractical as writing an emulator
for Flink or Spark and plugging them in as the dependency for runners/flink
and runners/spark respectively.

On Thu, Sep 13, 2018 at 2:07 PM Raghu Angadi <> wrote:

> On Thu, Sep 13, 2018 at 12:53 PM Romain Manni-Bucau <>
> wrote:
>> If usable by itself without google karma (can you use a worker without
>> dataflow itself?) it sounds awesome otherwise it sounds weird IMHO.
> Can you elaborate a bit more on using worker without dataflow? I
> essentially  see that as o part of Dataflow runner. A runner is specific to
> a platform.
> I am a Googler, but commenting as a community member.
> Raghu.
>> Le jeu. 13 sept. 2018 21:36, Kai Jiang <> a écrit :
>>> +1 (non googler)
>>> big help for transparency and for future runners.
>>> Best,
>>> Kai
>>> On Thu, Sep 13, 2018, 11:45 Xinyu Liu <> wrote:
>>>> Big +1 (non-googler).
>>>> From Samza Runner's perspective, we are very happy to see dataflow
>>>> worker code so we can learn and compete :).
>>>> Thanks,
>>>> Xinyu
>>>> On Thu, Sep 13, 2018 at 11:34 AM Suneel Marthi <>
>>>> wrote:
>>>>> +1 (non-googler)
>>>>> This is a great 👍 move
>>>>> Sent from my iPhone
>>>>> On Sep 13, 2018, at 2:25 PM, Tim Robertson <>
>>>>> wrote:
>>>>> +1 (non googler)
>>>>> It sounds pragmatic, helps with transparency should issues arise and
>>>>> enables more people to fix.
>>>>> On Thu, Sep 13, 2018 at 8:15 PM Dan Halperin <>
>>>>> wrote:
>>>>>> From my perspective as a (non-Google) community member, huge +1.
>>>>>> I don't see anything bad for the community about open sourcing more
>>>>>> of the probably-most-used runner. While the DirectRunner is probably
>>>>>> the most referential implementation of Beam, can't hurt to see more
>>>>>> code. Other runners or runner implementors can refer to this code
if they
>>>>>> want, and ignore it if they don't.
>>>>>> In terms of having more code and tests to support, well, that's par
>>>>>> for the course. Will this change make the things that need to be
done to
>>>>>> support them more obvious? (E.g., "this PR is blocked because someone
>>>>>> Google on Dataflow team has to fix something" vs "this PR is blocked
>>>>>> because the Apache Beam code in foo/bar/baz is failing, and anyone
who can
>>>>>> see the code can fix it"). The latter seems like a clear win for
>>>>>> community.
>>>>>> (As long as the code donation is handled properly, but that's
>>>>>> completely orthogonal and I have no reason to think it wouldn't be.)
>>>>>> Thanks,
>>>>>> Dan
>>>>>> On Thu, Sep 13, 2018 at 11:06 AM Lukasz Cwik <>
>>>>>> wrote:
>>>>>>> Yes, I'm specifically asking the community for opinions as to
>>>>>>> whether it should be accepted or not.
>>>>>>> On Thu, Sep 13, 2018 at 10:51 AM Raghu Angadi <>
>>>>>>> wrote:
>>>>>>>> This is terrific!
>>>>>>>> Is thread asking for opinions from the community about if
it should
>>>>>>>> be accepted? Assuming Google side decision is made to contribute,
big +1
>>>>>>>> from me to include it next to other runners.
>>>>>>>> On Thu, Sep 13, 2018 at 10:38 AM Lukasz Cwik <>
>>>>>>>> wrote:
>>>>>>>>> At Google we have been importing the Apache Beam code
base and
>>>>>>>>> integrating it with the Google portion of the codebase
that supports the
>>>>>>>>> Dataflow worker. This process is painful as we regularly
are making
>>>>>>>>> breaking API changes to support libraries related to
running portable
>>>>>>>>> pipelines (and sometimes in other places as well). This
has made it
>>>>>>>>> sometimes difficult for PR changes to make changes without
either breaking
>>>>>>>>> something for Google or waiting for a Googler to make
the change internally
>>>>>>>>> (e.g. dependency updates).
>>>>>>>>> This code is very similar to the other integrations that
exist for
>>>>>>>>> runners such as Flink/Spark/Apex/Samza. It is an adaption
layer that sits
>>>>>>>>> on top of an execution engine. There is no super secret
awesome stuff as
>>>>>>>>> this code was already publicly visible in the past when
it was part of the
>>>>>>>>> Google Cloud Dataflow github repo[1].
>>>>>>>>> Process wise the code will need to get approval from
Google to be
>>>>>>>>> donated and for it to go through the code donation process
but before we
>>>>>>>>> attempt to do that, I was wondering whether the community
would object to
>>>>>>>>> adding this code to the master branch?
>>>>>>>>> The up side is that people can make breaking changes
and fix it
>>>>>>>>> for all runners. It will also help Googlers contribute
more to the
>>>>>>>>> portability story as it will remove the burden of doing
the code import
>>>>>>>>> (wasted time) and it will allow people to develop in
master (can have the
>>>>>>>>> whole project loaded in a single IDE).
>>>>>>>>> The downsides are that this will represent more code
and unit
>>>>>>>>> tests to support.
>>>>>>>>> 1:

View raw message