beam-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Romain Manni-Bucau <rmannibu...@gmail.com>
Subject Re: Graal instead of docker?
Date Sat, 05 May 2018 18:49:04 GMT
Agree

The jvm is still mainstream for big data and it is trivial to have a remote
facade to support natives but no point to have it in runners, it is some
particular transforms or even dofn and sources only...


Le 5 mai 2018 19:03, "Andrew Pilloud" <apilloud@google.com> a écrit :

> Thanks for the examples earlier, I think Hazelcast is a great example of
> something portability might make more difficult. I'm not working on
> portability, but my understanding is that the data sent to the runner is a
> blob of code and the name of the container to run it in. A runner with a
> native language (java on Hazelcast for example) could run the code directly
> without the container if it is in a language it supports. So when Hazelcast
> sees a known java container specified, it just loads the java blob and runs
> it. When it sees another container it rejects the pipeline. You could use
> Graal in the Hazelcast runner to do this for a number of languages. I would
> expect that this could also be done in the direct runner, which similarly
> provides a native java environment, so portable Java pipelines can be
> tested without docker?
>
> For another way to frame this: if Beam was originally written in Go, we
> would be having a different discussion. A pipeline written entirely in java
> wouldn't be possible, so instead to enable Hazelcast, we would have to be
> able to run the java from portability without running the container.
>
> Andrew
>
> On Sat, May 5, 2018 at 1:48 AM Romain Manni-Bucau <rmannibucau@gmail.com>
> wrote:
>
>>
>>
>> 2018-05-05 9:27 GMT+02:00 Ismaël Mejía <iemejia@gmail.com>:
>>
>>> Graal would not be a viable solution for the reasons Henning and Andrew
>>> mentioned, or put in other words, when users choose a programming
>>> language
>>> they don’t choose only a ‘friendly’ syntax or programming model, they
>>> choose also the ecosystem that comes with it, and the libraries that make
>>> their life easier. However isolating these user libraries/dependencies
>>> is a
>>> hard problem and so far the standard solution to this problem is to use
>>> operating systems containers via docker.
>>>
>>
>> Graal solves that Ismael. Same kind of experience than running npm libs
>> on nashorn but with a more unified API to run any language soft.
>>
>>
>>>
>>> The Beam vision from day zero is to run pipelines written in multiple
>>> languages in runners in multiple systems, and so far we are not doing
>>> this
>>> in particular in the Apache runners. The portability work is the cleanest
>>> way to achieve this vision given the constraints.
>>>
>>
>> Hmm, did I read it wrong and we don't have specific integration of the
>> portable API in runners? This is what is messing up the runners and
>> limiting beam adoption on existing runners.
>> Portable API is a feature buildable on top of runner, not in runners.
>> Same as a runner implementing the 5-6 primitives can run anything, the
>> portable API should just rely on that and not require more integration.
>> It doesn't prevent more deep integrations as for some higher level
>> primitives existing in runners but it is not the case today for runners so
>> shouldn't exist IMHO.
>>
>>
>>>
>>> I agree however that for the Java SDK to Java runner case this can
>>> represent additional pain, docker ideally should not be a requirement for
>>> Java users with the Direct runner and debugging a pipeline should be as
>>> easy as it is today. I think the Univerrsal Local Runner exists to cover
>>> the Portable case, but after looking at this JIRA I am not sure if
>>> unification is coming (and by consequence if docker would be mandatory).
>>> https://issues.apache.org/jira/browse/BEAM-4239
>>>
>>> I suppose for the distributed runners that they must implement the full
>>> Portability APIs to be considered Beam multi language compliant but they
>>> can prefer for performance reasons to translate without the portability
>>> APIs the Java to Java case.
>>>
>>
>>
>> This is my issue, language portability must NOT impact runners at all, it
>> is just a way to forward primitives to a runner.
>> See it as a layer rewriting the pipeline and submitting it. No need to
>> modify any runner.
>>
>>
>>> On Sat, May 5, 2018 at 9:11 AM Reuven Lax <relax@google.com> wrote:
>>>
>>> > A beam cluster with the spark runner would include a spark cluster,
>>> plus
>>> what's needed for portability, plus the beam sdk.
>>>
>>> > On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> >> Le 5 mai 2018 08:43, "Reuven Lax" <relax@google.com> a écrit
:
>>>
>>> >> I don't believe we enforce docker anywhere. In fact if someone wanted
>>> to
>>> run an all-windows beam cluster, they would probably not use docker for
>>> their runner (docker runs on Windows, but not efficiently).
>>>
>>>
>>>
>>> >> Or doesnt run sometimes - a colleague hit that yesterday :(.
>>>
>>> >> What is a "beam cluster" - opposed to a spark or foink cluster? How
>>> would it work on windows servers?
>>>
>>>
>>> >> On Fri, May 4, 2018, 11:19 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> >>> 2018-05-05 2:33 GMT+02:00 Andrew Pilloud <apilloud@google.com>:
>>>
>>> >>>> What docker really buys is a package format and runtime environment
>>> that is language and operating system agnostic. The docker packaging and
>>> runtime format is the de facto standard for portable applications such as
>>> this, and there is a group trying to turn it into an actual standard.
>>>
>>> >>>> I would agree with you that dockerd has become bloated but there
are
>>> projects that solve that. There is no longer lock-in to dockerd, there
>>> are
>>> package format compatible docker replacements that eliminate the
>>> performance issues and overhead associated with docker. CRI-O (
>>> https://github.com/kubernetes-incubator/cri-o) is a really cool RedHat
>>> project which is a minimalist replacement for docker. I was recently
>>> working at a startup where I migrated our "data mover" appliance from
>>> Docker to CRI-O. Our application was able to get direct access to the
>>> ethernet driver and block devices which enabled a huge performance boost
>>> but we were also able to run containers produced by docker without
>>> modification.
>>>
>>> >>>> You mention that docker is "detail of one runner+vendor corrupting
>>> all
>>> the project and adding complexity and work to everyone". It sounds like
>>> you
>>> have a specific example you'd like to share? Is there a runner that is
>>> unable to move to portability because of docker?
>>>
>>>
>>> >>> IBM one for instance, some custom ones like an hazelcast based one,
>>> etc... More generally any runner developped outside beam itself - even if
>>> we take a snapshot today, most of beam's ones have the same pitall.
>>>
>>> >>> Note: i never said docker was a bad techno or so. Let me try to
>>> clarify.
>>>
>>> >>> Main issue is that you enforce docker usage which is still trendy.
It
>>> is like scla which was promishing to kill java, check what it does
>>> today...
>>> >>> It starts to be tooled but it is also very impacting on the
>>> deployment
>>> side and for a good number of beam users who deploy it outside the cloud
>>> it
>>> is an issue.
>>> >>> Keep in mind beam is embeddable by design, it is not a runner
>>> environment and with the docker choice it imposes some environment which
>>> is
>>> inconsistent with beam design itself and this is where this choice
>>> blocks.
>>>
>>>
>>>
>>> >>>> Andrew
>>>
>>> >>>> On Fri, May 4, 2018 at 4:32 PM Henning Rohde <herohde@google.com>
>>> wrote:
>>>
>>> >>>>> Romain,
>>>
>>> >>>>> Docker, unlike selinux, solves a great number of tangible
problems
>>> for us with IMO a relatively small tax. It does not have to be the only
>>> way. Some of the concerns you bring up along with possibilities were also
>>> discussed here: https://s.apache.org/beam-fn-api-container-contract. I
>>> encourage you to take a look.
>>>
>>> >>>>> Thanks,
>>> >>>>>   Henning
>>>
>>>
>>> >>>>> On Fri, May 4, 2018 at 3:18 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>
>>>
>>> >>>>>> Le 4 mai 2018 21:31, "Henning Rohde" <herohde@google.com>
a
>>> écrit :
>>>
>>> >>>>>> I disagree with the characterization of docker and the
>>> implications
>>> made towards portability. Graal looks like a neat project (and I never
>>> thought I would live to see the phrase "Practical Partial Evaluation"
>>> ..),
>>> but it doesn't address the needs of portability. In addition to Luke's
>>> examples, Go and most other languages don't work on it either. Docker
>>> containers also address packaging, OS dependencies, conflicting versions
>>> and distribution aspects in addition to truly universal language support.
>>>
>>>
>>> >>>>>> This is wrong, docker also has its conflicts, is not
universal
>>> (fails on windows and mac easily - as host or not, cloud vendors put
>>> layers
>>> limiting or corrupting it, and it is an infra constraint imposed and a
>>> vendor locking not welcomed in beam IMHO).
>>>
>>> >>>>>> This is my main concern. All the work done looks like
an
>>> implemzntation detail of one runner+vendor corrupting all the project and
>>> adding complexity and work to everyone instead of keeping it localised
>>> (technically it is possible).
>>>
>>> >>>>>> Would you accept i enforce you to use selinux? Using
docker is the
>>> same kind of constraint.
>>>
>>>
>>> >>>>>> That said, it's entirely fine for some runners to use
Jython,
>>> Graal,
>>> etc to provide a specialized offering similar to the direct runners, but
>>> it
>>> would be disjoint from portability IMO.
>>>
>>> >>>>>> On Fri, May 4, 2018 at 10:14 AM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>
>>>
>>> >>>>>>> Le 4 mai 2018 17:55, "Lukasz Cwik" <lcwik@google.com>
a écrit :
>>>
>>> >>>>>>> I did take a look at Graal a while back when thinking
about how
>>> execution environments could be defined, my concerns were related to it
>>> not
>>> supporting all of the features of a language.
>>> >>>>>>> For example, its typical for Python to load and
call native
>>> libraries and Graal can only execute C/C++ code that has been compiled to
>>> LLVM.
>>> >>>>>>> Also, a good amount of people interested in using
ML libraries
>>> will
>>> want access to GPUs to improve performance which I believe that Graal
>>> can't
>>> support.
>>>
>>> >>>>>>> It can be a very useful way to run simple lamda
functions written
>>> in some language directly without needing to use a docker environment but
>>> you could probably use something even lighter weight then Graal that is
>>> language specific like Jython.
>>>
>>>
>>>
>>> >>>>>>> Right, the jsr223 impl works very well but you can
also have a
>>> perf
>>> boost using native (like v8 java binding for js for instance). It is way
>>> more efficient than docker most of the time and not code intrusive at all
>>> in runners so likely more adoption-able and maintainable. That said all
>>> is
>>> doable behind the jsr223 so maybe not a big deal in terms of api. We just
>>> need to ensure portability work stay clean and actually portable and
>>> doesnt
>>> impact runners as poc done until today did.
>>>
>>> >>>>>>> Works for me.
>>>
>>>
>>> >>>>>>> On Thu, May 3, 2018 at 10:05 PM Romain Manni-Bucau
<
>>> rmannibucau@gmail.com> wrote:
>>>
>>> >>>>>>>> Hi guys
>>>
>>> >>>>>>>> Since some time there are efforts to have a
language portable
>>> support in beam but I cant really find a case it "works" being based on
>>> docker except for some vendor specific infra.
>>>
>>> >>>>>>>> Current solution:
>>>
>>> >>>>>>>> 1. Is runner intrusive (which is bad for beam
and prevents
>>> adoption of big data vendors)
>>> >>>>>>>> 2. Based on docker (which assumed a runtime
environment and is
>>> very ops/infra intrusive and likely too $$ quite often for what it
>>> brings)
>>>
>>> >>>>>>>> Did anyone had a look to graal which seems a
way to make the
>>> feature doable in a lighter manner and optimized compared to default
>>> jsr223
>>> impls?
>>>
>>
>>

Mime
View raw message