flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Replacing JobManager with Scala implementation
Date Fri, 05 Sep 2014 10:10:19 GMT
Hi Daniel,

the genesis of my proposal was that I resumed Asterios' effort to introduce
Akka to handle our RPC calls. Asterios idea was to incorporate Akka into
Flink transparently. That meant to use Akka for the MethodInvoker of the
proxies and as the server to handle the calls. While rebasing it on the
latest master of Flink I noticed that we left out some potential of Akka by
pursuing this approach. Among others, this includes the supervision,
monitoring and exception forwarding mechanism. By making the JobManager and
TaskManager (and possibly other components if it proves worth) actors, we
would get this for free, without having to go through the proxies, method
invoker and call actors. Furthermore, I think that this adds unnecessary
complexity. If we use Akka, why not properly? At this point you are right
that we did not vote on using Akka and I did not think about it. I thought
that by continuing Asterios work, it would comply with the project.

However, making JobManagers and TaskManagers actors comes at the price of
rewriting them, which is unavoidable. Since Akka offers Scala as well as
Java bindings, it would be possible to do it with both. Since Scala
supports pattern matching, lambdas and embraces more the functional
paradigm, it is in my view better suited for implementing an actor based
RPC system. With Java, the code will become more verbose and probably not
so neat to read.

For example, for each message it would be necessary to create a Java class
with the members, getters and setters whereas in Scala it is most of the
time a one line case class definition. For each call back it would be
necessary to implement an interface and overwrite the corresponding method
whereas for Scala it is just a lambda expression (to be precise an
anonymous function). For short call back functions we would have more Java
boiler plate code than actual function code.

In order to distinguish the different messages in Java, one would have a
long list of if(msg instanceof RegisterMessage){} else if (msg instanceof
SubmitTask){}.... and cannot directly access the members of the message
classes. In Scala it would be simply a pattern matching. Since both
languages are Turing-complete and even run in the same JVM, the differences
are thus conciseness and expressiveness. But in the end everything can also
be implemented in Java.

There is no functional argument for Scala just the advantages of a shorter
and probably more readable implementation. Since I will probably implement
it, I'd opt for the shorter solution and thus I started this vote. But
there is no necessity to use Scala here. If we decide for Akka, then Akka
will simply motivate the use of Scala but not enforce it.

Best regards,


On Fri, Sep 5, 2014 at 11:24 AM, Kostas Tzoumas <ktzoumas@apache.org> wrote:

> Hi Daniel,
> +1 on the argument about attracting developers being irrelevant, the
> argument can work both ways and is very brittle
> The reasons for using Akka as a library (irrespective of the programming
> language) have been clearly articulated in my opinion by Stephan and Till
> in this thread.
> The reasons for using the Scala Akka API versus the Java Akka API is simply
> ease and speed of development, as Scala is a better language for this task.
> Perhaps Till could expand the argumentation a bit more here, but I suspect
> that a language with pattern matching is a good fit for message
> passing-like systems.
> Kostas
> On Fri, Sep 5, 2014 at 12:38 AM, Ufuk Celebi <uce@apache.org> wrote:
> > Hey Daniel,
> >
> > On Thu, Sep 4, 2014 at 11:36 PM, Daniel Warneke <warneke@apache.org>
> > wrote:
> >
> > > I agree that the questions “akka” and “Scala” should be treated
> > > separately. Unfortunately, this is not how the discussion has been led
> so
> > > far. Instead, the new akka RPC service is used to motivate the
> necessity
> > > for Scala in the runtime core. I still don’t see that necessity. I
> tried
> > to
> > > find the implementation of the new akka RPC service on github. The only
> > > code I found was from Asterios, but it looks like he was perfectly able
> > to
> > > encapsulate the whole akka RPC thing in 5 Java classes [1].
> > >
> >
> > I don't think that anybody is talking about the necessity of Scala. Yes,
> > Akka and an actor based refactoring of core runtime parts result in a
> hard
> > dependency to Scala for the core (because Akka is written in Scala), but
> it
> > does *not* necessitate to do the refactoring itself in Scala, because
> there
> > is an Akka Java API as well.
> >
> > Are you concerned with the dependency to Scala or with using Akka's Scala
> > API?
> >
> > I think that Till started this thread and the [VOTE] exactly because he
> is
> > well aware that it is *not* necessary to do it in Scala. He sees good
> > reasons to do it in Scala and asks the community to vote on it. Again,
> > because he is aware that this is not a small or light weight change.
> >
> > The second argument (Scala will attract new developers to the project) is
> > > nothing but speculation. This might as well totally backfire and lead
> to
> > > the opposite.
> > >
> >
> > I agree that this point is speculative and both outlined outcomes
> (attract
> > or repel developers) are possible. But it is also not the only argument
> > that has been raised in favor of Scala. Other more technical (not
> > speculative) points have been given. It is the goal of the vote to find a
> > consensus about whether these points are sufficient or not.
> >
> >
> > > The only explanation I have for this push towards akka and Scala is
> that
> > > there are already plans to expand the usage of akka way beyond pure
> RPC.
> > In
> > > this case, I feel these plans should be clearly articulated on the dev
> > > list. A simple RPC service does not justify the proposed changes in my
> > > opinion.
> > >
> >
> > There is no push for Scala. It's a vote. And the reasons for going for
> Akka
> > have been repeated a few times by now.
> >
> > I think the way that Asterios initially introduced Akka beneath the
> > existing RPC proxy service (independently of whether he did in Scala or
> > Java) would not allow us to make use of central features of Akka (some of
> > which Till and Stephan outlined).
> >
> > Best wishes,
> >
> > Ufuk
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message