flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: [Discuss] Why different job's tasks can run in the single process.
Date Fri, 01 Jul 2016 15:52:18 GMT
I'm not aware of any plans to change this. It would be possible though, to
add a mode where a standalone cluster only accepts a single job.

On Fri, 1 Jul 2016 at 04:50 Jark Wu <wuchong.wc@alibaba-inc.com> wrote:

> As standalone mode has the disadvantage that the TaskManager JVM can’t
> isolate different jobs. Does we have any plan to improve this ?
>
> - Jark Wu
>
> > 在 2016年6月30日,下午4:19,Kevin Jacobs <kevin.jacobs@cern.ch> 写道:
> >
> > In my opinion the streaming process can be perfectly simulated on a
> single node. You can setup a message distribution system like Kafka on a
> single node, you can run Spark on a single node and the only thing you need
> to change when running it on a cluster is that you need to change the
> environment. So there is no need to setup a cluster when testing the
> streaming process.
> >
> > Regards,
> > Kevin
> >
> > On 30-06-16 09:54, Longda Feng wrote:
> >>
> >> This means Standalone mode is just for prototype.
> >> But I think we need a lightweight solution for streaming process,
> standalone is the best solution. Some times, we need setup a flink cluster
> on a small cluster. setup a yarn cluster isn't convenient.
> >> (1) in small company, the number of machine is small (2) When a data
> center is small, but we still need do some computing in this data center(3)
> some machines are in the while-list, they have been authorited to access
> some special data or machine, but the number of these machine is small.(4)
> some machine has critical data, they can't be shared with others, but the
> number of these machine is small.(5) when a team start to learn flink, he
> will setup a small cluster firstly, maybe he wo't want to setup a huge
> system, perfer to a small system
> >>
> >>
> >> regardsLongda
> >>
> >>
> ------------------------------------------------------------------From:Aljoscha
> Krettek <aljoscha@apache.org>Send Time:2016年6月29日(星期三) 21:48To:封仲淹(纪君祥)
<
> zhongyan.feng@alibaba-inc.com>; dev <dev@flink.apache.org>Subject:Re:
> [Discuss] Why different job's tasks can run in the single process.
> >> Hi,
> >> yes, you are definitely right that allowing to run multiple user code
> tasks
> >> in the same TaskManager JVM is not good for stability. This mode is
> still
> >> there from the very early days of Flink where Yarn was not yet
> available.
> >> In a production environment I would now recommend to always run one
> >> Flink-Yarn cluster per job to get good isolation between different jobs.
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >> On Wed, 29 Jun 2016 at 09:18 Longda Feng <zhongyan.feng@alibaba-inc.com
> >
> >> wrote:
> >>
> >>>  hi ,
> >>>  Sorry for asking the quest here? Any answer will be apprecated.
> >>>  Why different job's tasks can run in the single process. (There are
> some
> >>>  different job's tasks  in one TaskManager).It seems Flink-on-Yarn can
> let
> >>>  different job  run on different process. But for standalone mode, this
> >>>  problem still exists.
> >>>  Why design Flink like this?The advantage What I can thought is as
> >>>  following:(1) All task can share bigger memory pool.(2) The
> communication
> >>>  between the tasks in the same process will be fast.
> >>>  But this design will impact to the stability. Flink provide
> >>>  User-Define-Function interface, if one of the User-Define-Function
> crash,
> >>>  It maybe crack the whole JVM, If the TaskManager crash, all other
> job's
> >>>  task in this TaskManager will be impacted. Even if the JVM don't
> crash, but
> >>>  maybe lead to some other unexpected problem, what's more this will
> make the
> >>>  code too sophisticated。Normal framework like Spark/Storm/Samza won't
> run
> >>>  different job's tasks in the same process。As one normal user,
> stability has
> >>>  the highest priority.
> >>>
> >>>  ThanksLongda
> >>>
> >>>
> >>>
> >>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message