Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Message-ID: <BLU436-SMTP1118CE8A0040684BED25761D1720@phx.gbl>
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: [QUESTION] thread model in Flink makes me confused
From: "Wright, Eron" <ewright@live.com>
In-Reply-To: <a9b44dd8-d16f-4c48-890e-3accdf45c550.wuchong.wc@alibaba-inc.com>
Date: Wed, 11 May 2016 16:58:57 -0700
Content-Transfer-Encoding: quoted-printable
References: <a9b44dd8-d16f-4c48-890e-3accdf45c550.wuchong.wc@alibaba-inc.com>
To: dev@flink.apache.org,
 Jark Wu <wuchong.wc@alibaba-inc.com>
archived-at: Wed, 11 May 2016 23:59:07 -0000

One option is to use a separate cluster (JobManager + TaskManagers) for =
each job.   This is fairly straightforward with the YARN support - =
"flink run=E2=80=9D can launch a cluster for a job and tear it down =
afterwards.

Of course this means you must deploy YARN.   That doesn=E2=80=99t =
necessarily imply HDFS though a Hadoop-compatible filesystem (HCFS) is =
needed to support the YARN staging directory.=20

This approach also facilitates richer scheduling and multi-user =
scenarios.  =20

One downside is the loss of a unified web UI to view all jobs.


> On May 11, 2016, at 8:32 AM, Jark Wu <wuchong.wc@alibaba-inc.com> =
wrote:
>=20
>=20
> As I know, Flink uses thread model, that means one TaskManager process =
may run many different operator threads from different jobs. So tasks =
from different jobs will compete for memory and CPU in the one process. =
In the worst case scenario, the bad job will eat most of CPU and memroy =
which may lead to OOM, and then the regular job died too. And there's =
another problem, tasks from different jobs will print there logs into =
the same file(the taskmanager log file). This increases the difficulty =
of debugging.
>=20
> As I know, Storm will spawn workers for every job. The tasks in one =
worker belong to the same job. So I'm confused the purpose or advantages =
of Flink design. One more question, is there any tips to solves the =
issues above? Or any suggestions to implemention the similar desgin with =
Storm ?=20
>=20
> Thank you for any answers in advance!
>=20
> Regards,
> Jark Wu
>=20
>=20
>=20