Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Date: Thu, 2 Jun 2016 14:26:00 +0100 (BST)
From: <leon_mclare@tutanota.com>
To: Ufuk Celebi <uce@apache.org>
Cc: <user@flink.apache.org>
Message-ID: <KJGHg68--3-0@tutanota.com>
In-Reply-To: <CAKiyyaE8ieVi3d19Oc1paDPbxcyGh=4xsD2uUn7t+JBGZwVCtw@mail.gmail.com>
References: <KJAMBMP--3-0@tutanota.com> <<KJAMBMP--3-0@tutanota.com>> <CAKiyyaE8ieVi3d19Oc1paDPbxcyGh=4xsD2uUn7t+JBGZwVCtw@mail.gmail.com>
Subject: Re: Internal buffers
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_52842_1401601466.1464873960376"
archived-at: Thu, 02 Jun 2016 13:26:08 -0000

------=_Part_52842_1401601466.1464873960376
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Dear Ufuk,

the wiki entry is exactly what i was looking for. I found it quite 
complicated to understand on a first attempt but i will dedicate some more 
time for it in the future.

Thanks.

Regards
Leon

1. Jun 2016 13:06 by uce@apache.org:


> There is this in the Wiki:
> https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
>
> Buffers for data exchange come from the network buffer pool (by
> default 2048 * 32KB buffers). They are distributed to the running
> tasks and each logical channel between tasks needs at least one
> buffer.
>
> Tasks produce buffers, which are either consumed by the
> a) NettyConnectionManager who has a Thread pool for network
> communication shared by all tasks exchanging remote data (no Thread
> per buffer), or
> b) the consuming task thread (local exchange).
>
> Chained operators run in a single task and exchange records without
> serialization.
>
> On Wed, Jun 1, 2016 at 11:54 AM,  <> leon_mclare@tutanota.com> > wrote:
>> I have a question regarding how tuples are buffered between (possibly
>> chained) subtasks.
>>
>> Is it correct that there is a buffer for each vertex in the DAG of 
>> subtasks?
>> Regardless of task slot sharing? If yes, then the primary optimization in
>> this regard is operator chaining.
>>
>> Furthermore, how do these buffers translate into overhead? Is there a send
>> thread and a receive thread per buffer, similar to Apache Storm?
>>
>> I could not find details concerning such buffers in the relevant 
>> subsection
>> under Concepts.
>>
>> Thanks in advance.
------=_Part_52842_1401601466.1464873960376
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<html>
  <head>
    <meta http-equiv=3D"content-type" content=3D"text/html; charset=3DUTF-8=
">
  </head>
  <body>
Dear Ufuk,<br /><br />the wiki entry is exactly what i was looking for. I f=
ound it quite complicated to understand on a first attempt but i will dedic=
ate some more time for it in the future.<br /><br />Thanks.<br /><br />Rega=
rds<br />Leon<br /><br />1. Jun 2016 13:06 by <a href=3D"mailto:uce@apache.=
org" target=3D"_blank">uce@apache.org</a>:<br /><br /><blockquote class=3D"=
tutanota_quote" style=3D"border-left: 1px solid #93A3B8; padding-left: 10px=
; margin-left: 5px;">There is this in the Wiki:<br /><a target=3D"_blank" h=
ref=3D"https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+betw=
een+tasks">https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+=
between+tasks</a><br /><br />Buffers for data exchange come from the networ=
k buffer pool (by<br />default 2048 * 32KB buffers). They are distributed t=
o the running<br />tasks and each logical channel between tasks needs at le=
ast one<br />buffer.<br /><br />Tasks produce buffers, which are either con=
sumed by the<br />a) NettyConnectionManager who has a Thread pool for netwo=
rk<br />communication shared by all tasks exchanging remote data (no Thread=
<br />per buffer), or<br />b) the consuming task thread (local exchange).<b=
r /><br />Chained operators run in a single task and exchange records witho=
ut<br />serialization.<br /><br />On Wed, Jun 1, 2016 at 11:54 AM,  &lt;<a =
target=3D"_blank" href=3D"mailto:leon_mclare@tutanota.com">leon_mclare@tuta=
nota.com</a>&gt; wrote:<blockquote>I have a question regarding how tuples a=
re buffered between (possibly<br />chained) subtasks.<br /><br />Is it corr=
ect that there is a buffer for each vertex in the DAG of subtasks?<br />Reg=
ardless of task slot sharing? If yes, then the primary optimization in<br /=
>this regard is operator chaining.<br /><br />Furthermore, how do these buf=
fers translate into overhead? Is there a send<br />thread and a receive thr=
ead per buffer, similar to Apache Storm?<br /><br />I could not find detail=
s concerning such buffers in the relevant subsection<br />under Concepts.<b=
r /><br />Thanks in advance.</blockquote></blockquote>  </body>
</html>

------=_Part_52842_1401601466.1464873960376--