Mailing-List: contact user-help@storm.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@storm.apache.org
MIME-Version: 1.0
From: Arnaud BOS <arnaud.tlse@gmail.com>
Date: Tue, 14 Nov 2017 15:51:09 +0000
Message-ID: <CADD2gUsnALhKvp_HLvBFrvZOTTeybnyDBjHeXK26OyinVb6TcQ@mail.gmail.com>
Subject: Parallelism, field grouping and windowing
To: "user@storm.apache.org" <user@storm.apache.org>
Content-Type: multipart/alternative; boundary="f403043605c6b3c825055df35b2e"
archived-at: Tue, 14 Nov 2017 15:51:26 -0000

--f403043605c6b3c825055df35b2e
Content-Type: text/plain; charset="UTF-8"

Hi all,

Suppose I have a WindowedBolt subscribing to a KafkaSpout stream.
The WindowedBolt is a sliding window of size 2 with a sliding interval of 1.
The stream from KafkaSpout to WindowedBolt is partitioned by field (Field
Grouping).
To keep things simple let's assume the stream gets partitioned in two
groups only, A and B in the context of two supervisors with one worker
process each (setNumWorkers(2)):

By default the number of tasks of a spout/bolt is equal to the parallelism
hint, correct ?

If the WindowedBolt has a parallelism hint of 2 and 2 tasks (setNumTasks(2)
or not specified), one thread (executor) running exactly 1 task will
execute on each worker/supervisor, correct ?

Each executor/task will receive tuples from only one of the partitions, for
instance executor/task 1 gets tuples from substream A and executor/task 2
get tuples from substream B, correct ?

Assuming the latter is correct, if a node/supervisor fails, will the
remaining task begin to receive tuples from the other half of the stream
partition ? That is, if executor/task 2 (on supervisor 2) disappears, will
the tuples from A and B be interleaved in the sliding window of
executor/task 1 ? Or will the substream B just "hang" until another task is
started to handle it ?

All of this makes sense and my use case might look counterproductive but I
need to be sure of what's happening, my workflow is inherently sequential
and I don't want "streams" to interleave.

Thanks !

--f403043605c6b3c825055df35b2e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div>Hi all,<br><br></div>Suppose=
 I have a WindowedBolt subscribing to a KafkaSpout stream.<br></div>The Win=
dowedBolt is a sliding window of size 2 with a sliding interval of 1.<br></=
div>The stream from KafkaSpout to WindowedBolt is partitioned by field (Fie=
ld Grouping).</div><div>To keep things simple let&#39;s assume the stream g=
ets partitioned in two groups only, A and B in the context of two superviso=
rs with one worker process each (setNumWorkers(2)):</div><div><br></div><di=
v>By default the number of tasks of a spout/bolt is equal to the parallelis=
m hint, correct ?<br></div><div><br></div>If the WindowedBolt has a paralle=
lism hint of 2 and 2 tasks (setNumTasks(2) or not specified), one thread (e=
xecutor) running exactly 1 task will execute on each worker/supervisor, cor=
rect ?</div><div><br></div><div>Each executor/task will receive tuples from=
 only one of the partitions, for instance executor/task 1 gets tuples from =
substream A and executor/task 2 get tuples from substream B, correct ?<br><=
/div><div><br></div><div>Assuming the latter is correct, if a node/supervis=
or fails, will the remaining task begin to receive tuples from the other ha=
lf of the stream partition ? That is, if executor/task 2 (on supervisor 2) =
disappears, will the tuples from A and B be interleaved in the sliding wind=
ow of executor/task 1 ? Or will the substream B just &quot;hang&quot; until=
 another task is started to handle it ?<br></div><div><br></div>All of this=
 makes sense and my use case might look counterproductive but I need to be =
sure of what&#39;s happening, my workflow is inherently sequential and I do=
n&#39;t want &quot;streams&quot; to interleave.<br><br></div>Thanks !<br></=
div>

--f403043605c6b3c825055df35b2e--