flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Patil <vinay18.pa...@gmail.com>
Subject Re: [Discussion] Query Regarding Operator chaining
Date Mon, 04 Jul 2016 09:13:48 GMT
Just an update, the task will be executed by multiple threads , my bad I
asked the wrong way.
Can you please clarify other things.

Out of 8 node only 3 of them are getting utilized, reading the data from
Kafka , does it mean that the Kafka partitions are set to less number ?

What if we use rescale or rebalance since it evenly distributes , would
that ensure maximum use of resources ?

Regards,
Vinay Patil

On Fri, Jul 1, 2016 at 11:09 PM, Vinay Patil <vinay18.patil@gmail.com>
wrote:

> Hi,
>
> According to the documentation :
> *"**Each task is executed by one thread ,**Chaining operators together
> into tasks is a useful optimization: it reduces the overhead of
> thread-to-thread handover and buffering, and increases overall throughput
> while decreasing latency"*
> So does it mean that the single box (refer below mails) represent it as a *single
> task* and  the task will be executed by single thread only ?
>
> I am having 8 node cluster (parallelism set to 56), so what is the correct
> way to achieve maximum CPU utilization and parallelism ? Does complete
> stream chaining into a single box achieve maximum parallelism ?
>
> The data we are processing is huge volume of data (60,000 records per
> second), so wanted to be sure what we can correct to achieve better
> results.
>
> Regards,
> Vinay Patil
>
>
> On Fri, Jul 1, 2016 at 9:23 PM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>
>> Hi,
>> yes, the window operator is stateful, which means that it will pick up
>> where it left in case of a failure and restore.
>>
>> You're right about the graph, chained operators are shown as one box.
>>
>> Cheers,
>> Aljoscha
>>
>> On Fri, 1 Jul 2016 at 04:52 Vinay Patil <vinay18.patil@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Just watched the video on Robust Stream Processing .
>> > So when we say Window is a stateful operator , does it mean that even if
>> > the task manager doing the window operation fails,  will it pick up from
>> > the state left earlier when it comes up ? (Have not read more on state
>> for
>> > now)
>> >
>> >
>> > Also in one of our project when we deploy on cluster and check the Job
>> > Graph , everything is shown in one box , why this happens ? Is it
>> because
>> > of chaining of streams ?
>> > So the box here represent the function flow, right ?
>> >
>> >
>> >
>> > Regards,
>> > Vinay Patil
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message