flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator
Date Fri, 30 Jun 2017 13:35:15 GMT
Yes, in the end the requests to HBase are the bottle neck and the latency will manifest in
different places of the job depending on where there is a queue. If there is a queue between
map and flatMap elements will sit there and wait and you’ll see latency there. If map and
flatMap are chained you will see the latency in the form of Kafka consumer lag (Kafka itself
is the queue here).

> On 29. Jun 2017, at 18:30, sohimankotia <sohimankotia@gmail.com> wrote:
> Few last doubts :
> 1. So If I increase parallelism latency will decrease because load will get
> distributed  ?
> 2. But if load will increase latency will also increase if parallelism is
> more ?
> 3. Let's say If I remove partitioner , and Hbase Op is still there in Flat
> map . Then also this latency would be there ?
> 4. If yes in point 3 , then latency would from reading elements from kafka ?
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913p14076.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message