flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator
Date Wed, 28 Jun 2017 14:45:32 GMT
I see, what I consider highly likely here is that the lookup to HBase is the bottleneck. If
the lookup takes to long events “sit in a queue” between the map and flatMap operations.
If you replace the HBase lookup by some dummy code you should see the latency go away.

The reason you don’t see latency when you don’t have a custom partitioner is that here
the map and flatMap are chained together: sending an event from one operator to the next is
basically just a function call and there is therefore no queue that can be filled that makes
events “wait”.

Best,
Aljoscha

> On 28. Jun 2017, at 15:17, sohimankotia <sohimankotia@gmail.com> wrote:
> 
> Source is KafKa .
> FlatMap has HBase Lookup
> Sink is Kafka .
> 
> I tried to get stats over the days . I see that almost 40 % were having
> latency of 0 seconds , 10 % 0-30 sec, approx 10% 30-60 sec and 10 % around
> 60 - 120 sec and 30 % around 120 - 210 secs .
> 
> 
> 
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913p14036.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.


Mime
View raw message