flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Data point goes missing within iteration
Date Wed, 20 Jul 2016 11:42:32 GMT
@Paris Thanks for the prompt feedback! I really have to check out your PR :)

@Biblop: If I understand correctly, a possible workaround in the
meantime seems to be to use `setBufferTimeout(0)` on your
StreamExecutionEnvironment. Could you try that?

On Wed, Jul 20, 2016 at 12:30 PM, Paris Carbone <parisc@kth.se> wrote:
> This is possibly related to the way the queue between StreamIterationTail
> and Head is currently implemented.
> I think this part is a bit prone to records loss when things get wacky and
> backpressure kicks in (but at least it avoids deadlocks, right?).
> I don’t have the time availability to look into the code right now but I am
> going to focus on a progress/loops by September, hopefully with a FLIP,
> which solves that part as well.
> If  this is urgent, please go ahead and check this now, I think that queue
> timeouts cause this...
> Paris
> PS: on my yet incomplete PR (I know I know) I basically disabled queue
> polling timeouts since the checkpoint overhead on the StreamIterationHead
> almost always led to record loss.
> https://github.com/apache/flink/pull/1668
> On 20 Jul 2016, at 11:57, Maximilian Michels <mxm@apache.org> wrote:
> CC Gyula and Paris in case they might want to help out.
> On Tue, Jul 19, 2016 at 11:43 AM, Biplob Biswas
> <revolutionisme@gmail.com> wrote:
> Hi Ufuk,
> Thanks for the update, is there any known way to fix this issue? Any
> workaround that you know of, which I can try?
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Data-point-goes-missing-within-iteration-tp7776p8015.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.

View raw message