flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-941) Possible deadlock after increasing my data set size
Date Wed, 18 Jun 2014 13:08:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14035670#comment-14035670

Stephan Ewen commented on FLINK-941:

I was able to repdroduce the problem, thanks to [~uce].

The issue is that Broadcast variables are not 

I think the correct way to solve that is to make sure that we never allow backpressure on
intermediate results that are consumed by multiple targets. That does depend on the redesigned
handling of intermediate result partitions (which [~uce] has drafted).

> Possible deadlock after increasing my data set size
> ---------------------------------------------------
>                 Key: FLINK-941
>                 URL: https://issues.apache.org/jira/browse/FLINK-941
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: pre-apache-0.5.1
>            Reporter: Bastian Köcher
>            Assignee: Stephan Ewen
>         Attachments: IMPRO-3.SS14.G03.zip
> If I increase my data set, my algorithm stops at some point and doesn't continue anymore.
I already waited a quite amount of time, but nothing happens. The linux processor explorer
also displays that the process is sleeping and waiting for something to happen, could maybe
be a deadlock.
> I attached the source of my program, the class HAC_2 is the actual algorithm.
> Changing the line 271 from "if(Integer.parseInt(tokens[0]) > 282)" to "if(Integer.parseInt(tokens[0])
> 283)" at my PC "enables" the bug. The numbers 282, 283 are the numbers of the documents
in my test data and this line skips all documents with an id greater than that.

This message was sent by Atlassian JIRA

View raw message