flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Memory management issue
Date Tue, 08 Sep 2015 10:43:51 GMT

> On 08 Sep 2015, at 10:12, Schueler, Ricarda <Ricarda.Schueler@student.hpi.uni-potsdam.de>
wrote:
> 
> Hi,
> 
> we tested it with the version 0.9.1, but unfortunately the issue persists.

Thanks for helping me out debugging this Ricarda! :)

From what I can tell, this is not a deadlock in the network runtime, but a join deadlock within
an iteration.

https://gist.github.com/uce/3fd5ca45383402ed1b16

@Stephan, Fabian: What’s the best way to fix this for good?

@Ricarda: you can work your way around this by providing JoinHint.REPARTITION_SORT_MERGE as
a join hint in the bulk iteration, i.e.

joinedtriangles = joinedtriangles.join(graph, JoinHint.REPARTITION_SORT_MERGE).where({triangle
=> (triangle.edge3.vertex1, triangle.edge3.vertex2)}).equalTo("vertex1", "vertex2"){
  (triangle, edge) =>
    triangle.edge3.triangleCount = edge.triangleCount
    triangle
}.name("third triangle edge join”)

I saw that you were benchmarking this for a project. This should impact the runtime of your
program, so you might need to re-run the experiments.

– Ufuk


Mime
View raw message