flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Memory management issue
Date Tue, 08 Sep 2015 14:57:30 GMT
A quickfix would be to take the first join and give it a
"JoinHint.REPARTITION_HASH_BUILD_SECOND" hint.


The best thing would be to have batch exchanges for iterations.


The second best thing would be to recognize in the optimizer that a batch
exchange cannot happen (if inside an iteration) and instead set the
receiver task to break the pipeline (set TempMode.makePipelineBreaker())


On Tue, Sep 8, 2015 at 12:43 PM, Ufuk Celebi <uce@apache.org> wrote:

>
> > On 08 Sep 2015, at 10:12, Schueler, Ricarda <
> Ricarda.Schueler@student.hpi.uni-potsdam.de> wrote:
> >
> > Hi,
> >
> > we tested it with the version 0.9.1, but unfortunately the issue
> persists.
>
> Thanks for helping me out debugging this Ricarda! :)
>
> From what I can tell, this is not a deadlock in the network runtime, but a
> join deadlock within an iteration.
>
> https://gist.github.com/uce/3fd5ca45383402ed1b16
>
> @Stephan, Fabian: What’s the best way to fix this for good?
>
> @Ricarda: you can work your way around this by providing
> JoinHint.REPARTITION_SORT_MERGE as a join hint in the bulk iteration, i.e.
>
> joinedtriangles = joinedtriangles.join(graph,
> JoinHint.REPARTITION_SORT_MERGE).where({triangle =>
> (triangle.edge3.vertex1, triangle.edge3.vertex2)}).equalTo("vertex1",
> "vertex2"){
>   (triangle, edge) =>
>     triangle.edge3.triangleCount = edge.triangleCount
>     triangle
> }.name("third triangle edge join”)
>
> I saw that you were benchmarking this for a project. This should impact
> the runtime of your program, so you might need to re-run the experiments.
>
> – Ufuk
>
>

Mime
View raw message