flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andra Lungu <lungu.an...@gmail.com>
Subject Re: Memory segment error when migrating functional code from Flink 0.9 to 0.8
Date Mon, 09 Feb 2015 20:32:34 GMT
Hi Stephan,

I tried the solution with DeltaIteration#setSolutionSetUnManaged(),
unfortunatelly the error is still there... Even when I try to run it with
just one iteration... Also, I am not sure that the job can be broken into
subparts in this particular case.

Any other suggestions would be appreciated :)
Thanks!
Andra

On Mon, Feb 9, 2015 at 10:36 AM, Stephan Ewen <sewen@apache.org> wrote:

> This is actually a problem of the number of memory segments available to
> the hash table for the solution set.
>
> For complex pipelines, memory currently gets too fragmented.
>
> There are two workarounds, until we do the dynamic memory management, or
> break it into shorter pipelines: Break the job up into subparts, or move
> the solution set to the user code part of the heap. There is a flag to
> attach to the Delta Iterations, see
> "DeltaIteration#setSolutionSetUnManaged()"
>
> Greetinigs,
> Stephan
>
>
> On Mon, Feb 9, 2015 at 10:32 AM, Till Rohrmann <trohrmann@apache.org>
> wrote:
>
> > Hi Andra,
> >
> > have you tried increasing the number of network buffers in your cluster?
> > You can control by the configuration value:
> >
> > taskmanager.network.numberOfBuffers: #numberBuffers
> >
> > Greets,
> >
> > Till
> >
> > On Mon, Feb 9, 2015 at 9:56 AM, Andra Lungu <lungu.andra@gmail.com>
> wrote:
> >
> > > Hello everyone,
> > >
> > > I am implementing a graph algorithm as part of a course and I will also
> > add
> > > it to the Flink- Gelly examples.
> > > My problem is that I started developing it in the Gelly repository,
> which
> > > runs on flink 0.9. It works like a charm there, but in order to test in
> > on
> > > a cluster to see its real capabilities, I need to move it to the course
> > > repository, which runs on flink 0.8.
> > >
> > > Initially, I thought this migration should occur without incidents
> since
> > > flink 0.8 is more stable. Instead, I got the following exception:
> > > java.lang.IllegalArgumentException: *Too few memory segments provided.
> > Hash
> > > Table needs at least 33 memory segments.*
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:238)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.hash.CompactingHashTable.<init>(CompactingHashTable.java:227)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.initCompactingHashTable(IterationHeadPactTask.java:177)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:279)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
> > >     at
> > >
> > >
> >
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:257)
> > >     at java.lang.Thread.run(Thread.java:745)
> > >
> > > This is the code for Gelly, where all tests pass:
> > > https://github.com/andralungu/flink-graph/tree/minSpanningTree
> > > Unfortunately, the code for the course is private, so you cannot
> actually
> > > see it... maybe @aalexandrov can do something about the privacy
> settings
> > of
> > > this repo.
> > > https://github.com/andralungu/IMPRO-3.WS14/tree/dmst_algorithm
> > >
> > > Thank you!
> > > Andra
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message