flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Waury <robert.wa...@googlemail.com>
Subject Re: RuntimeException Gelly API: Memory ran out. Compaction failed.
Date Wed, 18 Mar 2015 14:40:31 GMT
Hi,

I managed to reproduce the behavior and as far as I can tell it seems to be
a problem with the memory allocation.

I have filed a bug report in JIRA to get the attention of somebody who
knows the runtime better than I do.

https://issues.apache.org/jira/browse/FLINK-1734

Cheers,
Robert

On Tue, Mar 17, 2015 at 3:52 PM, Mihail Vieru <vieru@informatik.hu-berlin.de
> wrote:

>  Hi Robert,
>
> thank you for your reply.
>
> I'm starting the job from the Scala IDE. So only one JobManager and one
> TaskManager in the same JVM.
> I've doubled the memory in the eclipse.ini settings but I still get the
> Exception.
>
> -vmargs
> -Xmx2048m
> -Xms100m
> -XX:MaxPermSize=512m
>
> Best,
> Mihail
>
>
> On 17.03.2015 10:11, Robert Waury wrote:
>
>   Hi,
>
>  can you tell me how much memory your job has and how many workers you are
> running?
>
>  From the trace it seems the internal hash table allocated only 7 MB for
> the graph data and therefore runs out of memory pretty quickly.
>
>  Skewed data could also be an issue but with a minimum of 5 pages and a
> maximum of 8 it seems to be distributed fairly even to the different
> partitions.
>
>  Cheers,
>  Robert
>
> On Tue, Mar 17, 2015 at 1:25 AM, Mihail Vieru <
> vieru@informatik.hu-berlin.de> wrote:
>
>> And the correct SSSPUnweighted attached.
>>
>>
>> On 17.03.2015 01:23, Mihail Vieru wrote:
>>
>>> Hi,
>>>
>>> I'm getting the following RuntimeException for an adaptation of the
>>> SingleSourceShortestPaths example using the Gelly API (see attachment).
>>> It's been adapted for unweighted graphs having vertices with Long values.
>>>
>>> As an input graph I'm using the social network graph (~200MB unpacked)
>>> from here: https://snap.stanford.edu/data/higgs-twitter.html
>>>
>>> For the small SSSPDataUnweighted graph (also attached) it terminates and
>>> computes the distances correctly.
>>>
>>>
>>> 03/16/2015 17:18:23    IterationHead(WorksetIteration (Vertex-centric
>>> iteration
>>> (org.apache.flink.graph.library.SingleSourceShortestPathsUnweighted$VertexDistanceUpdater@dca6fe4
>>> |
>>> org.apache.flink.graph.library.SingleSourceShortestPathsUnweighted$MinDistanceMessenger@6577e8ce)))(2/4)
>>> switched to FAILED
>>> java.lang.RuntimeException: Memory ran out. Compaction failed.
>>> numPartitions: 32 minPartition: 5 maxPartition: 8 number of overflow
>>> segments: 176 bucketSize: 217 Overall memory: 20316160 Partition memory:
>>> 7208960 Message: Index: 8, Size: 7
>>>     at
>>> org.apache.flink.runtime.operators.hash.CompactingHashTable.insert(CompactingHashTable.java:390)
>>>     at
>>> org.apache.flink.runtime.operators.hash.CompactingHashTable.buildTable(CompactingHashTable.java:337)
>>>     at
>>> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.readInitialSolutionSet(IterationHeadPactTask.java:216)
>>>     at
>>> org.apache.flink.runtime.iterative.task.IterationHeadPactTask.run(IterationHeadPactTask.java:278)
>>>     at
>>> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
>>>     at
>>> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:205)
>>>     at java.lang.Thread.run(Thread.java:745)
>>>
>>>
>>> Best,
>>> Mihail
>>>
>>
>>
>
>

Mime
View raw message