flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Memory ran out. Compaction failed. - Exception
Date Thu, 01 Jun 2017 15:54:30 GMT
Hi Marc,

The CompactingHashtable is not spillable (the only operator in batch
actually that isn't), so you can only either reduce your data size or
increase your memory.
When you are using unmanaged memory, make sure the allocation of managed
memory is reduced to a minimum.

On Mon, May 29, 2017 at 9:29 AM, Kaepke, Marc <marc.kaepke@haw-hamburg.de>
wrote:

> Hi,
>
> I tried to run an vertex-centric and a gather scatter iteration in Gelly
> and receive an exception like
>
> Exception in thread "main" org.apache.flink.runtime.client.JobExecutionException:
> Job execution failed.
>         at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$
> mcV$sp(JobManager.scala:900)
>         at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843)
>         at org.apache.flink.runtime.jobmanager.JobManager$$
> anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:843)
>         at scala.concurrent.impl.Future$PromiseCompletingRunnable.
> liftedTree1$1(Future.scala:24)
>         at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(
> Future.scala:24)
>         at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>         at akka.dispatch.ForkJoinExecutorConfigurator$
> AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>         at scala.concurrent.forkjoin.ForkJoinTask.doExec(
> ForkJoinTask.java:260)
>         at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.
> runTask(ForkJoinPool.java:1339)
>         at scala.concurrent.forkjoin.ForkJoinPool.runWorker(
> ForkJoinPool.java:1979)
>         at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: Memory ran out. Compaction failed.
> numPartitions: 32 minPartition: 37 maxPartition: 96 number of overflow
> segments: 0 bucketSize: 476 Overall memory: 85688320 Partition memory:
> 67043328 Message: null
>         at org.apache.flink.runtime.operators.hash.CompactingHashTable.
> insertRecordIntoPartition(CompactingHashTable.java:456)
>         at org.apache.flink.runtime.operators.hash.CompactingHashTable.
> insertOrReplaceRecord(CompactingHashTable.java:405)
>         at org.apache.flink.runtime.operators.hash.CompactingHashTable.
> buildTableWithUniqueKey(CompactingHashTable.java:316)
>         at org.apache.flink.runtime.iterative.task.IterationHeadTask.
> readInitialSolutionSet(IterationHeadTask.java:210)
>         at org.apache.flink.runtime.iterative.task.IterationHeadTask.run(
> IterationHeadTask.java:271)
>         at org.apache.flink.runtime.operators.BatchTask.invoke(
> BatchTask.java:355)
>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:655)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> parameter.setSolutionSetUnmanagedMemory(true); doesn’t  work for me.
>
>
> Please help.
>
>
> Best,
> Marc

Mime
View raw message