flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Memory exception
Date Sun, 10 May 2015 17:45:27 GMT
Hi Michele,

Flink manages its own memory using the MemoryManager. When a Flink worker
(TaskManager) starts, the MemoryManager allocates a certain share of the
JVM's free memory as MemorySegments: Default size of a MemorySegment is
32KB. When a Flink program is executed, the MemoryManager hands a certain
number of MemorySegments to each processing operator such as HashJoin or
Sort. These operators have a lower bound of MemorySegments which they need
for operations, e.g., 33 MemorySegments for a HashJoin.

To solve your problem, you should try to give more memory to the Flink JVM.
You can also configure a fixed amount of memory for the MemoryManager via
the taskmanager.memory.size configuration parameter [1].

Cheers, Fabian

[1]
http://ci.apache.org/projects/flink/flink-docs-master/setup/config.html#jobmanager-amp-taskmanager


2015-05-10 19:18 GMT+02:00 Michele Bertoni <michele1.bertoni@mail.polimi.it>
:

> Hi everybody,
> I am running a flink instance on my IDE
> sometimes (totally random) I start it i getting this error
>
> Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>         at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:306)
>         at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>         at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>         at
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>         at
> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:37)
>         at
> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:30)
>         at
> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>         at
> org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:30)
>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>         at
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:91)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>         at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>         at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>         at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>         at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.Exception: The data preparation for task 'CHAIN
> Join(Join at
> org.apache.flink.api.scala.UnfinishedJoinOperation.finish(joinDataSet.scala:224))
> -> Map (Map at
> LowLevel.FlinkImplementation.metaOperator.SemiJoinMD$.apply(SemiJoinMD.scala:33))
> -> Combine(Distinct at
> LowLevel.FlinkImplementation.metaOperator.SemiJoinMD$.apply(SemiJoinMD.scala:34))'
> , caused an error: Too few memory segments provided. Hash Join needs at
> least 33 memory segments.
>         at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472)
>         at
> org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:362)
>         at
> org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:217)
>         at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.IllegalArgumentException: Too few memory segments
> provided. Hash Join needs at least 33 memory segments.
>         at
> org.apache.flink.runtime.operators.hash.MutableHashTable.<init>(MutableHashTable.java:373)
>         at
> org.apache.flink.runtime.operators.hash.MutableHashTable.<init>(MutableHashTable.java:359)
>         at
> org.apache.flink.runtime.operators.hash.HashMatchIteratorBase.getHashJoin(HashMatchIteratorBase.java:48)
>         at
> org.apache.flink.runtime.operators.hash.NonReusingBuildFirstHashMatchIterator.<init>(NonReusingBuildFirstHashMatchIterator.java:78)
>         at
> org.apache.flink.runtime.operators.MatchDriver.prepare(MatchDriver.java:148)
>         at
> org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466)
>         ... 3 more
>
>
>
>
> when this happens to solve it i need either to restart the computer or run
> it from the downloaded flink instance instead of the IDE
>
> sometime the error is different: it says something like “you must start
> the memory manager with a positive value now is 0MB” but also this one of
> 33 memory segment is quite common
>
> it is quite annoying to restart everything and it happens quite often
>
> do you have any idea why it happens?
>
>
>
> thank you

Mime
View raw message