flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: [GitHub] incubator-flink pull request: Do not limit the client's heap space...
Date Fri, 13 Jun 2014 19:52:53 GMT
Thats good. We have the same problem on the receiver side. I assume the new
RPC service is not transferring the user-jar by first allocating it as a
whole in memory.


On Fri, Jun 13, 2014 at 9:48 PM, Ufuk Celebi <u.celebi@fu-berlin.de> wrote:

> +1
>
> But imo this will soon be subsumed by the upcoming RPC rework.
>
> Sent from my iPhone
>
> > On 13 Jun 2014, at 21:44, rmetzger <git@git.apache.org> wrote:
> >
> > GitHub user rmetzger opened a pull request:
> >
> >    https://github.com/apache/incubator-flink/pull/18
> >
> >    Do not limit the client's heap space (See FLINK-934)
> >
> >    Similar to  FLINK-934 and
> https://github.com/apache/incubator-flink/pull/14 we should not limit the
> heap space of the job submission client to 512MB.
> >
> >    A user actually had the following exception while using Flink:
> >    ```
> >    java.lang.OutOfMemoryError: Java heap space
> >        at
> eu.stratosphere.runtime.io.serialization.DataOutputSerializer.resize(DataOutputSerializer.java:243)
> >        at
> eu.stratosphere.runtime.io.serialization.DataOutputSerializer.write(DataOutputSerializer.java:87)
> >        at
> eu.stratosphere.nephele.jobgraph.JobGraph.writeRequiredJarFiles(JobGraph.java:706)
> >        at
> eu.stratosphere.nephele.jobgraph.JobGraph.write(JobGraph.java:641)
> >        at eu.stratosphere.nephele.ipc.RPC$Invocation.write(RPC.java:153)
> >        at
> eu.stratosphere.nephele.ipc.Client$Connection.sendParam(Client.java:469)
> >        at eu.stratosphere.nephele.ipc.Client.call(Client.java:689)
> >        at eu.stratosphere.nephele.ipc.RPC$Invoker.invoke(RPC.java:250)
> >        at com.sun.proxy.$Proxy1.submitJob(Unknown Source)
> >        at
> eu.stratosphere.nephele.client.JobClient.submitJobAndWait(JobClient.java:258)
> >        at eu.stratosphere.client.program.Client.run(Client.java:284)
> >        at eu.stratosphere.client.program.Client.run(Client.java:268)
> >        at eu.stratosphere.client.program.Client.run(Client.java:262)
> >        at
> eu.stratosphere.client.program.ContextEnvironment.execute(ContextEnvironment.java:50)
> >        at
> eu.stratosphere.example.java.wordcount.WordCount.main(WordCount.java:79)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >        at java.lang.reflect.Method.invoke(Method.java:483)
> >        at
> eu.stratosphere.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:384)
> >        at
> eu.stratosphere.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:302)
> >        at eu.stratosphere.client.program.Client.run(Client.java:220)
> >        at
> eu.stratosphere.client.CliFrontend.executeProgram(CliFrontend.java:327)
> >        at eu.stratosphere.client.CliFrontend.run(CliFrontend.java:314)
> >        at
> eu.stratosphere.client.CliFrontend.parseParameters(CliFrontend.java:927)
> >        at eu.stratosphere.client.CliFrontend.main(CliFrontend.java:951)
> >    ```
> >
> >    I verified that the change resolves the problem.
> >
> > You can merge this pull request into a Git repository by running:
> >
> >    $ git pull https://github.com/rmetzger/incubator-flink
> stratosphere_client_fix
> >
> > Alternatively you can review and apply these changes as the patch at:
> >
> >    https://github.com/apache/incubator-flink/pull/18.patch
> >
> > To close this pull request, make a commit to your master/trunk branch
> > with (at least) the following in the commit message:
> >
> >    This closes #18
> >
> > ----
> > commit 8e1bff41a21d75f5e7dc78fbc5c30835a3802781
> > Author: Robert Metzger <rmetzger@apache.org>
> > Date:   2014-06-13T19:40:34Z
> >
> >    Do not limit the client's heap space (See FLINK-934) [ci skip]
> >
> > ----
> >
> >
> > ---
> > If your project is set up for it, you can reply to this email and have
> your
> > reply appear on GitHub as well. If your project does not have this
> feature
> > enabled and wishes so, or if the feature is enabled but not working,
> please
> > contact infrastructure at infrastructure@apache.org or file a JIRA
> ticket
> > with INFRA.
> > ---
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message