Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Mon, 19 Sep 2016 03:54:20 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12935994.1454426794000.601665.1474257260585@Atlassian.JIRA>
In-Reply-To: <JIRA.12935994.1454426794000@Atlassian.JIRA>
References: <JIRA.12935994.1454426794000@Atlassian.JIRA> <JIRA.12935994.1454426794238@arcas>
Subject: [jira] [Commented] (FLINK-3322) MemoryManager creates too much GC
 pressure with iterative jobs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 19 Sep 2016 03:54:22 -0000


    [ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15502196#comment-15502196 ] 

ASF GitHub Bot commented on FLINK-3322:
---------------------------------------

Github user ramkrish86 commented on the issue:

    https://github.com/apache/flink/pull/2495
  
    > SingleSourceShortestPathsITCase
    
    Sure. Actually I ran mvn clean verify to ensure there are no code style issues in runtime package and ran tests in that package. I was not sure what other tests are impacted because of this change and also tried running all the example related test cases like PageRank, KMeans etc. 


> MemoryManager creates too much GC pressure with iterative jobs
> --------------------------------------------------------------
>
>                 Key: FLINK-3322
>                 URL: https://issues.apache.org/jira/browse/FLINK-3322
>             Project: Flink
>          Issue Type: Bug
>          Components: Local Runtime
>    Affects Versions: 1.0.0
>            Reporter: Gabor Gevay
>            Assignee: ramkrishna.s.vasudevan
>            Priority: Critical
>             Fix For: 1.0.0
>
>         Attachments: FLINK-3322.docx, FLINK-3322_reusingmemoryfordrivers.docx
>
>
> When taskmanager.memory.preallocate is false (the default), released memory segments are not added to a pool, but the GC is expected to take care of them. This puts too much pressure on the GC with iterative jobs, where the operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. (It will generate some lookuptables to /tmp on first run for a few minutes.) (I think the slowdown might also depend somewhat on taskmanager.memory.fraction, because more unused non-managed memory results in rarer GCs.)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)