Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@giraph.apache.org
Date: Wed, 18 Jul 2012 12:58:34 +0000 (UTC)
From: "Alessandro Presta (JIRA)" <jira@apache.org>
To: giraph-dev@incubator.apache.org
Message-ID: <2111468212.68768.1342616314363.JavaMail.jiratomcat@issues-vm>
In-Reply-To: <714788634.41608.1342099594929.JavaMail.jiratomcat@issues-vm>
Subject: [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core
 when memory is low
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417050#comment-13417050 ] 

Alessandro Presta commented on GIRAPH-249:
------------------------------------------

Got it, thanks.
I don't know, it sounds reasonable to have quadratic partitions in the number of workers.
I guess 10% is ok as a starting point (assuming an ideal load distribution, it would be more than the average partition in a 10-worker job), and then we can finely tune depending on the job. It might turn out that the ideal ratio depends on the number of workers.

Regarding the empty lists, are you referring to inputSplitCache or to the final workerPartitionMap?
                
> Move part of the graph out-of-core when memory is low
> -----------------------------------------------------
>
>                 Key: GIRAPH-249
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-249
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch
>
>
> There has been some talk about Giraph's scaling limitations due to keeping the whole graph and messages in RAM.
> We need to investigate methods to fall back to disk when running out of memory, while gracefully degrading performance.
> This issue is for graph storage. Messages should probably be a separate issue, although the interplay between the two is crucial.
> We should also discuss what are our primary goals here: completing a job (albeit slowly) instead of failing when the graph is too big, while still encouraging memory optimizations and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode, making it almost a standard way of operating.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira