giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-249) Move part of the graph out-of-core when memory is low
Date Mon, 16 Jul 2012 16:55:34 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415377#comment-13415377
] 

Eli Reisman commented on GIRAPH-249:
------------------------------------

I just ran the latest 249 patch with 21% of the data I was able to run just using trunk +
patch 246, 256, and 250 this weekend. It died in the first couple minutes on input super step
with this listing on the failed workers:

Jul 16, 2012 4:48:52 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.HashMap.newKeyIterator(HashMap.java:840)
	at java.util.HashMap$KeySet.iterator(HashMap.java:874)
	at java.util.HashSet.iterator(HashSet.java:153)
	at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
	at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:33)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:157)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Jul 16, 2012 4:49:14 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
Jul 16, 2012 4:48:59 PM org.jboss.netty.channel.socket.nio.NioWorker
WARNING: Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3209)
	at java.lang.String.<init>(String.java:215)
	at java.lang.StringBuilder.toString(StringBuilder.java:430)
	at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:232)
	at sun.net.www.protocol.file.Handler.parseURL(Handler.java:50)
	at java.net.URL.<init>(URL.java:596)
	at java.net.URL.<init>(URL.java:464)
	at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:977)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:169)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:194)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	at org.jboss.netty.channel.AdaptiveReceiveBufferSizePredictorFactory.getPredictor(AdaptiveReceiveBufferSizePredictorFactory.java:65)
	at org.jboss.netty.channel.socket.nio.DefaultNioSocketChannelConfig.getReceiveBufferSizePredictor(DefaultNioSocketChannelConfig.java:148)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:305)
	at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:274)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:194)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

I will run a couple more tests with less data/smaller graph.splitmb to see where the line
is, and examine the Graphite metrics too.

                
> Move part of the graph out-of-core when memory is low
> -----------------------------------------------------
>
>                 Key: GIRAPH-249
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-249
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch, GIRAPH-249.patch,
GIRAPH-249.patch
>
>
> There has been some talk about Giraph's scaling limitations due to keeping the whole
graph and messages in RAM.
> We need to investigate methods to fall back to disk when running out of memory, while
gracefully degrading performance.
> This issue is for graph storage. Messages should probably be a separate issue, although
the interplay between the two is crucial.
> We should also discuss what are our primary goals here: completing a job (albeit slowly)
instead of failing when the graph is too big, while still encouraging memory optimizations
and high-memory clusters; or restructuring Giraph to be as efficient as possible in disk mode,
making it almost a standard way of operating.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message