giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching" <avery.ch...@gmail.com>
Subject Review Request: GIRAPH-389 Multithreading should intelligently allocate the thread pools
Date Sat, 27 Oct 2012 15:14:16 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7754/
-----------------------------------------------------------

Review request for giraph.


Description
-------

Do not create more input split loading threads than input splits to allow workers to equally
load up input splits
Do not create more compute threads than partitions on the worker
Removed duplicate reserveInputSplit method in BspServiceWorker
Minor optimization to only get the input split zknodes once and pass to all threads.


This addresses bug GIRAPH-389.
    https://issues.apache.org/jira/browse/GIRAPH-389


Diffs
-----

  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
1402363 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java
1402363 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/InputSplitPathOrganizer.java
1402363 
  http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/InputSplitsCallable.java
1402363 

Diff: https://reviews.apache.org/r/7754/diff/


Testing
-------

passed unittests
ran pagerankbenchmark on a real cluster and observed that the limiting works


Thanks,

Avery Ching


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message