giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar Athivarapu" <pava...@outlook.com>
Subject Re: Review Request 23340: Decouple netty server threads from message processing
Date Fri, 18 Jul 2014 23:13:52 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23340/#review48166
-----------------------------------------------------------

Ship it!


Ship It!

- Pavan Kumar Athivarapu


On July 18, 2014, 9:30 p.m., Sergey Edunov wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23340/
> -----------------------------------------------------------
> 
> (Updated July 18, 2014, 9:30 p.m.)
> 
> 
> Review request for giraph.
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> Our profiling shows that a lot of apps are neither CPU nor memory or network bound. Instead
they waste a lot of time waiting for lock in MessageStore. That happens in netty threads.

> We should be able to put messages into queue and then process them in other set of threads.

> It has to be configurable because adding another thread level will introduce additional
overhead.
> 
> I introduced new options: 
> giraph.async.message.store (false by default) that will enable async messaging
> giraph.async.message.store.threads (8 by default) that will configure number of background
threads required to process messages.
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java 036510e 
>   giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java
db22503 
>   giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapper.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/PartitionMessage.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/comm/messages/queue/package-info.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 3d16e9c 
>   giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java b2a5c84 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 0d90a59 
>   giraph-core/src/test/java/org/apache/giraph/comm/messages/queue/AsyncMessageStoreWrapperTest.java
PRE-CREATION 
>   giraph-examples/src/test/java/org/apache/giraph/TestCheckpointing.java 387b937 
> 
> Diff: https://reviews.apache.org/r/23340/diff/
> 
> 
> Testing
> -------
> 
> I run pagerank and it gives ~7% improvement over and along with G1 GC it gives ~15% improvement.
And CPU usage is now close to 90% 
> 
> 
> Thanks,
> 
> Sergey Edunov
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message