hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suraj Menon" <surajsme...@apache.org>
Subject Review Request: Patch for Combiner implementation on spilling queue. Fixes for performance in spilling queue and added buffers for synchronous spills and bytebuffer streams to reuse ByteBuffer.
Date Mon, 04 Feb 2013 14:20:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9288/
-----------------------------------------------------------

Review request for hama.


Description
-------

The combiner is implemented in CombineSpilledDataProcessor. Apart from this, the patch contains
synchronous spilling buffers. Also due to an oversight, reads from spilling buffer was expensive
both in memory and CPU. It is now faster and uses less resources than previous version.


Diffs
-----

  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/Constants.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/BufferedReadStatus.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/ByteBufferInputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/ByteBufferOutputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/CombineSpilledDataProcessor.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/DirectByteBufferInputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/DirectByteBufferOutputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/DualChannelByteBufferStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/DuplexByteArrayChannel.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/PreFetchCache.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/ReadIndexStatus.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/ReusableByteBuffer.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpillWriteIndexStatus.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpilledByteBuffer.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpilledDataInputBuffer.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpilledDataProcessor.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpilledDataReadStatus.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SpillingDataOutputBuffer.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SyncFlushByteBufferOutputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/SyncReadByteBufferInputStream.java
PRE-CREATION 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/io/WriteSpilledDataProcessor.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/DiskQueue.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MemoryQueue.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/MessageQueue.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SortedMessageQueue.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/bsp/message/queue/SpillingQueue.java
1441692 
  http://svn.apache.org/repos/asf/hama/trunk/core/src/test/java/org/apache/hama/bsp/message/TestMessageIO.java
1441692 

Diff: https://reviews.apache.org/r/9288/diff/


Testing
-------

Added unit test cases in TestMessageIO and passes mvn install.


Thanks,

Suraj Menon


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message