giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar Athivarapu" <pava...@outlook.com>
Subject Review Request 21006: GIRAPH-892 [decode from bytebuf directly]
Date Fri, 02 May 2014 02:13:04 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21006/
-----------------------------------------------------------

Review request for giraph, Avery Ching and Maja Kabiljo.


Repository: giraph-git


Description
-------

Currently writableRequests are deserialized from network buffers into a bytearray based datastructure
(vertexIdData - ByteArrayVertexIdEdges, ByteArrayVertexIdMessages, etc.) and then immediately
stored into MessageStore, EdgeStore, etc. 
In this diff we ensure that we can directly read from network buffers (while storing in MessageStore
for example) without creating an (unnecessary intermediate ByteArray based) vertexIdData object.


Diffs
-----

  giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java 8350a55 
  giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 24848db 
  giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java 54234c5 
  giraph-core/src/main/java/org/apache/giraph/comm/SendVertexIdDataCache.java afce3ba 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java
2381078 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStore.java 2af7642 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/MessagesIterable.java 3b22ab3

  giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java
acf68ea 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/out_of_core/DiskBackedMessageStore.java
346e3b3 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntByteArrayMessageStore.java
cc14c6d 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntFloatMessageStore.java
3318610 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongByteArrayMessageStore.java
9e4325f 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongDoubleMessageStore.java
76d9ffa 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/InboundByteCounter.java bcc888d 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
0166713 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java 98a61e6

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java d379eda

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java
601cd2f 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerDataRequest.java 4f80224

  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java 793768a

  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
3ac0962 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java 181e681 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 2862c3e 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java e257b4a 
  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
2e8c935 
  giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java dd8f2a3 
  giraph-core/src/main/java/org/apache/giraph/utils/AbstractVertexIdData.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterable.java d14172e 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterator.java 28b2dc8 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 5c56038 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java 762802b 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java 0ac8fdf

  giraph-core/src/main/java/org/apache/giraph/utils/ByteBufDataInput.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteBufDataOutput.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdData.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdMessages.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterable.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterator.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataInput.java 0ecea77

  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java 0ff366d

  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteBufDataInput.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteBufDataOutput.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedDataOutput.java 54ef514 
  giraph-core/src/main/java/org/apache/giraph/utils/Extractable.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterable.java 2c24e89

  giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterator.java d36c94f

  giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterable.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterator.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/RequestUtils.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeArrayReads.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeBufferReads.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java 20ed92b

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java 4b413da

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteBufDataOutput.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeDirectByteBufDataInput.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeHeapByteBufDataInput.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeReads.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteArrayMessageWrite.java 8673732

  giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteStructMessageWrite.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdData.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdDataIterator.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdgeIterator.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java bad11d6 
  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageBytesIterator.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageIterator.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessages.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 236bc88 
  giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java fcdfa5c 
  giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java 97e88f8 

Diff: https://reviews.apache.org/r/21006/diff/


Testing
-------

ran test jobs on cluster with option giraph.requestUtils.receiveBufs=false as well as giraph.requestUtils.receiveBufs=true
mvn clean verify with hadoop_1 and hadoop_2 profiles for giraph-core


Thanks,

Pavan Kumar Athivarapu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message