giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro Presta" <alessan...@fb.com>
Subject Re: Review Request: GIRAPH-528
Date Thu, 07 Mar 2013 05:31:27 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9732/
-----------------------------------------------------------

(Updated March 7, 2013, 5:30 a.m.)


Review request for giraph.


Description
-------

Summary of changes:
- Vertex: only one abstract Vertex class. All vertices are mutable. Removed some unused methods
that unnecessarily exposed internals. It delegates edge storage to VertexEdges. When an Iterable
is passed to setEdges(), it checks whether it's an instance of VertexEdges (just copy the
reference) or not (create and initialize VertexEdges with the Iterable). The former is the
case with edge input (i.e. EdgeStore), the latter often happens with vertex input.
- VertexEdges: interface for the out-edges data structure. Removed unused return values for
add()/remove(): we don't need them because PartitionStats are updated after each compute(),
so it's irrational to require them and then discard them.
- EdgeStore: incoming edges are aggregated in the VertexEdges implementation of choice. Moving
them to the owner vertex involves simply passing a reference.
- ImmutableClassesGiraphConfiguration: added factory methods for VertexEdges. Depending on
the needs, one can only instantiate the class, or also initialize it with either an initial
capacity or an Iterable. For example, in a VertexInputFormat, one may already have a list
of edges (call with iterable) or read the number of edges and then stream over the input (call
with capacity, then add each edge).
- Edge: removed Comparable implementation. It was only used by a test. We shouldn't expose
more functionality to the user than is needed to write Giraph applications.
- ByteArrayEdges: this is the default VertexEdges implementation, as it's generic and reasonably
efficient. I moved the representative edge inside ByteArrayEdgeIterator for better encapsulation.
The "only one iterator at a time" constraint differs from other implementations, and the gain
is negligible (we still have to instantiate an ExtendedDataOutput for each iterator creation).
- ArrayListEdges, HashMapEdges, LongDoubleArrayEdges, LongNullArrayEdges, LongDoubleHashMapEdges,
LongNullHashSetEdges: these are the other provided implementations. the first two are generic,
the second two are backed by dynamic arrays of primitives, the last two are backed by primitive
open hash maps. The iterators they return are read-only. ArrayListEdges can recognize if it's
being initialized with an ArrayList, and in that case just copies the reference (optimized
for some VertexInputFormats).
- general cleanup: removed unneeded or buggy classes like BspUtils; fixed some tests that
could only work with Java collection-backed edges (introduced the EdgeIterables helper class
to correctly compare two edge iterables); removed the plethora of Vertex implementations,
most of which were either unused or only used by a test.


This addresses bug GIRAPH-528.
    https://issues.apache.org/jira/browse/GIRAPH-528


Diffs
-----

  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/TestAccumuloVertexFormat.java
8894199503a972072fc69a49ad1faf5419aa4551 
  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java
30d40f621c028a5d01d64b0c035fa3adf374b73a 
  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeOutputFormat.java
70288f68dcb3131834703f6bd0b500d238bc32bf 
  giraph-core/pom.xml 5ce3eaa126f7f4dece5cec98f648c7da84c2646a 
  giraph-core/src/main/java/org/apache/giraph/benchmark/AggregatorsBenchmark.java 4e4704262bf7ce70c4584c8172a60e1f7a0aeb28

  giraph-core/src/main/java/org/apache/giraph/benchmark/ByteArrayVertexPageRankBenchmark.java
7e51c261c5157ad0935e1388f8a9a7a9f0f38acc 
  giraph-core/src/main/java/org/apache/giraph/benchmark/EdgeListVertexPageRankBenchmark.java
fe1e346b39bec2eb2ee29bd68c813e5a21e35b2e 
  giraph-core/src/main/java/org/apache/giraph/benchmark/HashMapVertexPageRankBenchmark.java
ef3754078a373947117acd18ee75cd6ef8cf3d92 
  giraph-core/src/main/java/org/apache/giraph/benchmark/HashMapVertexShortestPathsBenchmark.java
a9d6deb8007f107b4c1e29b0f0eaa2d400ffe20d 
  giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphByteArrayVertexPageRankBenchmark.java
9144641f3c359c28588ad489993cdb2657041b14 
  giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphEdgeListVertexPageRankBenchmark.java
712ca99f9076729270619e66cb0da83fc78e0a36 
  giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphRepresentativeVertexPageRankBenchmark.java
96288323e6028e779113d2520ea9edad497bb0e1 
  giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 06ee80cff101e3b759e19703cd2915209629d895

  giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankComputation.java 57cc201e5a01e9a9c4307a82867ceb1322697646

  giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankVertex.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java c8e33dde71019ac249ffec43e0bfdebca0920637

  giraph-core/src/main/java/org/apache/giraph/benchmark/RepresentativeVertexPageRankBenchmark.java
331ae41a2c0df6b124cbf33944b05f080b49ce94 
  giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsBenchmark.java 1843da92119cb31025a000f02b0e84a5e47dcc20

  giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsComputation.java 19f557509c38dd9a2f74fbdaad06a89d30b29ff7

  giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsVertex.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/bsp/BspOutputFormat.java 65c3ebc7974574f9814ab821d1c63237c8a49037

  giraph-core/src/main/java/org/apache/giraph/bsp/BspUtils.java 335047d506f15c6047c546df60a15481bfaa0cae

  giraph-core/src/main/java/org/apache/giraph/comm/MsgList.java 140b6e836dad8eeb099e14c994c2d40e0269acac

  giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java f239c1a6a47620c49d4dc3e71066f7629fa5b15d

  giraph-core/src/main/java/org/apache/giraph/comm/SendMutationsCache.java 9ea968a84abab1081be052692445ebb3db145dcf

  giraph-core/src/main/java/org/apache/giraph/comm/SendPartitionCache.java 911b6ba42a2f338a71aa247fc70f0f85541d9c50

  giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java 7b4baa1a7ffc74343cb959e1df23aa9ab24f7376

  giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java 43311f4e4fe7d61fd77fa91ae23280c02a4940a7

  giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java
6e6cb9b2382c702a6040691634f7f623effae9ba 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/SequentialFileMessageStore.java
c2c6141cc3efc243ee6b1a6c4aea570b542d541f 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
0fc1858f16c0302c60064c912f3549981eb756b7 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 697b6ce54afb1339c5fd52ab7add229a8afd19d0

  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java f301bbfd89397c716f47a0885b9fa085eedca742

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 5c2a01a226202eb1ceba5857f140eef4ed31ed48

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 3ea8d3be522afd5b58710654fda1e8fceeb9d207

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java fcdd57bdf7a44cf222b5d8b1e597216d92b80551

  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
e6c4cc640515fddf15cff024fa783cf7eaff890b 
  giraph-core/src/main/java/org/apache/giraph/edge/ArrayListEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/ByteArrayEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/ConfigurableVertexEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/DefaultEdge.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/Edge.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/EdgeFactory.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/EdgeNoValue.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/HashMapEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/HashMultimapEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/LongDoubleArrayEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/LongDoubleHashMapEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/LongNullArrayEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/LongNullHashSetEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/MultiRandomAccessVertexEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/MutableEdge.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/ReuseObjectsVertexEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/StrictRandomAccessVertexEdges.java PRE-CREATION

  giraph-core/src/main/java/org/apache/giraph/edge/VertexEdges.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/edge/package-info.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java c7aff7c220214ce0220000ab009a9bf8407486cc

  giraph-core/src/main/java/org/apache/giraph/graph/DefaultEdge.java 039f0d7f876800d186780d7296ed0e86601abf07

  giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexResolver.java c88b2b9cedf1c29bad31e4ffb821a4c6a953ea95

  giraph-core/src/main/java/org/apache/giraph/graph/Edge.java 185e3c357a8cc163d85c1b235e57bb9e46e74054

  giraph-core/src/main/java/org/apache/giraph/graph/EdgeFactory.java a3e6efb311091446d7baaa21f46845460815c770

  giraph-core/src/main/java/org/apache/giraph/graph/EdgeNoValue.java 4ac6759724a4ae6542071584c8638fb6c067ba59

  giraph-core/src/main/java/org/apache/giraph/graph/EdgeStore.java 621036787e95cf340e3a873e5a5be33d02ab447f

  giraph-core/src/main/java/org/apache/giraph/graph/GiraphTransferRegulator.java 726c21e35a04fb86e7177cdb5b2bdae6ed46a104

  giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java 20fa5c50ed38ba2b70b5471354093a01e5440b56

  giraph-core/src/main/java/org/apache/giraph/graph/MutableEdge.java 52e4c47f12d710edba6cb011ef20ee8119d0aa46

  giraph-core/src/main/java/org/apache/giraph/graph/ReverseEdgeDuplicator.java 4415cc219d8b977210f04cfd4d82cdc2015029e9

  giraph-core/src/main/java/org/apache/giraph/graph/Vertex.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/graph/VertexChanges.java ef61dbb47639b573eff93cca6da1c76f9facf1e2

  giraph-core/src/main/java/org/apache/giraph/graph/VertexMutations.java fa333412eed62793d87665d1630a8ce6f6faa1d6

  giraph-core/src/main/java/org/apache/giraph/graph/VertexResolver.java 4a36706f39077f0d64505138d1c677ae20b84a6a

  giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java ed6fad1619f1e71316f062bc073a0fe0bbd83404

  giraph-core/src/main/java/org/apache/giraph/io/ReverseEdgeDuplicator.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java 1b1c896eb7b0578d1e8110650e1528120f865935

  giraph-core/src/main/java/org/apache/giraph/io/VertexValueReader.java 923ca5c55bf6ac8d12b8662a36dab26e8d128a8f

  giraph-core/src/main/java/org/apache/giraph/io/VertexWriter.java 82a19bb7f834327913f2db711a2c5c2c570329e3

  giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexInputFormat.java
5092352d78520d5e27b36c3ca93cd0e0bed2a8e6 
  giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexOutputFormat.java
934663e4cfe83225d194872aeb1c72a2bf02f414 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IdWithValueTextOutputFormat.java
352f0540cd80b9f6d34ebc78a5203a71ed4e4244 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IntIntNullIntTextInputFormat.java
b00e495a2deccb756353dae7b8323714b9f54ae1 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IntNullNullNullTextInputFormat.java
dda3f2f3399728d3cf51dda6412d03aa10a744a0 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IntNullReverseTextEdgeInputFormat.java
1e3b64342adf8be4be85e0c2be5d236233c5b7cf 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexInputFormat.java
21ca427a6ad3b4afb2971f2e7c434fc383a39be8 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexOutputFormat.java
0599742ee94672f7ab06c655a630bd7fcae9bc5e 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexInputFormat.java
2df20f105f510c9577bcc26dabb2c86b6537b770 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexOutputFormat.java
9a751ae5d2a16871386a13e0fd83e65f6d050015 
  giraph-core/src/main/java/org/apache/giraph/io/formats/LongDoubleDoubleAdjacencyListVertexInputFormat.java
4e352014b2cb54d921a80a5427b503b70b945d0c 
  giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomEdgeInputFormat.java
202486380d53b9654a10bf9ea2bc9c093616418a 
  giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomVertexInputFormat.java
4da8f9dcd9aa401acb9d45bb55d3acd86dae461b 
  giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexInputFormat.java
6a5813b2ca54e6714298bb14a3ba991038dcc6e9 
  giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexOutputFormat.java
0538db972fabe2ad35c8893062d672a139af60ce 
  giraph-core/src/main/java/org/apache/giraph/io/formats/TextDoubleDoubleAdjacencyListVertexInputFormat.java
36d00dbafb7fc0e245545e050d248cb6e2f2fa2c 
  giraph-core/src/main/java/org/apache/giraph/io/formats/TextEdgeInputFormat.java c9f5df1c622243f555ccaec19e3891d8369fe9fb

  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexInputFormat.java e359f66d211f65e27793f93e2a08848d52824383

  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java 9f1fe1f353530da2b3b6cc3456b2c10836b88ce6

  giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java 16b0d4829659b53f1c2b0c11fa9987a7bff2fd1e

  giraph-core/src/main/java/org/apache/giraph/partition/ByteArrayPartition.java 2260837309b8c767af7ad310a52d0e05caad65f7

  giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java 585ab851780676a532bdfe0dce4c0e2fa5d1d317

  giraph-core/src/main/java/org/apache/giraph/partition/Partition.java 657c054de5b45f8e3a976e64aaee7842b2013452

  giraph-core/src/main/java/org/apache/giraph/partition/SimplePartition.java cbf6bc33e06b62e2a8d3d255046f619e16676ed8

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayEdges.java 1d8fc26b895a4ed6cb338cb5163ed6cd1b2e3a34

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java bd464d54c18edfd575c0df06e88720dc138157c1

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java 1cfd21e447d6a04d2eb5f1cc50fd4acad403dd4f

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java fd06783c8133c5e25d5d996a49ca3de0cf3e1be2

  giraph-core/src/main/java/org/apache/giraph/utils/ComparisonUtils.java bb940eab1be82e4e69acf45727d3858ea59bfe44

  giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java bd2a9c46eb17e7a42d59d55a5906385bc9a6d5b5

  giraph-core/src/main/java/org/apache/giraph/utils/EdgeComparator.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/EdgeIterables.java ab288fb1ccebbc1c5446149212d3d6236699fac0

  giraph-core/src/main/java/org/apache/giraph/utils/EmptyIterable.java 382c39c35b127a865c1d103b371f851c851155f7

  giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java 247130be4501f0008cdad22a199f20eda9646d64

  giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 889798fb8813fa2db005376591d07890783a5105

  giraph-core/src/main/java/org/apache/giraph/utils/ReflectionUtils.java ae2c556c865e46abb3ce049aebd709ce73d5826b

  giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableDoubleArrayIterator.java cdf662ecd9bf54374f24d4caaf01d640024d566b

  giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableIntArrayIterator.java c9ba0efa3bcf757477d59cd2958b87e6fd0fbce9

  giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongArrayIterator.java c580f9c55b44a05d4125d30669e74a166c89bd6b

  giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongFloatEdgeArrayIterable.java
b7670580939a8e51a8e2eaafc9ba72592875149b 
  giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongNullEdgeArrayIterable.java
18f280ae2bb15750b32bb79e6b2d0c7e8813418b 
  giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java da6d6cd9d42baf38b180e80af0cf66212926b171

  giraph-core/src/main/java/org/apache/giraph/utils/WritableUtils.java 129923d917269cafa9f1c1f468488d4674a23622

  giraph-core/src/main/java/org/apache/giraph/vertex/ByteArrayVertex.java 1e56b208a114a4dd2d987488df0ef32cf0211537

  giraph-core/src/main/java/org/apache/giraph/vertex/ByteArrayVertexBase.java 26c3f62faccbb08f95ae62b46b9920805dbf59f9

  giraph-core/src/main/java/org/apache/giraph/vertex/EdgeListVertex.java 882bbb8a8277d8fd8a2433d1af72e1ca559d6897

  giraph-core/src/main/java/org/apache/giraph/vertex/EdgeListVertexBase.java ec04569669829f8df64c72de0465f9b6ed7e4628

  giraph-core/src/main/java/org/apache/giraph/vertex/HashMapVertex.java 2160c3be096271a9c60a27b4113d607b4baa78ee

  giraph-core/src/main/java/org/apache/giraph/vertex/IntIntNullIntVertex.java a2090e8e12bdc17fce367c47d2567b44c40898f9

  giraph-core/src/main/java/org/apache/giraph/vertex/IntNullNullNullVertex.java f36f6dbde928c8e63cba015d26b917490657ff61

  giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleFloatDoubleEdgeListVertex.java
9d4f6e7f513a0bf0b0322405e3e4edae689d0c10 
  giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleFloatDoubleVertex.java befadea477f40b1c74c6f87a461921ed8e78d78b

  giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleNullDoubleVertex.java 63f1d28363e1bebaeec8ac8723491ca2af085131

  giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphByteArrayVertex.java a50f48dbb43c5aba82450dd32911e61e33adca6b

  giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphEdgeListVertex.java b60d8953ab3191e588c02e1503fed4bb753bd66b

  giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphRepresentativeVertex.java 4733e2a6011ec8e1cc4eef1d2eb61abe777ec310

  giraph-core/src/main/java/org/apache/giraph/vertex/MutableVertex.java a6d7ce523b1b99dc9de28d3762fd3027da01506e

  giraph-core/src/main/java/org/apache/giraph/vertex/RepresentativeVertex.java f805007b8bb8f89e9388cf89c2e81f92328b2b1c

  giraph-core/src/main/java/org/apache/giraph/vertex/RepresentativeVertexBase.java 4de6ed85b499e74b04e93c3780324a6b9e9f2b83

  giraph-core/src/main/java/org/apache/giraph/vertex/SimpleMutableVertex.java e12f783f407edc1bfdf4c0dcc7f2ffec526e2a78

  giraph-core/src/main/java/org/apache/giraph/vertex/SimpleVertex.java ecea3fba7d721f0d51e003ffdc629fc4563b60f7

  giraph-core/src/main/java/org/apache/giraph/vertex/Vertex.java a959ef8ec476fd063d88f42598aa3751d603a140

  giraph-core/src/main/java/org/apache/giraph/vertex/package-info.java 5d983e240c715b401619a942a90fffbaac2b08a0

  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 3b510b2e1b615df88010918ff222fcd5b3431077

  giraph-core/src/main/java/org/apache/giraph/worker/DefaultWorkerContext.java 0ffdc0f129cadbb6a69836c46a9b9e13af94bd64

  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java c577cbbedd09356db25dae3f90a7af8450e6f816

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java de1ae599af993f1fc2ffd1b1c151b1c287a1dad2

  giraph-core/src/test/java/org/apache/giraph/BspCase.java 4867a5006539e6d0744cd3b401f133cbee027709

  giraph-core/src/test/java/org/apache/giraph/bsp/BspUtilsTest.java 5c77d124d40b4938ca04907f1708f78f98f9ea0e

  giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java 381ca7c3521979d59415af3992beb8ac4160c1fe

  giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java c42754fb5cc3488e3f9e8859f4bb3b425d358cb2

  giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java 7468c477992ac5e1ca95a8997b86d230e70bc8ac

  giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java aa89b857dd963cb98818a9060ba6e660948cd3d8

  giraph-core/src/test/java/org/apache/giraph/comm/TestMessageStores.java 5ce7a89354639e81068360a978a8db387667550b

  giraph-core/src/test/java/org/apache/giraph/conf/TestObjectCreation.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/edge/TestMultiGraphEdges.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/edge/TestMultiRandomAccessEdges.java PRE-CREATION

  giraph-core/src/test/java/org/apache/giraph/edge/TestNullValueEdges.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/edge/TestStrictGraphEdges.java PRE-CREATION

  giraph-core/src/test/java/org/apache/giraph/edge/TestStrictRandomAccessEdges.java PRE-CREATION

  giraph-core/src/test/java/org/apache/giraph/graph/TestVertexAndEdges.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/io/TestAdjacencyListTextVertexOutputFormat.java
036924fb87c92f46f7773be74b2083313a497445 
  giraph-core/src/test/java/org/apache/giraph/io/TestEdgeInput.java c0806221388e290ffa4e028fba679b66c57f2f5d

  giraph-core/src/test/java/org/apache/giraph/io/TestIdWithValueTextOutputFormat.java e77033874ed65bdab4597269e785005ff870716c

  giraph-core/src/test/java/org/apache/giraph/io/TestJsonBase64Format.java b3c63f6b2dfb559343d8c6a2dabe67cd3d95e672

  giraph-core/src/test/java/org/apache/giraph/io/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java
7ef8eae152abf885ada059fac61e3dbc8e13b7bc 
  giraph-core/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java
d1e267e988b1ff3f90e56cd07b0020fe4a769b57 
  giraph-core/src/test/java/org/apache/giraph/master/TestMasterObserver.java f67a6e0ac15e00528e95df43e54c2e1e107e4b6e

  giraph-core/src/test/java/org/apache/giraph/partition/TestGiraphTransferRegulator.java 2671708bc024d5029fe04265ef4cff51e661a209

  giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java b4cddf6c6f2d858b1a2ab85433dc5f4b93ebe82c

  giraph-core/src/test/java/org/apache/giraph/utils/ComparisonUtilsTest.java 789c9eeb3b73198e31d5ac23c3d5e5603d92e479

  giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java 04c7a3c028f91029d47448a1341256924d15830c

  giraph-core/src/test/java/org/apache/giraph/vertex/TestIntIntNullIntVertex.java 37e87683c63562ccd76ac5b6edb5680104d63e81

  giraph-core/src/test/java/org/apache/giraph/vertex/TestMultiGraphVertex.java 173c453ddfe5118f02ce393642e3f89901e78f2c

  giraph-core/src/test/java/org/apache/giraph/vertex/TestMutableVertex.java 2b347c6804fd9c488406b1ebba3e91675f4626d8

  giraph-examples/pom.xml a6255113387bbf74d33bbc2c00f0a622edbc0787 
  giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java c00fda0cc5f6b9e113ed08c59492f7dc169c350e

  giraph-examples/src/main/java/org/apache/giraph/examples/ConnectedComponentsVertex.java
b5c90e5b233ace68cd90ecd0ae222c5e90aeda92 
  giraph-examples/src/main/java/org/apache/giraph/examples/IdentityVertex.java 70000356121581d3c2ba65235401801f6dcc3a6e

  giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleDoubleDoubleTextInputFormat.java
PRE-CREATION 
  giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleFloatDoubleTextInputFormat.java
b1305cc74ce90fadfe740b80547aff69447fba94 
  giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleDoubleDoubleTextInputFormat.java
PRE-CREATION 
  giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleFloatDoubleTextInputFormat.java
1d41a943dab833db46b31aee09151bb471a08b59 
  giraph-examples/src/main/java/org/apache/giraph/examples/PartitionContextTestVertex.java
f86c32397a6854ec2c2c4046adbc4f03be7b9562 
  giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertex.java 5669cf27cec180599aa9da1266926a20e559d82f

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 09c7f5616931fa078fecb6816c387b806a4bc85b

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCombinerVertex.java be0f16cb4a1780e213238cd9510340cdbd812cdf

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleFailVertex.java 521ed97121d1f57b519f6f0fa67556e14b4b23d2

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleInDegreeCountVertex.java
e05da3ecaced37b228cf18741e399b1f4bca4e59 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMasterComputeVertex.java
e481ff7570d8e08006190bc560a24ee8bc7156bc 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMsgVertex.java 8016490a60fc99dc89d19b2f5280dc9670554ba8

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java 6032e88d52ed979c356a68c2367ad0019a48477a

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleOutDegreeCountVertex.java
1e010a101babc872369061d7fdce5c0971e399ae 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java f36f1acb3c49fa95188020ce75f1e2131f9e4040

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java
1bec7ebad125bec73469a7fe0489b5b63c5ee128 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java 2afe15dde832249c2af6a19e1a8fe4fdc6664af8

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTextVertexOutputFormat.java
a57c6d24297ee52157fd0651dba9c74202e8e42c 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTriangleClosingVertex.java
469de85254e1dfd10c392f7fa615619632c1827a 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
f6488d5d0857758bd624576fbcc0ce72b0b62a8c 
  giraph-examples/src/main/java/org/apache/giraph/examples/VerifyMessage.java 994b59dc5174b3ff657805f896a41c4229ea243a

  giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueDoubleEdgeTextOutputFormat.java
PRE-CREATION 
  giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueFloatEdgeTextOutputFormat.java
ef58bb8367e09434179d268f2a51ca748855f510 
  giraph-examples/src/test/java/org/apache/giraph/TestBspBasic.java 0d6d1d0126744c3bec4c78b613f87cd669fabbf8

  giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTest.java
49a2f33ac26e37133fe9a518fdb7abd1adac06db 
  giraph-examples/src/test/java/org/apache/giraph/examples/RandomWalkWithRestartVertexTest.java
4052fe1b8b5e575a79f4a07f0f95ea6e30a1f3a5 
  giraph-examples/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java
a372a2ddc4f5fde1e73b935d1c0aebab36326a95 
  giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java
969c1f41e5f4a3d6f0d7085abe2d95492f92da1d 
  giraph-examples/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java
2f9704d01d81a978e8c6c32b256260c97288141b 
  giraph-examples/src/test/java/org/apache/giraph/vertex/TestVertexTypes.java bcfbd39c01825e33300b9e5ce6deff3110e0eaf7

  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/TestHBaseRootMarkerVertextFormat.java
fe5b72e4413cb640f099a6ebb0aa0aa93a499d3e 
  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java
f45f07ae2340b9a0536e5bd944c8a4ff53ade089 
  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeOutputFormat.java
a1313570099e9d3075dc12f89be9a6c3091faae5 
  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java 1bd0235f997ab93c2029e11c4b71230837a04cff

  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogEdgeInputFormat.java
d8987da2c7dfbcc47e379e5a61e965bd97c7b1d5 
  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java
9d52b64751dffc63baa4e39331337892d8f668d2 
  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexOutputFormat.java
4bab7dd09248c4ed5cfc0be51ee5b7f75e0cb3a6 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java cef96b530f65cde8ee5cc8ef30dd8a242aea6f76

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java 34dc4df1f654ef8cc1888802124041cbf9378ab2

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertex.java 31f0e64b2b15d45be8e6630a4e8a383784564420

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexReader.java c5974de8a0e457b54b273086cd3ab05b5b27042b

  giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexWriter.java ea24fc526de052ef8a1d53680a1c9fadda32a0a0

  giraph-hive/src/main/java/org/apache/giraph/hive/output/VertexToHive.java a72cf0b8ec6a136326870049759b92c5b064d6f7

  pom.xml 901e31fcc0bc868b8742401cd20e01b06fc60bcd 

Diff: https://reviews.apache.org/r/9732/diff/


Testing (updated)
-------

1) mvn verify
2) benchmarks (https://gist.github.com/apresta/7b8ff52951da76ee8a47)
In short: no significant differences when using byte arrays. Using primitive arrays is faster,
and becomes feasible when using edge input too (no copies are performed when moving edges
to their source vertices).
3) check results on real dataset: 100% agreement


Thanks,

Alessandro Presta


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message