giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Review Request: GIRAPH-667: Decouple Vertex data and Computation, make Computation and Combiner classes switchable
Date Wed, 15 May 2013 18:40:48 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11182/
-----------------------------------------------------------

Review request for giraph.


Description
-------

Currently, our Vertex class holds a lot of stuff which shouldn't be there, related to global
graph and worker state. We should decouple vertex and computation, vertex should be provided
by the infrastructure and user should implement the computation.

In many real-world applications there are several different stages, where different kind of
computation is done and different type of messages are sent. This can be done currently by
having complicated compute() and encoding the message type inside of the message. Much better
would be to provide a way to change which Computation is used by Giraph. Applications can
then be considered as pieces which can be put together in a pipeline coordinated by master.

Main changes:
- Added Computation<I, V, E, M1, M2> class, where M1 is incoming and M2 outgoing message
type. Computation has compute(vertex, messages) method which should do what Vertex.compute(messages)
was doing. Methods left in Vertex are only the ones which handle the Vertex data, all other
stuff are moved to Computation. Also Vertex doesn't need GraphState anymore.
- MasterCompute has methods setComputation and setCombiner - if these are not called everything
works as before - Computation and Combiner from previous superstep are used.
- BasicComputation<I, V, E, M> class is what current application should extend - here
incoming and outgoing message type is the same. Switching existing application to these stuff
is pretty straightforward: 
  * Instead of extending Vertex, one should extend BasicComputation
  * In compute method, wherever some vertex data methods are called (i.e. get/setValue, edges
methods) we prefix that with vertex.*
- Cleaned up GraphState - that one was used all over the place and had many fields which were
used only by ComputeCallable
- No need for PartitionContext anymore - Computation can hold whichever reusable objects we
need and they can be accessed directly from compute()
- This touches a lot of classes because I removed M parameter from almost everywhere, since
that one is not constant anymore; had to rename all *Vertex* classes to *Computation*. Classes
which hold the main change are BspService/Master/Worker, Vertex, Computation, BasicComputation,
MasterCompute, SuperstepClasses, GraphState, ComputeCallable, ImmutableClassesGiraphConfiguration.
To see how new stuff are used, take a look at TestSwitchClasses.


This addresses bug GIRAPH-667.
    https://issues.apache.org/jira/browse/GIRAPH-667


Diffs
-----

  giraph-accumulo/src/main/java/org/apache/giraph/io/accumulo/AccumuloVertexInputFormat.java
221fabd3016782db70c15bc6c56d41913164d664 
  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/TestAccumuloVertexFormat.java
6698c9a514a191b2aa900b4f1a833685af02391d 
  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java
5855cfcd4225db9289ebdfe3e85e52b627bd429b 
  giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeOutputFormat.java
f999497a8782143afc0c42ac693470d197528075 
  giraph-core/src/main/java/org/apache/giraph/benchmark/AggregatorsBenchmark.java 12304bb426dc8c11d2a3bda161b969c64d7e3f15

  giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 0f8d284998974beba3884240a8ede71bb81c1098

  giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankVertex.java 9900a44bbe01eb9d5b1838fe4cd6c84db848e8b5

  giraph-core/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java 5c7e019dbed30a6e66734a301b3ab9017a2b7c65

  giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsBenchmark.java 8e6c877c84d0a43a935ae6259fd3663d02ee8c56

  giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsVertex.java c1b77d1ff0a4803c3b8c93732f1395e30fe30fac

  giraph-core/src/main/java/org/apache/giraph/benchmark/WeightedPageRankBenchmark.java 3fc514a2fb90048d1aca57198efed3c89d283bec

  giraph-core/src/main/java/org/apache/giraph/benchmark/WeightedPageRankVertex.java 70f0f619ea1f3283cac6ce96d94d3bd508171179

  giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 178c96fd8106017751df1d3aa0295b238611ae54

  giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedService.java 2281903e943370c6b66508ca0cf8584b8bbc1aec

  giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java 5f84ece17082b1d0d5ede7aeabfe557945d4e8d0

  giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1c7bde4c83cd7cb9507d426b5ee2ade34202088d

  giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java 1e8bdf99200a5b48efa1d7b2b5e5ea5ca0526f83

  giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java fbc911f5665a7a6f304166a3f1acde00a7d71296

  giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 7d2a888254d9c52ca8b445923b7a1ff619c83f77

  giraph-core/src/main/java/org/apache/giraph/comm/SendMutationsCache.java 67f74f1f29ef73e9c86298d645b82e8e9bcd45af

  giraph-core/src/main/java/org/apache/giraph/comm/SendPartitionCache.java 31cf05271dc6aa899bc8fff956555797d200e2ac

  giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java 743a6f83d6727c7365eeadb1b30c5e0fc41274c7

  giraph-core/src/main/java/org/apache/giraph/comm/WorkerClient.java ebabf45482bd7c4f2fc47b2f2958a6245d886a82

  giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java bc0637faa52d6f79d6616da4a17a86fdd2166494

  giraph-core/src/main/java/org/apache/giraph/comm/WorkerServer.java e373b2cb525046ddc61c8e74f21d7fc150134191

  giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java
3cd1175054aa2401becc42f93f54b6d7e83ee372 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 26abb94ebd62609fb98144f5b8453f023f34b58a

  giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java
02bfb1fbaa0448541683b6ca194cea6167f33301 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java
PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStoreFactory.java 3c13c4795fe45e354d49dbab620ed9e7a0f03920

  giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java
7db02667febf5db3080782f9e816cdff28e7382f 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/SequentialFileMessageStore.java
3fe44308a0b0c1989bfeb1b8601a5f75e5336cb3 
  giraph-core/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java fc6560dd454c5f8db5f32203edbfd723012750e3

  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java 319f41a99b3dfd94be0f412e599e0c5cc0208d16

  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1f04bcff4ccbce7cacb453689bc5361753f1d935

  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerAggregatorRequestProcessor.java
d1cce64c8dba8e17f50bc03dfde671e0e2ec41ac 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 9c09524577840f2080234664db909ca9f3ea516e

  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
db4ff5d0abbb15353107636d706127c561d8ef39 
  giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java ed0861ef8eab0a70f04a561add12e10eea2fb34a

  giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/WorkerRequestServerHandler.java
b4e7ddae59d1c9d76462eca34205300486f43f67 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendPartitionCurrentMessagesRequest.java
037f4a055c8978b9708024beb24c1a09f91e07d5 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendPartitionMutationsRequest.java
a96842d37093bf9a6d5a7474d158f3093b95c23e 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendVertexRequest.java 1de3cbb372a659c64826b4d948f06ec90d2ccf82

  giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
04b633b47b63e0d4b62891aa8b5ebedec67ea892 
  giraph-core/src/main/java/org/apache/giraph/comm/requests/WorkerRequest.java 4d9382f0b6b7279e31ba84dfc4c24f4a0e144d24

  giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java fad20b0714b55bf5b0a00903fa2a0686eb673c3b

  giraph-core/src/main/java/org/apache/giraph/conf/AllOptions.java cceaaef8a141e4867abd9c69ab6fb74959abedc2

  giraph-core/src/main/java/org/apache/giraph/conf/DefaultImmutableClassesGiraphConfigurable.java
77564ee8d954266f4bf5f5f798ad4f712b43c725 
  giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 10e49756718c217fdedf223ae853876c39c64f4d

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 754fad9207263d62f1dc378e9b9de48d7aa0c0ad

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java bbf50e57ec76eba0a8553f1a7890d161701d358a

  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfigurable.java
e4351a2c555e11793b0eea4ded6a0e8b82237cb7 
  giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
a9add4f44a79e8bbba764be748d26d9311e1dd00 
  giraph-core/src/main/java/org/apache/giraph/edge/ConfigurableOutEdges.java 631b209d2d02f5eb54be2a1fe74ab26dff7715f4

  giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java 9e2d246fe5afa0010c94e9f79bf3515fc4bd2d63

  giraph-core/src/main/java/org/apache/giraph/edge/MutableEdgesIterable.java 82486f49166dc568e1dd7a665d48310326b49951

  giraph-core/src/main/java/org/apache/giraph/edge/MutableEdgesWrapper.java cd845d0cbd5f2296a12fa4a25969469c0759b702

  giraph-core/src/main/java/org/apache/giraph/graph/Computation.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java 0fc5fdfda88b56e617c4eb8317bb5f76dc0ec41c

  giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexResolver.java 52df38dd3fca3deb0b21175d056cd677d5eb78e8

  giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexValueFactory.java 47902d13a4bcfd26603fa27e1fb41d2ef9abf800

  giraph-core/src/main/java/org/apache/giraph/graph/GiraphTransferRegulator.java 3c2286d958b307456f31d86ec86a52b8a434f44d

  giraph-core/src/main/java/org/apache/giraph/graph/GraphMapper.java 4181d24547f0fd54dc001b8368b223746c54a383

  giraph-core/src/main/java/org/apache/giraph/graph/GraphState.java 93ad5dfc435f561dc2f18d4305d28244a4021f45

  giraph-core/src/main/java/org/apache/giraph/graph/GraphStateAware.java 76cef433283f543f30f2cb9a8036d32e4b45862a

  giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java 82e1b1e745491f1bc6b6fa8ea2509a0de794ac28

  giraph-core/src/main/java/org/apache/giraph/graph/Vertex.java 61624e5846f382dc046ae7acad00637e4ef47278

  giraph-core/src/main/java/org/apache/giraph/graph/VertexChanges.java 947463699a6b355a20dd278ac7ff2019fb159337

  giraph-core/src/main/java/org/apache/giraph/graph/VertexMutations.java 75c0aef06212303a1669bd9c30e8e14e54da8870

  giraph-core/src/main/java/org/apache/giraph/graph/VertexResolver.java 1fc0ddcc23b6487d892110622d5272ccebdbb099

  giraph-core/src/main/java/org/apache/giraph/graph/VertexValueFactory.java e62bb01139dc3f3da46282f8881227bd05541e68

  giraph-core/src/main/java/org/apache/giraph/integration/SuperstepHashPartitionerFactory.java
c03d718312bf2d0b61e01bca94449adbcd53640e 
  giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java 6d87bf2f2748b37717878d8d85e68bd30e33d650

  giraph-core/src/main/java/org/apache/giraph/io/GiraphInputFormat.java 58d79a63d403a2cc7c274d42abf5814d46f7f3c3

  giraph-core/src/main/java/org/apache/giraph/io/SimpleVertexWriter.java e4c34962dceb996e59580b62b7cc26f67a2b1282

  giraph-core/src/main/java/org/apache/giraph/io/VertexOutputFormat.java 154f7e41de45a1f63d3934e79e714e82b2b78e20

  giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java 3f6bb3fdc866e8ef9a181ff2f30fee54272459d5

  giraph-core/src/main/java/org/apache/giraph/io/VertexValueReader.java 0b06a4a05de5658be3b14c924c0cb4f9665f34b0

  giraph-core/src/main/java/org/apache/giraph/io/VertexWriter.java a4285c1176dd50e993afa23256da963c7c546340

  giraph-core/src/main/java/org/apache/giraph/io/filters/DefaultEdgeInputFilter.java ad52496f9b8f2a919a6f165eb48b325b95515bf5

  giraph-core/src/main/java/org/apache/giraph/io/filters/DefaultVertexInputFilter.java 2976cbc79510b8ac663dab33ca2398dbf6cee152

  giraph-core/src/main/java/org/apache/giraph/io/filters/VertexInputFilter.java d9af103aa9dded4ec80d58652a9b4882cdbf9d98

  giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexOutputFormat.java
f71ef25acbae07a43653a68473ff0f54072ea82e 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IdWithValueTextOutputFormat.java
6dd74688d317edeedc91fd3c150996e308b1f785 
  giraph-core/src/main/java/org/apache/giraph/io/formats/IntIntTextVertexValueInputFormat.java
a7dbef8fb0fca599c04e344a144f429bac18145b 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexOutputFormat.java
7d8fcf65267bc126ccfcf223a26f517c94c2a85c 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexInputFormat.java
7dfd6078f8b9408674ba03690f4597a3f975ed79 
  giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexOutputFormat.java
d0a330538807227a95da2ff853f07b36cae86a04 
  giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomIntNullVertexInputFormat.java
c9390bacd2c40c2945d157f42dae2e032d4a305f 
  giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomVertexInputFormat.java
5d293ebacb58b0bbfd7856a40e3ae66d0151b9b1 
  giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexInputFormat.java
0f2d929a6d5fa31156ead71b5b09a806ec73c043 
  giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexOutputFormat.java
c4ed65c2eb866cf2ad49881773fcdf022ac00bde 
  giraph-core/src/main/java/org/apache/giraph/io/formats/TextDoubleDoubleAdjacencyListVertexInputFormat.java
6e62b714d9f3f61fa67b91539c75222e8bfa2e03 
  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexInputFormat.java b08e6f767a2f342b8a3b22d9d687de513acc8509

  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java a3073f9bd62baad934b64d9dfe189f5b32e7cb95

  giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexValueInputFormat.java 6d133ae5c10b0e0dbfe26e4f126749cd35c1fdd8

  giraph-core/src/main/java/org/apache/giraph/io/formats/multi/EdgeInputFormatDescription.java
569cee93d9819bbc08975a2c87f095b47d85004e 
  giraph-core/src/main/java/org/apache/giraph/io/formats/multi/MultiEdgeInputFormat.java c377fbcd59bf076f68074e2fa2849d01f9f8a463

  giraph-core/src/main/java/org/apache/giraph/io/formats/multi/MultiVertexInputFormat.java
72929d961252d53e979df1fe3f8449f9ca0fde55 
  giraph-core/src/main/java/org/apache/giraph/io/formats/multi/VertexInputFormatDescription.java
bdd5a74001b04a92de0da68e163d0e1799958400 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeReader.java c0a2cd1dbbc8127d226ba40322a3246f9623c0cc

  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexOutputFormat.java 8110209eea16a5b0fc4fe45b82c2737849c367a9

  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexReader.java 3a8ac506ea4a8ef290528cad3db4af0b462a583e

  giraph-core/src/main/java/org/apache/giraph/io/iterables/EdgeReaderWrapper.java efd8fe742318fdad86d0b358c7b465316d30e8cb

  giraph-core/src/main/java/org/apache/giraph/io/iterables/VertexReaderWrapper.java 614f9453ab434fefef3ab4d5a8f1daee4e8edcd5

  giraph-core/src/main/java/org/apache/giraph/io/superstep_output/MultiThreadedSuperstepOutput.java
a8dff876ba45689705cacea2ebd47a9b07eb1aaf 
  giraph-core/src/main/java/org/apache/giraph/io/superstep_output/NoOpSuperstepOutput.java
82684b2265857da1b68296d56a6d447cb201a990 
  giraph-core/src/main/java/org/apache/giraph/io/superstep_output/SynchronizedSuperstepOutput.java
f94bd56d0b877fa778732b502431e753c09b8733 
  giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java cc6b126ca3e2fa30f8e66fff4648da30d5c465a8

  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java f00116a7cea96be8c18c327eda767a2a2698d9dc

  giraph-core/src/main/java/org/apache/giraph/master/MasterAggregatorHandler.java f769c3a24962e0ac9e275db838d4ed9cad2e9007

  giraph-core/src/main/java/org/apache/giraph/master/MasterCompute.java 1c36e03def9262d712ea04a4609801d74f4b6c27

  giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java ba2f8eb7a093e616f15fc0875fa0b03999f1861b

  giraph-core/src/main/java/org/apache/giraph/master/SuperstepClasses.java PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/partition/BasicPartition.java dc9192ed85c10c083153f0ecf0ae9bd005e1b84b

  giraph-core/src/main/java/org/apache/giraph/partition/ByteArrayPartition.java c4669d38868af4c244ee750536e2080748357eba

  giraph-core/src/main/java/org/apache/giraph/partition/DefaultPartitionContext.java c22c802e26807fa27425a93e6033e92a02d23e59

  giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java a4739f1d147ff689afbe5a1e8d7b94897c9e3d88

  giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java e2e04ddf44145009717e843a29f6d4a12eed01db

  giraph-core/src/main/java/org/apache/giraph/partition/HashMasterPartitioner.java 5faf3675c17d40b1e015110a886ef6a37cba0031

  giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java f7343a11f5632fcbd742c43e70d2dde29b2363f3

  giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java 227e23483920a62df8592f384301c8ac7ae9aadc

  giraph-core/src/main/java/org/apache/giraph/partition/HashRangeWorkerPartitioner.java a6e764d86c5a9f4a96e1ec09f867b94e84544b62

  giraph-core/src/main/java/org/apache/giraph/partition/HashWorkerPartitioner.java 599ea0c88ca1a3eb5d2ef099d58f62ad0b0fd9fb

  giraph-core/src/main/java/org/apache/giraph/partition/MasterGraphPartitioner.java 130ee07b2ffcbe6181528447ee4fba8283ac33ec

  giraph-core/src/main/java/org/apache/giraph/partition/Partition.java 1ca0b61cec3f6938211864da8c0503cc184b86e3

  giraph-core/src/main/java/org/apache/giraph/partition/PartitionContext.java 412f6e3aa1a3a533425313470e95a008f1178a95

  giraph-core/src/main/java/org/apache/giraph/partition/PartitionStore.java 4206ce343ec36dce39fa9a558f392d063b0c8e5f

  giraph-core/src/main/java/org/apache/giraph/partition/RangeMasterPartitioner.java f9b0329983abe715fd7d1fd33c7be8f54cbc2bc9

  giraph-core/src/main/java/org/apache/giraph/partition/RangePartitionerFactory.java 29f7898120eafac08ee17327da2072e3ad5a5670

  giraph-core/src/main/java/org/apache/giraph/partition/RangeSplitHint.java 9634c3386669441d43121e1872c5a90bf47ffcb2

  giraph-core/src/main/java/org/apache/giraph/partition/RangeWorkerPartitioner.java 5a494a53f0549277a5ed86d033435d0d8cf3d094

  giraph-core/src/main/java/org/apache/giraph/partition/ReusesObjectsPartition.java 4153d554881f1586545242ba7c00f951630077ca

  giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
9ac2e11e780499dac6ea9235ca73cacaaf7fa4fd 
  giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
5772a7b5cd3cf6fa49c09934900630fab3a8094c 
  giraph-core/src/main/java/org/apache/giraph/partition/SimplePartition.java d6a46bd3c178506865b85b8b596d95dde59064b2

  giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionStore.java 74cc3a7d8e719fb1c728e77eaf345e75ca973508

  giraph-core/src/main/java/org/apache/giraph/partition/SimpleRangeMasterPartitioner.java
bf34ecde3dde7fd4d868d396f97e473cbcda76ba 
  giraph-core/src/main/java/org/apache/giraph/partition/SimpleRangeWorkerPartitioner.java
f94c14bedc20605ca2bef29d01440423d8ba2a02 
  giraph-core/src/main/java/org/apache/giraph/partition/WorkerGraphPartitioner.java 5a78b1d54109d80d94911971c522c543cecc8e33

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 2cfa661ee7204e73e6aa3705f742e9f58acfc4e9

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java 2c5f2f7f69b06001f004992c396bf49e6fd6785b

  giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java 0280c58b4e11ed356e77761771db930303f53a80

  giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java 6016ba40241e5b339b44a594e51d6278eaee1967

  giraph-core/src/main/java/org/apache/giraph/utils/InMemoryVertexInputFormat.java 65d99db8edea57c5c2d75c8c346cead581d5d783

  giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java be2d2a9c127250fb302da9eedc9af82613d049e2

  giraph-core/src/main/java/org/apache/giraph/utils/ReflectionUtils.java d70eecbd2429d5824b57e057a390a575e8ea83f2

  giraph-core/src/main/java/org/apache/giraph/utils/TestGraph.java 3577a9e4482054e91fb5094ac93207ff7fc44ea5

  giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java 0c9ee076aba989e32288bc9f9f6d182625a3a5cf

  giraph-core/src/main/java/org/apache/giraph/utils/WritableUtils.java c607ca326046e693b8a21b50a968ef6404b56a6d

  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 03a4876c1cec2d8598a3bb086ca92a91050bb159

  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 351a114390a90528a6f4c1b46dcdb77c50e845aa

  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java 33fb5159cc87e0653c0e8933d706159aa97954a4

  giraph-core/src/main/java/org/apache/giraph/worker/InputSplitsCallable.java a8298c5d9e2c0631ef08dd044eaf0a69ae089e0a

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 1c292ad1db3d2b0149c3f568c12a7279d831b1ca

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java
cf5e8ad85a6663baec90e0d81c6fcffbcdf9e4ef 
  giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java 9a8a8b8bc4de65708a6416e7643ef4b22897fad3

  giraph-core/src/main/java/org/apache/giraph/worker/WorkerContext.java d3ffaeadf5b32f039dc8abe620b1fcf4fcac08e0

  giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java 91b842afbaac2962bbf8f46ec1cabbcfc5b94146

  giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 58aa7d1c63523522f27e375a7e65baf770e5fe05

  giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java f1f8e26e2fe99edea1de6a173bf9aab9ad4d47ca

  giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java c27156fb98f86afec352ed1cf70b95e307805d34

  giraph-core/src/test/java/org/apache/giraph/comm/TestMessageStores.java 53b8a2488171bb3c116b9296d5b7467bf890dc8e

  giraph-core/src/test/java/org/apache/giraph/conf/TestObjectCreation.java 9075145eaa7e6744ec656890c8c95ca387ab22e0

  giraph-core/src/test/java/org/apache/giraph/graph/TestVertexAndEdges.java 15d2bb0aad6f17bca90631dc7b95ea5535dc95f0

  giraph-core/src/test/java/org/apache/giraph/io/TestAdjacencyListTextVertexOutputFormat.java
6849e3a938bbe20d7e29ad432baf55504d4cf80a 
  giraph-core/src/test/java/org/apache/giraph/io/TestEdgeInput.java cb1a8daa94ef235cb22d4d2d2ddaed7d74e79fa2

  giraph-core/src/test/java/org/apache/giraph/io/TestFilters.java 83a366da7f3f7b3c80f055d077394f444bff218d

  giraph-core/src/test/java/org/apache/giraph/io/TestIdWithValueTextOutputFormat.java ff4d2b1f572049c747becea19fc61ca0e2d4e493

  giraph-core/src/test/java/org/apache/giraph/io/TestJsonBase64Format.java a6aa78a0399dc6c839b5f1334e8e812b73808d7a

  giraph-core/src/test/java/org/apache/giraph/io/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java
601c824684d7298eedad9093dc83762362482e6c 
  giraph-core/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java
4ee8ea19b30d55cd3c81bb5281f27a15d45a6412 
  giraph-core/src/test/java/org/apache/giraph/master/TestComputationCombinerTypes.java PRE-CREATION

  giraph-core/src/test/java/org/apache/giraph/master/TestMasterObserver.java 9d6b21536ee85ea6835e55e1e5bb477219105e65

  giraph-core/src/test/java/org/apache/giraph/master/TestSwitchClasses.java PRE-CREATION 
  giraph-core/src/test/java/org/apache/giraph/partition/TestGiraphTransferRegulator.java 9b655af1f5c252d285862c75cf44be4ae3c29d05

  giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java 5a93d41e30bb90e3a7f2f2dbce7c44f9d5c8d88c

  giraph-core/src/test/java/org/apache/giraph/utils/ComputationCountEdges.java PRE-CREATION

  giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java 0039ad62502a740db156c37bcc4445aa5801b6b2

  giraph-core/src/test/java/org/apache/giraph/vertices/IntIntNullVertexDoNothing.java c98d5804fa3921b23f4fc54a6554b076c8a1a44c

  giraph-core/src/test/java/org/apache/giraph/vertices/VertexCountEdges.java 9060bc7077ac23a9b362d10698c5facae5dcf55a

  giraph-core/src/test/java/org/apache/giraph/vertices/VertexDoNothing.java fac3fce3eec0280ac10f65b3f4ebfb166fa84d1a

  giraph-core/src/test/java/org/apache/giraph/yarn/TestYarnJob.java f9d5544edaee5471a4087c9ec81b4a08ddea2a20

  giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java d08519ba3dc6e5517e99dea640f0f9d16c6594f4

  giraph-examples/src/main/java/org/apache/giraph/examples/ConnectedComponentsVertex.java
dbeb6bf98721cf11744f45682681599f18fe0487 
  giraph-examples/src/main/java/org/apache/giraph/examples/IdentityVertex.java 30cca86547de3806c271099eea87e95429b8f548

  giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleDoubleTextInputFormat.java
62bea5a7f186d9e731d4fa25f11f238d47d33407 
  giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleNullTextInputFormat.java
fdc90505b52a43e70ebc0802ca6af46d6c2c128d 
  giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleDoubleTextInputFormat.java
7dc8475c095207cc7fed199cc05ed80205085549 
  giraph-examples/src/main/java/org/apache/giraph/examples/PageRankVertex.java 9678b31a2747d899888d99933716aa5fe3d40757

  giraph-examples/src/main/java/org/apache/giraph/examples/PartitionContextTestVertex.java
f617d8ea577c1083bf6db72c047dec336f801667 
  giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertex.java 2d2c988b7163b7d3a4584df91e211c1d3f73b873

  giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertexMasterCompute.java
9e5dbbf9416f771ee957b0d3765c002dced0b2a9 
  giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkWithRestartVertex.java
6f3eb6cdc43b14c2d3776704c9a84178ba12c450 
  giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkWorkerContext.java 2566f43b7fbb91933034df7b3645435807c20fcd

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 03d977b1bed88a8dc61498ad44ea7fb895eefe1b

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCombinerVertex.java f4475aeef0784bad456cb91fef22d651d7bc4d21

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleFailVertex.java c0e206c304e0df2608893beee70fb3d52980f024

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleInDegreeCountVertex.java
3bcf3f5324aec428c00454e4d578333e05e6ced8 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleLongDoubleDoubleDoubleIdentityVertex.java
c7349d15c847b4eedca5a14d073487b97d2cf6eb 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMasterComputeVertex.java
8a21ad788ad5b3bf4c24ee63a7d99d9d4448fe30 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMsgVertex.java 024fe9d0841a1b6c99ded1d2d2a18e27c6d77858

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java a468491c6a8acd3256fb9bfe14edc559fe743f4b

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleOutDegreeCountVertex.java
c830fa2a6060995776e0227a53bbb969779b9b0f 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 7a63e8d7052437a9e1d0c859805d5c58b60f289d

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java
13d1d7c1e37571fe0207cc93659e26bb4ddad580 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java 6f8b352d2e2500390c21d775b662610ce669badb

  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTextVertexOutputFormat.java
157e6ef6c4e005b20d06b510190ea3070c6d4335 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTriangleClosingVertex.java
f44cb188117b21e068aedcc6d03b1ef1210b8900 
  giraph-examples/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
8a6f77562b9ffa3c7fa5ae87a9afbf000ef80305 
  giraph-examples/src/main/java/org/apache/giraph/examples/VerifyMessage.java 6e3c589be8d55dbc6dbe4a51c92a77ad282fe09f

  giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueDoubleEdgeTextOutputFormat.java
d3281535b2107ac2eeacd20318459544974b69cd 
  giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueNullEdgeTextOutputFormat.java
85f3556b247a6aa51a5a392e70986377b1fc6e44 
  giraph-examples/src/test/java/org/apache/giraph/TestAutoCheckpoint.java 652913b4965a3ccc8f5294bc4976d8f369489807

  giraph-examples/src/test/java/org/apache/giraph/TestBspBasic.java 38ba27fae8c9725545c63ea97b8b7e080cfdb0d3

  giraph-examples/src/test/java/org/apache/giraph/TestGraphPartitioner.java 12f0d8d65a18b7cf167b59635561f11fcf397a7c

  giraph-examples/src/test/java/org/apache/giraph/TestManualCheckpoint.java 766e1af3fb838991f757a6d5cc7cf797407d4d64

  giraph-examples/src/test/java/org/apache/giraph/TestMaxSuperstep.java 0989ac5e9fbfe5c1e23484015cbac29478d86bb9

  giraph-examples/src/test/java/org/apache/giraph/TestMutateGraph.java da85fc4fa8e8d309cf5e332975d0a696916ea43e

  giraph-examples/src/test/java/org/apache/giraph/TestNotEnoughMapTasks.java 6da9c500717b00ed58da21e06a96e06eb873422d

  giraph-examples/src/test/java/org/apache/giraph/TestPartitionContext.java 4b042dfda701e61df0e79cb117475a090f876ab8

  giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
53cdeab311b1a5ec2b5ab92fa55c1dca790d8dd5 
  giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTest.java
5d712594ef83d7e0a7501f96b0008cf8a44bae50 
  giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTestInMemory.java
e4c029c4ac5bb1fc3b59f5a2af98c353ce330d32 
  giraph-examples/src/test/java/org/apache/giraph/examples/PageRankVertexTest.java 2a392915093cc64ccb5f9d4c037a509bbfe9c0f1

  giraph-examples/src/test/java/org/apache/giraph/examples/RandomWalkWithRestartVertexTest.java
a2a891e28575c858c3208187eef7e4f9c8bf3431 
  giraph-examples/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java
10065777cf51ac4b02f9ebbbe1f3a77b31ae9eb2 
  giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java
bfe8002cde95c823133936438442db218754648c 
  giraph-examples/src/test/java/org/apache/giraph/examples/TestPageRank.java 922e7363caa8b8a90033c9f300a20edc9f093283

  giraph-examples/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java
23d055e2e9f8e4b9af26354e3f060b97748112a0 
  giraph-examples/src/test/java/org/apache/giraph/vertex/TestVertexTypes.java 0898d635e5a98f7e5ee871e41df2b751c907e1d4

  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/TestHBaseRootMarkerVertextFormat.java
5091b48ae4f6c01396747ccc67a6c03a0127dce1 
  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java
5ecb7dde8e8a3a02b3abacf4a002564b7e30e87e 
  giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeOutputFormat.java
859f519f9f4dfdf7d0e590a77e018392ae21d673 
  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java 2182a7133a7e011453f2f9a38d0e342a42870ad9

  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java
08b263fddc2ad16246bd51ef684481fd3567d393 
  giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexOutputFormat.java
37b3b74aecd6b5f7d71e1c5afb8219fe36141261 
  giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java da9ee2fc048c791388e3f5681f757d50f52674f7

  giraph-hive/src/main/java/org/apache/giraph/hive/common/DefaultConfigurableAndTableSchemaAware.java
4ba69c9ed474243bedf9cf9fc751d866e3e8ae55 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/AbstractHiveToEdge.java 73fc553f1c92ed11eede41d5dc1af636202f5f0b

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeInputFormat.java f35b6ea021e9547319561b23fd1bc5fa8dcc734d

  giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java ba992672f20007e4abc949dcc43deb2ce0e219e7

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/AbstractHiveToVertex.java
b08a09fdcfba9bd6a648f4cd4df7b95093c1615a 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertex.java c670876980396093bd46fb20c5c6c94b42c2253b

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexInputFormat.java
824d8a6a1ac52915cf44fc6c6ab9e81de793ef39 
  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexReader.java 719d8ad1c200cf74d8bfc8921f1549157c26dbc5

  giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/SimpleHiveToVertex.java 6df269d451f3470dc6d46d29a2e60dac452a214f

  giraph-hive/src/main/java/org/apache/giraph/hive/output/AbstractVertexToHive.java 52fe93f1375dd7c7278e7f7c93ccf9c950a7ff9d

  giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexOutputFormat.java 05441f4c0aec209b60cd970e94ce5f6116feff0b

  giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexWriter.java 357cf89c95096537f2ee7a99fdce389dbeafb79a

  giraph-hive/src/main/java/org/apache/giraph/hive/output/SimpleVertexToHive.java fdfb3630a293a73036da396c1769e53f11601fe8

  giraph-hive/src/main/java/org/apache/giraph/hive/output/VertexToHive.java 85e438c8d111953c9960f39d7e16c1e92259d7ab

  giraph-hive/src/main/java/org/apache/giraph/hive/output/examples/HiveOutputIntIntVertex.java
7955915c6b560fbb1c9bdca7e91bc3d8ccba8ab0 
  giraph-hive/src/test/java/org/apache/giraph/hive/input/HiveEdgeInputTest.java 2ad87fa469933337399a2d858ff659822c754f62

  giraph-hive/src/test/java/org/apache/giraph/hive/input/HiveVertexInputTest.java b6ab139054d5f597809f374454f7443b42f1e8dd

  giraph-hive/src/test/java/org/apache/giraph/hive/output/HiveOutputTest.java 673c9d503075e99956d8db5696bcd245bbb22ded

  giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/VertexCountEdges.java a6db2afad47262216758894cd1f1da1b538f4376

  giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/VertexSumEdges.java d8527b4f693c05fc2629fd4f58354bec50c71bc0

  giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/package-info.java 552bb91ac4ec2d0f5719970023d2b83ba3be7987


Diff: https://reviews.apache.org/r/11182/diff/


Testing
-------

mvn clean install
Added some tests for new features
Real application run on a cluster


Thanks,

Maja Kabiljo


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message