giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Review Request: GIRAPH-667: Decouple Vertex data and Computation, make Computation and Combiner classes switchable
Date Wed, 15 May 2013 19:01:13 GMT
I'm not sure I understand the reason of two different data types for
incoming and outgoing messages. can you relate on that?


On Wed, May 15, 2013 at 8:40 PM, Maja Kabiljo <majakabiljo@fb.com> wrote:

>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11182/
> -----------------------------------------------------------
>
> Review request for giraph.
>
>
> Description
> -------
>
> Currently, our Vertex class holds a lot of stuff which shouldn't be there,
> related to global graph and worker state. We should decouple vertex and
> computation, vertex should be provided by the infrastructure and user
> should implement the computation.
>
> In many real-world applications there are several different stages, where
> different kind of computation is done and different type of messages are
> sent. This can be done currently by having complicated compute() and
> encoding the message type inside of the message. Much better would be to
> provide a way to change which Computation is used by Giraph. Applications
> can then be considered as pieces which can be put together in a pipeline
> coordinated by master.
>
> Main changes:
> - Added Computation<I, V, E, M1, M2> class, where M1 is incoming and M2
> outgoing message type. Computation has compute(vertex, messages) method
> which should do what Vertex.compute(messages) was doing. Methods left in
> Vertex are only the ones which handle the Vertex data, all other stuff are
> moved to Computation. Also Vertex doesn't need GraphState anymore.
> - MasterCompute has methods setComputation and setCombiner - if these are
> not called everything works as before - Computation and Combiner from
> previous superstep are used.
> - BasicComputation<I, V, E, M> class is what current application should
> extend - here incoming and outgoing message type is the same. Switching
> existing application to these stuff is pretty straightforward:
>   * Instead of extending Vertex, one should extend BasicComputation
>   * In compute method, wherever some vertex data methods are called (i.e.
> get/setValue, edges methods) we prefix that with vertex.*
> - Cleaned up GraphState - that one was used all over the place and had
> many fields which were used only by ComputeCallable
> - No need for PartitionContext anymore - Computation can hold whichever
> reusable objects we need and they can be accessed directly from compute()
> - This touches a lot of classes because I removed M parameter from almost
> everywhere, since that one is not constant anymore; had to rename all
> *Vertex* classes to *Computation*. Classes which hold the main change are
> BspService/Master/Worker, Vertex, Computation, BasicComputation,
> MasterCompute, SuperstepClasses, GraphState, ComputeCallable,
> ImmutableClassesGiraphConfiguration. To see how new stuff are used, take a
> look at TestSwitchClasses.
>
>
> This addresses bug GIRAPH-667.
>     https://issues.apache.org/jira/browse/GIRAPH-667
>
>
> Diffs
> -----
>
>
> giraph-accumulo/src/main/java/org/apache/giraph/io/accumulo/AccumuloVertexInputFormat.java
> 221fabd3016782db70c15bc6c56d41913164d664
>
> giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/TestAccumuloVertexFormat.java
> 6698c9a514a191b2aa900b4f1a833685af02391d
>
> giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java
> 5855cfcd4225db9289ebdfe3e85e52b627bd429b
>
> giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeOutputFormat.java
> f999497a8782143afc0c42ac693470d197528075
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/AggregatorsBenchmark.java
> 12304bb426dc8c11d2a3bda161b969c64d7e3f15
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
> 0f8d284998974beba3884240a8ede71bb81c1098
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankVertex.java
> 9900a44bbe01eb9d5b1838fe4cd6c84db848e8b5
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
> 5c7e019dbed30a6e66734a301b3ab9017a2b7c65
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsBenchmark.java
> 8e6c877c84d0a43a935ae6259fd3663d02ee8c56
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsVertex.java
> c1b77d1ff0a4803c3b8c93732f1395e30fe30fac
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/WeightedPageRankBenchmark.java
> 3fc514a2fb90048d1aca57198efed3c89d283bec
>
> giraph-core/src/main/java/org/apache/giraph/benchmark/WeightedPageRankVertex.java
> 70f0f619ea1f3283cac6ce96d94d3bd508171179
>   giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java
> 178c96fd8106017751df1d3aa0295b238611ae54
>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedService.java
> 2281903e943370c6b66508ca0cf8584b8bbc1aec
>
> giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java
> 5f84ece17082b1d0d5ede7aeabfe557945d4e8d0
>
> giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java
> 1c7bde4c83cd7cb9507d426b5ee2ade34202088d
>   giraph-core/src/main/java/org/apache/giraph/comm/SendCache.java
> 1e8bdf99200a5b48efa1d7b2b5e5ea5ca0526f83
>   giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java
> fbc911f5665a7a6f304166a3f1acde00a7d71296
>   giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java
> 7d2a888254d9c52ca8b445923b7a1ff619c83f77
>   giraph-core/src/main/java/org/apache/giraph/comm/SendMutationsCache.java
> 67f74f1f29ef73e9c86298d645b82e8e9bcd45af
>   giraph-core/src/main/java/org/apache/giraph/comm/SendPartitionCache.java
> 31cf05271dc6aa899bc8fff956555797d200e2ac
>   giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java
> 743a6f83d6727c7365eeadb1b30c5e0fc41274c7
>   giraph-core/src/main/java/org/apache/giraph/comm/WorkerClient.java
> ebabf45482bd7c4f2fc47b2f2958a6245d886a82
>
> giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java
> bc0637faa52d6f79d6616da4a17a86fdd2166494
>   giraph-core/src/main/java/org/apache/giraph/comm/WorkerServer.java
> e373b2cb525046ddc61c8e74f21d7fc150134191
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java
> 3cd1175054aa2401becc42f93f54b6d7e83ee372
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java
> 26abb94ebd62609fb98144f5b8453f023f34b58a
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java
> 02bfb1fbaa0448541683b6ca194cea6167f33301
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/InMemoryMessageStoreFactory.java
> PRE-CREATION
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStoreFactory.java
> 3c13c4795fe45e354d49dbab620ed9e7a0f03920
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java
> 7db02667febf5db3080782f9e816cdff28e7382f
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/SequentialFileMessageStore.java
> 3fe44308a0b0c1989bfeb1b8601a5f75e5336cb3
>
> giraph-core/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java
> fc6560dd454c5f8db5f32203edbfd723012750e3
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java
> 319f41a99b3dfd94be0f412e599e0c5cc0208d16
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java
> 1f04bcff4ccbce7cacb453689bc5361753f1d935
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerAggregatorRequestProcessor.java
> d1cce64c8dba8e17f50bc03dfde671e0e2ec41ac
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java
> 9c09524577840f2080234664db909ca9f3ea516e
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
> db4ff5d0abbb15353107636d706127c561d8ef39
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java
> ed0861ef8eab0a70f04a561add12e10eea2fb34a
>
> giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/WorkerRequestServerHandler.java
> b4e7ddae59d1c9d76462eca34205300486f43f67
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendPartitionCurrentMessagesRequest.java
> 037f4a055c8978b9708024beb24c1a09f91e07d5
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendPartitionMutationsRequest.java
> a96842d37093bf9a6d5a7474d158f3093b95c23e
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendVertexRequest.java
> 1de3cbb372a659c64826b4d948f06ec90d2ccf82
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java
> 04b633b47b63e0d4b62891aa8b5ebedec67ea892
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/WorkerRequest.java
> 4d9382f0b6b7279e31ba84dfc4c24f4a0e144d24
>
> giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java
> fad20b0714b55bf5b0a00903fa2a0686eb673c3b
>   giraph-core/src/main/java/org/apache/giraph/conf/AllOptions.java
> cceaaef8a141e4867abd9c69ab6fb74959abedc2
>
> giraph-core/src/main/java/org/apache/giraph/conf/DefaultImmutableClassesGiraphConfigurable.java
> 77564ee8d954266f4bf5f5f798ad4f712b43c725
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java
> 10e49756718c217fdedf223ae853876c39c64f4d
>
> giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
> 754fad9207263d62f1dc378e9b9de48d7aa0c0ad
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
> bbf50e57ec76eba0a8553f1a7890d161701d358a
>
> giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfigurable.java
> e4351a2c555e11793b0eea4ded6a0e8b82237cb7
>
> giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
> a9add4f44a79e8bbba764be748d26d9311e1dd00
>
> giraph-core/src/main/java/org/apache/giraph/edge/ConfigurableOutEdges.java
> 631b209d2d02f5eb54be2a1fe74ab26dff7715f4
>   giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java
> 9e2d246fe5afa0010c94e9f79bf3515fc4bd2d63
>
> giraph-core/src/main/java/org/apache/giraph/edge/MutableEdgesIterable.java
> 82486f49166dc568e1dd7a665d48310326b49951
>
> giraph-core/src/main/java/org/apache/giraph/edge/MutableEdgesWrapper.java
> cd845d0cbd5f2296a12fa4a25969469c0759b702
>   giraph-core/src/main/java/org/apache/giraph/graph/Computation.java
> PRE-CREATION
>   giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java
> 0fc5fdfda88b56e617c4eb8317bb5f76dc0ec41c
>
> giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexResolver.java
> 52df38dd3fca3deb0b21175d056cd677d5eb78e8
>
> giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexValueFactory.java
> 47902d13a4bcfd26603fa27e1fb41d2ef9abf800
>
> giraph-core/src/main/java/org/apache/giraph/graph/GiraphTransferRegulator.java
> 3c2286d958b307456f31d86ec86a52b8a434f44d
>   giraph-core/src/main/java/org/apache/giraph/graph/GraphMapper.java
> 4181d24547f0fd54dc001b8368b223746c54a383
>   giraph-core/src/main/java/org/apache/giraph/graph/GraphState.java
> 93ad5dfc435f561dc2f18d4305d28244a4021f45
>   giraph-core/src/main/java/org/apache/giraph/graph/GraphStateAware.java
> 76cef433283f543f30f2cb9a8036d32e4b45862a
>   giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java
> 82e1b1e745491f1bc6b6fa8ea2509a0de794ac28
>   giraph-core/src/main/java/org/apache/giraph/graph/Vertex.java
> 61624e5846f382dc046ae7acad00637e4ef47278
>   giraph-core/src/main/java/org/apache/giraph/graph/VertexChanges.java
> 947463699a6b355a20dd278ac7ff2019fb159337
>   giraph-core/src/main/java/org/apache/giraph/graph/VertexMutations.java
> 75c0aef06212303a1669bd9c30e8e14e54da8870
>   giraph-core/src/main/java/org/apache/giraph/graph/VertexResolver.java
> 1fc0ddcc23b6487d892110622d5272ccebdbb099
>
> giraph-core/src/main/java/org/apache/giraph/graph/VertexValueFactory.java
> e62bb01139dc3f3da46282f8881227bd05541e68
>
> giraph-core/src/main/java/org/apache/giraph/integration/SuperstepHashPartitionerFactory.java
> c03d718312bf2d0b61e01bca94449adbcd53640e
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java
> 6d87bf2f2748b37717878d8d85e68bd30e33d650
>   giraph-core/src/main/java/org/apache/giraph/io/GiraphInputFormat.java
> 58d79a63d403a2cc7c274d42abf5814d46f7f3c3
>   giraph-core/src/main/java/org/apache/giraph/io/SimpleVertexWriter.java
> e4c34962dceb996e59580b62b7cc26f67a2b1282
>   giraph-core/src/main/java/org/apache/giraph/io/VertexOutputFormat.java
> 154f7e41de45a1f63d3934e79e714e82b2b78e20
>   giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java
> 3f6bb3fdc866e8ef9a181ff2f30fee54272459d5
>   giraph-core/src/main/java/org/apache/giraph/io/VertexValueReader.java
> 0b06a4a05de5658be3b14c924c0cb4f9665f34b0
>   giraph-core/src/main/java/org/apache/giraph/io/VertexWriter.java
> a4285c1176dd50e993afa23256da963c7c546340
>
> giraph-core/src/main/java/org/apache/giraph/io/filters/DefaultEdgeInputFilter.java
> ad52496f9b8f2a919a6f165eb48b325b95515bf5
>
> giraph-core/src/main/java/org/apache/giraph/io/filters/DefaultVertexInputFilter.java
> 2976cbc79510b8ac663dab33ca2398dbf6cee152
>
> giraph-core/src/main/java/org/apache/giraph/io/filters/VertexInputFilter.java
> d9af103aa9dded4ec80d58652a9b4882cdbf9d98
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexOutputFormat.java
> f71ef25acbae07a43653a68473ff0f54072ea82e
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/IdWithValueTextOutputFormat.java
> 6dd74688d317edeedc91fd3c150996e308b1f785
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/IntIntTextVertexValueInputFormat.java
> a7dbef8fb0fca599c04e344a144f429bac18145b
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexOutputFormat.java
> 7d8fcf65267bc126ccfcf223a26f517c94c2a85c
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexInputFormat.java
> 7dfd6078f8b9408674ba03690f4597a3f975ed79
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexOutputFormat.java
> d0a330538807227a95da2ff853f07b36cae86a04
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomIntNullVertexInputFormat.java
> c9390bacd2c40c2945d157f42dae2e032d4a305f
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomVertexInputFormat.java
> 5d293ebacb58b0bbfd7856a40e3ae66d0151b9b1
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexInputFormat.java
> 0f2d929a6d5fa31156ead71b5b09a806ec73c043
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexOutputFormat.java
> c4ed65c2eb866cf2ad49881773fcdf022ac00bde
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/TextDoubleDoubleAdjacencyListVertexInputFormat.java
> 6e62b714d9f3f61fa67b91539c75222e8bfa2e03
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexInputFormat.java
> b08e6f767a2f342b8a3b22d9d687de513acc8509
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java
> a3073f9bd62baad934b64d9dfe189f5b32e7cb95
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexValueInputFormat.java
> 6d133ae5c10b0e0dbfe26e4f126749cd35c1fdd8
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/multi/EdgeInputFormatDescription.java
> 569cee93d9819bbc08975a2c87f095b47d85004e
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/multi/MultiEdgeInputFormat.java
> c377fbcd59bf076f68074e2fa2849d01f9f8a463
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/multi/MultiVertexInputFormat.java
> 72929d961252d53e979df1fe3f8449f9ca0fde55
>
> giraph-core/src/main/java/org/apache/giraph/io/formats/multi/VertexInputFormatDescription.java
> bdd5a74001b04a92de0da68e163d0e1799958400
>
> giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeReader.java
> c0a2cd1dbbc8127d226ba40322a3246f9623c0cc
>
> giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexOutputFormat.java
> 8110209eea16a5b0fc4fe45b82c2737849c367a9
>
> giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexReader.java
> 3a8ac506ea4a8ef290528cad3db4af0b462a583e
>
> giraph-core/src/main/java/org/apache/giraph/io/iterables/EdgeReaderWrapper.java
> efd8fe742318fdad86d0b358c7b465316d30e8cb
>
> giraph-core/src/main/java/org/apache/giraph/io/iterables/VertexReaderWrapper.java
> 614f9453ab434fefef3ab4d5a8f1daee4e8edcd5
>
> giraph-core/src/main/java/org/apache/giraph/io/superstep_output/MultiThreadedSuperstepOutput.java
> a8dff876ba45689705cacea2ebd47a9b07eb1aaf
>
> giraph-core/src/main/java/org/apache/giraph/io/superstep_output/NoOpSuperstepOutput.java
> 82684b2265857da1b68296d56a6d447cb201a990
>
> giraph-core/src/main/java/org/apache/giraph/io/superstep_output/SynchronizedSuperstepOutput.java
> f94bd56d0b877fa778732b502431e753c09b8733
>
> giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java
> cc6b126ca3e2fa30f8e66fff4648da30d5c465a8
>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java
> f00116a7cea96be8c18c327eda767a2a2698d9dc
>
> giraph-core/src/main/java/org/apache/giraph/master/MasterAggregatorHandler.java
> f769c3a24962e0ac9e275db838d4ed9cad2e9007
>   giraph-core/src/main/java/org/apache/giraph/master/MasterCompute.java
> 1c36e03def9262d712ea04a4609801d74f4b6c27
>   giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java
> ba2f8eb7a093e616f15fc0875fa0b03999f1861b
>   giraph-core/src/main/java/org/apache/giraph/master/SuperstepClasses.java
> PRE-CREATION
>
> giraph-core/src/main/java/org/apache/giraph/partition/BasicPartition.java
> dc9192ed85c10c083153f0ecf0ae9bd005e1b84b
>
> giraph-core/src/main/java/org/apache/giraph/partition/ByteArrayPartition.java
> c4669d38868af4c244ee750536e2080748357eba
>
> giraph-core/src/main/java/org/apache/giraph/partition/DefaultPartitionContext.java
> c22c802e26807fa27425a93e6033e92a02d23e59
>
> giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java
> a4739f1d147ff689afbe5a1e8d7b94897c9e3d88
>
> giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java
> e2e04ddf44145009717e843a29f6d4a12eed01db
>
> giraph-core/src/main/java/org/apache/giraph/partition/HashMasterPartitioner.java
> 5faf3675c17d40b1e015110a886ef6a37cba0031
>
> giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java
> f7343a11f5632fcbd742c43e70d2dde29b2363f3
>
> giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java
> 227e23483920a62df8592f384301c8ac7ae9aadc
>
> giraph-core/src/main/java/org/apache/giraph/partition/HashRangeWorkerPartitioner.java
> a6e764d86c5a9f4a96e1ec09f867b94e84544b62
>
> giraph-core/src/main/java/org/apache/giraph/partition/HashWorkerPartitioner.java
> 599ea0c88ca1a3eb5d2ef099d58f62ad0b0fd9fb
>
> giraph-core/src/main/java/org/apache/giraph/partition/MasterGraphPartitioner.java
> 130ee07b2ffcbe6181528447ee4fba8283ac33ec
>   giraph-core/src/main/java/org/apache/giraph/partition/Partition.java
> 1ca0b61cec3f6938211864da8c0503cc184b86e3
>
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionContext.java
> 412f6e3aa1a3a533425313470e95a008f1178a95
>
> giraph-core/src/main/java/org/apache/giraph/partition/PartitionStore.java
> 4206ce343ec36dce39fa9a558f392d063b0c8e5f
>
> giraph-core/src/main/java/org/apache/giraph/partition/RangeMasterPartitioner.java
> f9b0329983abe715fd7d1fd33c7be8f54cbc2bc9
>
> giraph-core/src/main/java/org/apache/giraph/partition/RangePartitionerFactory.java
> 29f7898120eafac08ee17327da2072e3ad5a5670
>
> giraph-core/src/main/java/org/apache/giraph/partition/RangeSplitHint.java
> 9634c3386669441d43121e1872c5a90bf47ffcb2
>
> giraph-core/src/main/java/org/apache/giraph/partition/RangeWorkerPartitioner.java
> 5a494a53f0549277a5ed86d033435d0d8cf3d094
>
> giraph-core/src/main/java/org/apache/giraph/partition/ReusesObjectsPartition.java
> 4153d554881f1586545242ba7c00f951630077ca
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
> 9ac2e11e780499dac6ea9235ca73cacaaf7fa4fd
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
> 5772a7b5cd3cf6fa49c09934900630fab3a8094c
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimplePartition.java
> d6a46bd3c178506865b85b8b596d95dde59064b2
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionStore.java
> 74cc3a7d8e719fb1c728e77eaf345e75ca973508
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimpleRangeMasterPartitioner.java
> bf34ecde3dde7fd4d868d396f97e473cbcda76ba
>
> giraph-core/src/main/java/org/apache/giraph/partition/SimpleRangeWorkerPartitioner.java
> f94c14bedc20605ca2bef29d01440423d8ba2a02
>
> giraph-core/src/main/java/org/apache/giraph/partition/WorkerGraphPartitioner.java
> 5a78b1d54109d80d94911971c522c543cecc8e33
>
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java
> 2cfa661ee7204e73e6aa3705f742e9f58acfc4e9
>
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java
> 2c5f2f7f69b06001f004992c396bf49e6fd6785b
>
> giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java
> 0280c58b4e11ed356e77761771db930303f53a80
>
> giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
> 6016ba40241e5b339b44a594e51d6278eaee1967
>
> giraph-core/src/main/java/org/apache/giraph/utils/InMemoryVertexInputFormat.java
> 65d99db8edea57c5c2d75c8c346cead581d5d783
>
> giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java
> be2d2a9c127250fb302da9eedc9af82613d049e2
>   giraph-core/src/main/java/org/apache/giraph/utils/ReflectionUtils.java
> d70eecbd2429d5824b57e057a390a575e8ea83f2
>   giraph-core/src/main/java/org/apache/giraph/utils/TestGraph.java
> 3577a9e4482054e91fb5094ac93207ff7fc44ea5
>   giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java
> 0c9ee076aba989e32288bc9f9f6d182625a3a5cf
>   giraph-core/src/main/java/org/apache/giraph/utils/WritableUtils.java
> c607ca326046e693b8a21b50a968ef6404b56a6d
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
> 03a4876c1cec2d8598a3bb086ca92a91050bb159
>
> giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java
> 351a114390a90528a6f4c1b46dcdb77c50e845aa
>
> giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java
> 33fb5159cc87e0653c0e8933d706159aa97954a4
>
> giraph-core/src/main/java/org/apache/giraph/worker/InputSplitsCallable.java
> a8298c5d9e2c0631ef08dd044eaf0a69ae089e0a
>
> giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java
> 1c292ad1db3d2b0149c3f568c12a7279d831b1ca
>
> giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallableFactory.java
> cf5e8ad85a6663baec90e0d81c6fcffbcdf9e4ef
>
> giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java
> 9a8a8b8bc4de65708a6416e7643ef4b22897fad3
>   giraph-core/src/main/java/org/apache/giraph/worker/WorkerContext.java
> d3ffaeadf5b32f039dc8abe620b1fcf4fcac08e0
>   giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java
> 91b842afbaac2962bbf8f46ec1cabbcfc5b94146
>   giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
> 58aa7d1c63523522f27e375a7e65baf770e5fe05
>   giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java
> f1f8e26e2fe99edea1de6a173bf9aab9ad4d47ca
>   giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
> c27156fb98f86afec352ed1cf70b95e307805d34
>   giraph-core/src/test/java/org/apache/giraph/comm/TestMessageStores.java
> 53b8a2488171bb3c116b9296d5b7467bf890dc8e
>   giraph-core/src/test/java/org/apache/giraph/conf/TestObjectCreation.java
> 9075145eaa7e6744ec656890c8c95ca387ab22e0
>
> giraph-core/src/test/java/org/apache/giraph/graph/TestVertexAndEdges.java
> 15d2bb0aad6f17bca90631dc7b95ea5535dc95f0
>
> giraph-core/src/test/java/org/apache/giraph/io/TestAdjacencyListTextVertexOutputFormat.java
> 6849e3a938bbe20d7e29ad432baf55504d4cf80a
>   giraph-core/src/test/java/org/apache/giraph/io/TestEdgeInput.java
> cb1a8daa94ef235cb22d4d2d2ddaed7d74e79fa2
>   giraph-core/src/test/java/org/apache/giraph/io/TestFilters.java
> 83a366da7f3f7b3c80f055d077394f444bff218d
>
> giraph-core/src/test/java/org/apache/giraph/io/TestIdWithValueTextOutputFormat.java
> ff4d2b1f572049c747becea19fc61ca0e2d4e493
>   giraph-core/src/test/java/org/apache/giraph/io/TestJsonBase64Format.java
> a6aa78a0399dc6c839b5f1334e8e812b73808d7a
>
> giraph-core/src/test/java/org/apache/giraph/io/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java
> 601c824684d7298eedad9093dc83762362482e6c
>
> giraph-core/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java
> 4ee8ea19b30d55cd3c81bb5281f27a15d45a6412
>
> giraph-core/src/test/java/org/apache/giraph/master/TestComputationCombinerTypes.java
> PRE-CREATION
>
> giraph-core/src/test/java/org/apache/giraph/master/TestMasterObserver.java
> 9d6b21536ee85ea6835e55e1e5bb477219105e65
>
> giraph-core/src/test/java/org/apache/giraph/master/TestSwitchClasses.java
> PRE-CREATION
>
> giraph-core/src/test/java/org/apache/giraph/partition/TestGiraphTransferRegulator.java
> 9b655af1f5c252d285862c75cf44be4ae3c29d05
>
> giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java
> 5a93d41e30bb90e3a7f2f2dbce7c44f9d5c8d88c
>
> giraph-core/src/test/java/org/apache/giraph/utils/ComputationCountEdges.java
> PRE-CREATION
>   giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java
> 0039ad62502a740db156c37bcc4445aa5801b6b2
>
> giraph-core/src/test/java/org/apache/giraph/vertices/IntIntNullVertexDoNothing.java
> c98d5804fa3921b23f4fc54a6554b076c8a1a44c
>
> giraph-core/src/test/java/org/apache/giraph/vertices/VertexCountEdges.java
> 9060bc7077ac23a9b362d10698c5facae5dcf55a
>
> giraph-core/src/test/java/org/apache/giraph/vertices/VertexDoNothing.java
> fac3fce3eec0280ac10f65b3f4ebfb166fa84d1a
>   giraph-core/src/test/java/org/apache/giraph/yarn/TestYarnJob.java
> f9d5544edaee5471a4087c9ec81b4a08ddea2a20
>
> giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java
> d08519ba3dc6e5517e99dea640f0f9d16c6594f4
>
> giraph-examples/src/main/java/org/apache/giraph/examples/ConnectedComponentsVertex.java
> dbeb6bf98721cf11744f45682681599f18fe0487
>
> giraph-examples/src/main/java/org/apache/giraph/examples/IdentityVertex.java
> 30cca86547de3806c271099eea87e95429b8f548
>
> giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleDoubleTextInputFormat.java
> 62bea5a7f186d9e731d4fa25f11f238d47d33407
>
> giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleNullTextInputFormat.java
> fdc90505b52a43e70ebc0802ca6af46d6c2c128d
>
> giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleDoubleTextInputFormat.java
> 7dc8475c095207cc7fed199cc05ed80205085549
>
> giraph-examples/src/main/java/org/apache/giraph/examples/PageRankVertex.java
> 9678b31a2747d899888d99933716aa5fe3d40757
>
> giraph-examples/src/main/java/org/apache/giraph/examples/PartitionContextTestVertex.java
> f617d8ea577c1083bf6db72c047dec336f801667
>
> giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertex.java
> 2d2c988b7163b7d3a4584df91e211c1d3f73b873
>
> giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertexMasterCompute.java
> 9e5dbbf9416f771ee957b0d3765c002dced0b2a9
>
> giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkWithRestartVertex.java
> 6f3eb6cdc43b14c2d3776704c9a84178ba12c450
>
> giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkWorkerContext.java
> 2566f43b7fbb91933034df7b3645435807c20fcd
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
> 03d977b1bed88a8dc61498ad44ea7fb895eefe1b
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCombinerVertex.java
> f4475aeef0784bad456cb91fef22d651d7bc4d21
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleFailVertex.java
> c0e206c304e0df2608893beee70fb3d52980f024
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleInDegreeCountVertex.java
> 3bcf3f5324aec428c00454e4d578333e05e6ced8
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleLongDoubleDoubleDoubleIdentityVertex.java
> c7349d15c847b4eedca5a14d073487b97d2cf6eb
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMasterComputeVertex.java
> 8a21ad788ad5b3bf4c24ee63a7d99d9d4448fe30
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMsgVertex.java
> 024fe9d0841a1b6c99ded1d2d2a18e27c6d77858
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
> a468491c6a8acd3256fb9bfe14edc559fe743f4b
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleOutDegreeCountVertex.java
> c830fa2a6060995776e0227a53bbb969779b9b0f
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
> 7a63e8d7052437a9e1d0c859805d5c58b60f289d
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java
> 13d1d7c1e37571fe0207cc93659e26bb4ddad580
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java
> 6f8b352d2e2500390c21d775b662610ce669badb
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTextVertexOutputFormat.java
> 157e6ef6c4e005b20d06b510190ea3070c6d4335
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTriangleClosingVertex.java
> f44cb188117b21e068aedcc6d03b1ef1210b8900
>
> giraph-examples/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
> 8a6f77562b9ffa3c7fa5ae87a9afbf000ef80305
>
> giraph-examples/src/main/java/org/apache/giraph/examples/VerifyMessage.java
> 6e3c589be8d55dbc6dbe4a51c92a77ad282fe09f
>
> giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueDoubleEdgeTextOutputFormat.java
> d3281535b2107ac2eeacd20318459544974b69cd
>
> giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueNullEdgeTextOutputFormat.java
> 85f3556b247a6aa51a5a392e70986377b1fc6e44
>   giraph-examples/src/test/java/org/apache/giraph/TestAutoCheckpoint.java
> 652913b4965a3ccc8f5294bc4976d8f369489807
>   giraph-examples/src/test/java/org/apache/giraph/TestBspBasic.java
> 38ba27fae8c9725545c63ea97b8b7e080cfdb0d3
>
> giraph-examples/src/test/java/org/apache/giraph/TestGraphPartitioner.java
> 12f0d8d65a18b7cf167b59635561f11fcf397a7c
>
> giraph-examples/src/test/java/org/apache/giraph/TestManualCheckpoint.java
> 766e1af3fb838991f757a6d5cc7cf797407d4d64
>   giraph-examples/src/test/java/org/apache/giraph/TestMaxSuperstep.java
> 0989ac5e9fbfe5c1e23484015cbac29478d86bb9
>   giraph-examples/src/test/java/org/apache/giraph/TestMutateGraph.java
> da85fc4fa8e8d309cf5e332975d0a696916ea43e
>
> giraph-examples/src/test/java/org/apache/giraph/TestNotEnoughMapTasks.java
> 6da9c500717b00ed58da21e06a96e06eb873422d
>
> giraph-examples/src/test/java/org/apache/giraph/TestPartitionContext.java
> 4b042dfda701e61df0e79cb117475a090f876ab8
>
> giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
> 53cdeab311b1a5ec2b5ab92fa55c1dca790d8dd5
>
> giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTest.java
> 5d712594ef83d7e0a7501f96b0008cf8a44bae50
>
> giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTestInMemory.java
> e4c029c4ac5bb1fc3b59f5a2af98c353ce330d32
>
> giraph-examples/src/test/java/org/apache/giraph/examples/PageRankVertexTest.java
> 2a392915093cc64ccb5f9d4c037a509bbfe9c0f1
>
> giraph-examples/src/test/java/org/apache/giraph/examples/RandomWalkWithRestartVertexTest.java
> a2a891e28575c858c3208187eef7e4f9c8bf3431
>
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java
> 10065777cf51ac4b02f9ebbbe1f3a77b31ae9eb2
>
> giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java
> bfe8002cde95c823133936438442db218754648c
>
> giraph-examples/src/test/java/org/apache/giraph/examples/TestPageRank.java
> 922e7363caa8b8a90033c9f300a20edc9f093283
>
> giraph-examples/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java
> 23d055e2e9f8e4b9af26354e3f060b97748112a0
>
> giraph-examples/src/test/java/org/apache/giraph/vertex/TestVertexTypes.java
> 0898d635e5a98f7e5ee871e41df2b751c907e1d4
>
> giraph-hbase/src/test/java/org/apache/giraph/io/hbase/TestHBaseRootMarkerVertextFormat.java
> 5091b48ae4f6c01396747ccc67a6c03a0127dce1
>
> giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java
> 5ecb7dde8e8a3a02b3abacf4a002564b7e30e87e
>
> giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeOutputFormat.java
> 859f519f9f4dfdf7d0e590a77e018392ae21d673
>
> giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java
> 2182a7133a7e011453f2f9a38d0e342a42870ad9
>
> giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java
> 08b263fddc2ad16246bd51ef684481fd3567d393
>
> giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexOutputFormat.java
> 37b3b74aecd6b5f7d71e1c5afb8219fe36141261
>   giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
> da9ee2fc048c791388e3f5681f757d50f52674f7
>
> giraph-hive/src/main/java/org/apache/giraph/hive/common/DefaultConfigurableAndTableSchemaAware.java
> 4ba69c9ed474243bedf9cf9fc751d866e3e8ae55
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/AbstractHiveToEdge.java
> 73fc553f1c92ed11eede41d5dc1af636202f5f0b
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeInputFormat.java
> f35b6ea021e9547319561b23fd1bc5fa8dcc734d
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java
> ba992672f20007e4abc949dcc43deb2ce0e219e7
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/AbstractHiveToVertex.java
> b08a09fdcfba9bd6a648f4cd4df7b95093c1615a
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertex.java
> c670876980396093bd46fb20c5c6c94b42c2253b
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexInputFormat.java
> 824d8a6a1ac52915cf44fc6c6ab9e81de793ef39
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexReader.java
> 719d8ad1c200cf74d8bfc8921f1549157c26dbc5
>
> giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/SimpleHiveToVertex.java
> 6df269d451f3470dc6d46d29a2e60dac452a214f
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/AbstractVertexToHive.java
> 52fe93f1375dd7c7278e7f7c93ccf9c950a7ff9d
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexOutputFormat.java
> 05441f4c0aec209b60cd970e94ce5f6116feff0b
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexWriter.java
> 357cf89c95096537f2ee7a99fdce389dbeafb79a
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/SimpleVertexToHive.java
> fdfb3630a293a73036da396c1769e53f11601fe8
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/VertexToHive.java
> 85e438c8d111953c9960f39d7e16c1e92259d7ab
>
> giraph-hive/src/main/java/org/apache/giraph/hive/output/examples/HiveOutputIntIntVertex.java
> 7955915c6b560fbb1c9bdca7e91bc3d8ccba8ab0
>
> giraph-hive/src/test/java/org/apache/giraph/hive/input/HiveEdgeInputTest.java
> 2ad87fa469933337399a2d858ff659822c754f62
>
> giraph-hive/src/test/java/org/apache/giraph/hive/input/HiveVertexInputTest.java
> b6ab139054d5f597809f374454f7443b42f1e8dd
>
> giraph-hive/src/test/java/org/apache/giraph/hive/output/HiveOutputTest.java
> 673c9d503075e99956d8db5696bcd245bbb22ded
>
> giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/VertexCountEdges.java
> a6db2afad47262216758894cd1f1da1b538f4376
>
> giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/VertexSumEdges.java
> d8527b4f693c05fc2629fd4f58354bec50c71bc0
>
> giraph-hive/src/test/java/org/apache/giraph/hive/vertexes/package-info.java
> 552bb91ac4ec2d0f5719970023d2b83ba3be7987
>
> Diff: https://reviews.apache.org/r/11182/diff/
>
>
> Testing
> -------
>
> mvn clean install
> Added some tests for new features
> Real application run on a cluster
>
>
> Thanks,
>
> Maja Kabiljo
>
>


-- 
   Claudio Martella
   claudio.martella@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message