giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-528) Decouple vertex implementation from edge storage
Date Thu, 07 Mar 2013 06:12:15 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595591#comment-13595591
] 

Hudson commented on GIRAPH-528:
-------------------------------

Integrated in Giraph-trunk-Commit #784 (See [https://builds.apache.org/job/Giraph-trunk-Commit/784/])
    GIRAPH-528: Decouple vertex implementation from edge storage (apresta) (Revision 3f5009aea0797563304dd8ba1aa6fb06ebaab83b)

     Result = FAILURE
alessandro : http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=3f5009aea0797563304dd8ba1aa6fb06ebaab83b
Files : 
* giraph-core/src/test/java/org/apache/giraph/vertex/TestIntIntNullIntVertex.java
* giraph-core/src/test/java/org/apache/giraph/graph/TestVertexAndEdges.java
* giraph-examples/src/test/java/org/apache/giraph/examples/SimpleTriangleClosingVertexTest.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsVertex.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableIntArrayIterator.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomVertexInputFormat.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTriangleClosingVertex.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java
* giraph-core/src/test/java/org/apache/giraph/utils/ComparisonUtilsTest.java
* giraph-examples/src/main/java/org/apache/giraph/examples/ConnectedComponentsVertex.java
* giraph-core/src/test/java/org/apache/giraph/vertex/TestMultiGraphVertex.java
* giraph-core/src/test/java/org/apache/giraph/edge/TestMultiGraphEdges.java
* giraph-core/src/test/java/org/apache/giraph/io/TestLongDoubleDoubleAdjacencyListVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/TextDoubleDoubleAdjacencyListVertexInputFormat.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleOutDegreeCountVertex.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/HashMultimapEdges.java
* giraph-hive/src/main/java/org/apache/giraph/hive/output/HiveVertexWriter.java
* giraph-core/src/main/java/org/apache/giraph/io/VertexWriter.java
* giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleDoubleDoubleTextInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/graph/GiraphTransferRegulator.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleShortestPathsVertex.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/RepresentativeVertexPageRankBenchmark.java
* giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/TestAccumuloVertexFormat.java
* giraph-core/src/main/java/org/apache/giraph/worker/DefaultWorkerContext.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java
* giraph-examples/src/test/java/org/apache/giraph/vertex/TestVertexTypes.java
* giraph-core/src/main/java/org/apache/giraph/partition/SimplePartition.java
* giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankComputation.java
* giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java
* giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayEdges.java
* giraph-core/src/main/java/org/apache/giraph/edge/EdgeNoValue.java
* giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphEdgeListVertexPageRankBenchmark.java
* giraph-examples/pom.xml
* giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/ByteArrayEdges.java
* giraph-core/src/test/java/org/apache/giraph/comm/ConnectionTest.java
* giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
* giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/vertex/EdgeListVertexBase.java
* giraph-core/src/main/java/org/apache/giraph/partition/DiskBackedPartitionStore.java
* giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueDoubleEdgeTextOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/vertex/package-info.java
* giraph-core/src/main/java/org/apache/giraph/edge/LongNullArrayEdges.java
* giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleFloatDoubleVertex.java
* giraph-core/src/main/java/org/apache/giraph/vertex/SimpleMutableVertex.java
* giraph-core/src/main/java/org/apache/giraph/vertex/HashMapVertex.java
* giraph-core/src/test/java/org/apache/giraph/edge/TestStrictRandomAccessEdges.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java
* giraph-core/src/main/java/org/apache/giraph/graph/DefaultVertexResolver.java
* giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleDoubleDoubleTextInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongNullEdgeArrayIterable.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/ByteArrayVertexPageRankBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/edge/HashMapEdges.java
* giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java
* giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphEdgeListVertex.java
* giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexOutputFormat.java
* giraph-examples/src/test/java/org/apache/giraph/examples/SimpleShortestPathsVertexTest.java
* giraph-core/src/main/java/org/apache/giraph/edge/Edge.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/IdWithValueTextOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/IntNullNullNullTextInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/comm/messages/SequentialFileMessageStore.java
* giraph-core/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java
* giraph-core/src/main/java/org/apache/giraph/utils/EdgeComparator.java
* giraph-core/src/test/java/org/apache/giraph/vertex/TestMutableVertex.java
* giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java
* giraph-core/src/main/java/org/apache/giraph/edge/EdgeFactory.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/AdjacencyListTextVertexOutputFormat.java
* giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleInDegreeCountVertex.java
* giraph-core/src/test/java/org/apache/giraph/comm/TestMessageStores.java
* giraph-core/src/main/java/org/apache/giraph/comm/SendMutationsCache.java
* giraph-examples/src/test/java/org/apache/giraph/examples/ConnectedComponentsVertexTest.java
* giraph-core/src/main/java/org/apache/giraph/edge/ArrayListEdges.java
* giraph-core/src/main/java/org/apache/giraph/vertex/MutableVertex.java
* giraph-core/src/test/java/org/apache/giraph/io/TestEdgeInput.java
* giraph-core/src/main/java/org/apache/giraph/graph/VertexResolver.java
* giraph-core/src/main/java/org/apache/giraph/comm/MsgList.java
* giraph-examples/src/test/java/org/apache/giraph/TestBspBasic.java
* giraph-core/src/main/java/org/apache/giraph/edge/MultiRandomAccessVertexEdges.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/TextEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongFloatEdgeArrayIterable.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/edge/LongDoubleHashMapEdges.java
* giraph-core/src/main/java/org/apache/giraph/comm/WorkerClientRequestProcessor.java
* giraph-hbase/src/test/java/org/apache/giraph/io/hbase/TestHBaseRootMarkerVertextFormat.java
* giraph-core/src/main/java/org/apache/giraph/graph/Edge.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java
* giraph-hbase/src/test/java/org/apache/giraph/io/hbase/edgemarker/TableEdgeOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/io/VertexValueReader.java
* giraph-examples/src/main/java/org/apache/giraph/examples/LongDoubleFloatDoubleTextInputFormat.java
* giraph-core/src/test/java/org/apache/giraph/BspCase.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMasterComputeVertex.java
* giraph-hive/src/main/java/org/apache/giraph/hive/output/VertexToHive.java
* giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java
* giraph-core/src/main/java/org/apache/giraph/bsp/BspOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/AggregatorsBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/utils/ComparisonUtils.java
* giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java
* giraph-core/src/main/java/org/apache/giraph/edge/MutableEdge.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/DefaultEdge.java
* giraph-core/src/main/java/org/apache/giraph/edge/StrictRandomAccessVertexEdges.java
* giraph-core/src/main/java/org/apache/giraph/utils/EmptyIterable.java
* giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveToVertex.java
* giraph-examples/src/main/java/org/apache/giraph/examples/VertexWithDoubleValueFloatEdgeTextOutputFormat.java
* giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/partition/Partition.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java
* giraph-core/src/main/java/org/apache/giraph/graph/VertexChanges.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableDoubleArrayIterator.java
* giraph-core/src/test/java/org/apache/giraph/bsp/BspUtilsTest.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java
* giraph-core/src/main/java/org/apache/giraph/vertex/RepresentativeVertex.java
* giraph-core/src/main/java/org/apache/giraph/vertex/SimpleVertex.java
* pom.xml
* giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatGiraphRunner.java
* giraph-core/src/main/java/org/apache/giraph/edge/package-info.java
* giraph-core/src/main/java/org/apache/giraph/vertex/EdgeListVertex.java
* giraph-examples/src/test/java/org/apache/giraph/examples/TryMultiIpcBindingPortsTest.java
* giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java
* giraph-core/src/test/java/org/apache/giraph/partition/TestGiraphTransferRegulator.java
* giraph-core/src/main/java/org/apache/giraph/partition/ByteArrayPartition.java
* giraph-core/src/main/java/org/apache/giraph/utils/ReflectionUtils.java
* giraph-core/src/test/java/org/apache/giraph/conf/TestObjectCreation.java
* giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/IntIntNullIntTextInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
* giraph-core/src/main/java/org/apache/giraph/graph/EdgeStore.java
* giraph-accumulo/src/test/java/org/apache/giraph/io/accumulo/edgemarker/AccumuloEdgeOutputFormat.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleFailVertex.java
* giraph-core/src/test/java/org/apache/giraph/master/TestMasterObserver.java
* giraph-core/src/main/java/org/apache/giraph/comm/SendPartitionCache.java
* giraph-core/src/test/java/org/apache/giraph/edge/TestMultiRandomAccessEdges.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphRepresentativeVertexPageRankBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java
* giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleFloatDoubleEdgeListVertex.java
* giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java
* giraph-core/src/test/java/org/apache/giraph/partition/TestPartitionStores.java
* giraph-core/src/main/java/org/apache/giraph/graph/ComputeCallable.java
* giraph-examples/src/main/java/org/apache/giraph/examples/PartitionContextTestVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java
* giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
* giraph-core/src/test/java/org/apache/giraph/edge/TestStrictGraphEdges.java
* giraph-core/src/test/java/org/apache/giraph/edge/TestNullValueEdges.java
* giraph-core/src/main/java/org/apache/giraph/graph/VertexMutations.java
* giraph-core/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java
* giraph-core/src/main/java/org/apache/giraph/io/ReverseEdgeDuplicator.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleCombinerVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/ReuseObjectsVertexEdges.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/vertex/ByteArrayVertex.java
* giraph-core/src/main/java/org/apache/giraph/graph/MutableEdge.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/PseudoRandomEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/edge/VertexEdges.java
* giraph-examples/src/main/java/org/apache/giraph/examples/NormalizingLongDoubleFloatDoubleTextInputFormat.java
* giraph-core/src/test/java/org/apache/giraph/io/TestTextDoubleDoubleAdjacencyListVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java
* giraph-examples/src/main/java/org/apache/giraph/examples/IdentityVertex.java
* giraph-core/src/main/java/org/apache/giraph/edge/LongNullHashSetEdges.java
* giraph-core/src/main/java/org/apache/giraph/graph/DefaultEdge.java
* giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/JsonBase64VertexInputFormat.java
* giraph-core/pom.xml
* giraph-core/src/main/java/org/apache/giraph/utils/EdgeIterables.java
* giraph-core/src/test/java/org/apache/giraph/io/TestJsonBase64Format.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/SequenceFileVertexOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/graph/ReverseEdgeDuplicator.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/JsonLongDoubleFloatDoubleVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/utils/WritableUtils.java
* giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java
* giraph-core/src/main/java/org/apache/giraph/vertex/IntNullNullNullVertex.java
* giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java
* giraph-core/src/main/java/org/apache/giraph/graph/EdgeNoValue.java
* giraph-core/src/main/java/org/apache/giraph/vertex/Vertex.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/MultiGraphByteArrayVertexPageRankBenchmark.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleMsgVertex.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/LongDoubleDoubleAdjacencyListVertexInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/graph/Vertex.java
* CHANGELOG
* giraph-core/src/main/java/org/apache/giraph/benchmark/EdgeListVertexPageRankBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/HashMapVertexShortestPathsBenchmark.java
* giraph-examples/src/main/java/org/apache/giraph/examples/VerifyMessage.java
* giraph-core/src/main/java/org/apache/giraph/utils/UnmodifiableLongArrayIterator.java
* giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java
* giraph-examples/src/main/java/org/apache/giraph/examples/RandomWalkVertex.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleSuperstepVertex.java
* giraph-core/src/main/java/org/apache/giraph/vertex/IntIntNullIntVertex.java
* giraph-core/src/main/java/org/apache/giraph/vertex/LongDoubleNullDoubleVertex.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/ShortestPathsComputation.java
* giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphRepresentativeVertex.java
* giraph-examples/src/test/java/org/apache/giraph/examples/RandomWalkWithRestartVertexTest.java
* giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java
* giraph-core/src/test/java/org/apache/giraph/io/TestAdjacencyListTextVertexOutputFormat.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/vertex/HiveVertexReader.java
* giraph-hcatalog/src/main/java/org/apache/giraph/io/hcatalog/HCatalogEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/edge/LongDoubleArrayEdges.java
* giraph-core/src/main/java/org/apache/giraph/vertex/MultiGraphByteArrayVertex.java
* giraph-core/src/main/java/org/apache/giraph/io/formats/IntNullReverseTextEdgeInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/edge/ConfigurableVertexEdges.java
* giraph-examples/src/main/java/org/apache/giraph/examples/SimpleTextVertexOutputFormat.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java
* giraph-core/src/main/java/org/apache/giraph/vertex/ByteArrayVertexBase.java
* giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java
* giraph-core/src/test/java/org/apache/giraph/io/TestIdWithValueTextOutputFormat.java
* giraph-core/src/main/java/org/apache/giraph/bsp/BspUtils.java
* giraph-core/src/main/java/org/apache/giraph/vertex/RepresentativeVertexBase.java
* giraph-core/src/main/java/org/apache/giraph/benchmark/HashMapVertexPageRankBenchmark.java
* giraph-core/src/main/java/org/apache/giraph/graph/EdgeFactory.java

                
> Decouple vertex implementation from edge storage
> ------------------------------------------------
>
>                 Key: GIRAPH-528
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-528
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-528.patch, GIRAPH-528.patch, GIRAPH-528.patch, GIRAPH-528.patch,
GIRAPH-528.patch, GIRAPH-528.patch
>
>
> This is meant to address the following issues:
> 1) The Vertex hierarchy is too complex and sometimes hard to work with (Vertex, SimpleVertex,
MutableVertex, SimpleMutableVertex...).
> 2) Changing the underlying edge storage implementation for an existing algorithm requires
editing your vertex to extend a different one.
> 3) In the general case (e.g. when not using ByteArrayVertex with the current EdgeStore),
moving edges from the EdgeStore to the vertices is an additional step that can be avoided.
> My proposal is the following:
> - Make EdgeStore an interface. An implementation should deal with (concurrent) insertion
of edges during input superstep; iteration over a vertex's edges during computation; insertion/deletion
of edges during mutations (optional?); checkpointing.
> - The default EdgeStore will be the current byte-array implementation, which is generic
(works with any choice of <I, V, E, M>) and reasonably optimized.
> - Only one Vertex class, which the user extends for the sole purpose of defining compute().
I don't necessarily agree that it should be an interface, because we still want to provide
methods like getEdges(), sendMessage(), and those should delegate to the EdgeStore/MessageStore
of choice.
> - Switching edge storage implementation is done by passing the EdgeStore class as an
option. One can also define his own ad-hoc EdgeStore (e.g., backed by primitive arrays).
> I think we should also extend this idea to MessageStore, making it possible to override
that functionality too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message