giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-908) support for partitioned input in giraph
Date Sun, 08 Jun 2014 19:01:02 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14021265#comment-14021265
] 

Hudson commented on GIRAPH-908:
-------------------------------

ABORTED: Integrated in Giraph-trunk-Commit #1445 (See [https://builds.apache.org/job/Giraph-trunk-Commit/1445/])
GIRAPH-908: support for partitioned input in giraph (pavanka) (pavanka: http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=4a133f5766c09362917e0416af503c0a00b24e87)
* giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java
* giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
* giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java
* giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java
* giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java
* giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java
* giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/partition/LongMappingStorePartitionerFactory.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java
* giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java
* giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java
* giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java
* giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java
* giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java
* giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java
* giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java
* giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java
* giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java
* giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java
* giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java
* giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java
* giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java
* giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java
* giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java
* giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java
* giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
* giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
* giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java
* CHANGELOG
* giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java
* giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java
* giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java
* giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java
* giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java
* giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java
* giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java
* giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java
* giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java
* giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java
* giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java
* giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java
* giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java
* giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java
* giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java


> support for partitioned input in giraph
> ---------------------------------------
>
>                 Key: GIRAPH-908
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-908
>             Project: Giraph
>          Issue Type: Sub-task
>            Reporter: Pavan Kumar
>            Assignee: Pavan Kumar
>         Attachments: GIRAPH-908.patch
>
>
> When the graph we need to work on is already partitioned into a fixed number of buckets,
with properties such as high edge-locality, low fan-out to other buckets, etc.  [for instance
using techniques such as https://people.cam.cornell.edu/~jugander/papers/wsdm13-blp.pdf ]
> we should be able to partition our graph based on such a mapping, to improve local-requests,
etc. thus avoiding huge network communication. this diff is especially useful we repeatedly
run algorithms on top of the same graph. In such case, we can compute the partitioning once
& then use it to speed up processing, (also requiring less network bandwidth, etc.) for
the rest applications on same/ similar graphs.
> The diff is big & partly reviewed by my colleagues. Putting it up for review



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message