giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Kabiljo" <ikabi...@fb.com>
Subject Re: Review Request 22234: support for partitioned input in giraph
Date Thu, 05 Jun 2014 02:31:28 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22234/#review44791
-----------------------------------------------------------



giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java
<https://reviews.apache.org/r/22234/#comment79329>

    Lets rename it to something meaningful - MappingStoreLongPartitionerFactory
    
    There is no reason to have ByteWritable anywhere in it, just put '?'. 


- Igor Kabiljo


On June 4, 2014, 3:11 p.m., Pavan Kumar Athivarapu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22234/
> -----------------------------------------------------------
> 
> (Updated June 4, 2014, 3:11 p.m.)
> 
> 
> Review request for giraph, Avery Ching, Sergey Edunov, Igor Kabiljo, and Maja Kabiljo.
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> There are many changes here:
> 
> A new input format - MappingInputFormat - and related dependencies has been defined
> New hive input format classes to read Mapping table + some examples which can be used
for running hellopagerank with modifications in test plan have been defined
> Changes to main giraph classes to read the mapping & use it for getting partition
info - in worker, master & partition sections
> New mapping section defined to define & declare MappingStore format & some sample
implementations
> The code can take 2 paths based on what the user wants
> 
> Embed info into vertexId by implementing proper contracts (TranslateEdge.interface +
2 methods in MappingStore)
> then use an EmbeddedGraphPartitioner to directly read worker info off of vertex ids [translate
once & user freely for the rest of app]
> 
> Always read from the MappingStore (this has more overhead probably because of cache misses
in processor) [never translate but pay cost of map lookup each time partition / worker info
is needed]
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ec0ddbb 
>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java bda967d

>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 3337621 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 6b36418 
>   giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
95e029d 
>   giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 
>   giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java 15dbe07 
>   giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java
4200d79 
>   giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java 7cc5651

>   giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java
1eeece7 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
8ab692f 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
2989598 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java
15b0756 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java
600d7a3 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 
>   giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 828eac4

>   giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java e3e04d6

>   giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java
4e19cd2 
>   giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java 603910b 
>   giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java c7ad63b

>   giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java 2388673 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java PRE-CREATION

>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/22234/diff/
> 
> 
> Testing
> -------
> 
> ran pagerank jobs multiple times
> mvn clean verify
> 
> 
> Thanks,
> 
> Pavan Kumar Athivarapu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message