giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Kabiljo" <ikabi...@fb.com>
Subject Re: Review Request 22234: support for partitioned input in giraph
Date Wed, 04 Jun 2014 22:53:42 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/22234/#review44768
-----------------------------------------------------------



giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java
<https://reviews.apache.org/r/22234/#comment79311>

    as is, you don't need to override these functions?
    
    Also, it might be better to add:
    byte getEmbededInfo(long id),
    and have implementation of getPartition.
    
    (you could also implement above two methods, through two abstract primitive methods:
    long embedTargetInfo(long id);
    long removeTargetInfo(long id);



giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java
<https://reviews.apache.org/r/22234/#comment79312>

    it is not clear that unset can ever happen - all vertex IDs and edges are translated (with
target being -1 here)
    
    I assume you need to add 1, so that if ID is not in the map is treated the same as if
embedTargetInfo was never called (i.e. user creating vertices with specific IDs later)


- Igor Kabiljo


On June 4, 2014, 3:11 p.m., Pavan Kumar Athivarapu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/22234/
> -----------------------------------------------------------
> 
> (Updated June 4, 2014, 3:11 p.m.)
> 
> 
> Review request for giraph, Avery Ching, Sergey Edunov, Igor Kabiljo, and Maja Kabiljo.
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> There are many changes here:
> 
> A new input format - MappingInputFormat - and related dependencies has been defined
> New hive input format classes to read Mapping table + some examples which can be used
for running hellopagerank with modifications in test plan have been defined
> Changes to main giraph classes to read the mapping & use it for getting partition
info - in worker, master & partition sections
> New mapping section defined to define & declare MappingStore format & some sample
implementations
> The code can take 2 paths based on what the user wants
> 
> Embed info into vertexId by implementing proper contracts (TranslateEdge.interface +
2 methods in MappingStore)
> then use an EmbeddedGraphPartitioner to directly read worker info off of vertex ids [translate
once & user freely for the rest of app]
> 
> Always read from the MappingStore (this has more overhead probably because of cache misses
in processor) [never translate but pay cost of map lookup each time partition / worker info
is needed]
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java ec0ddbb 
>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java bda967d

>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 3337621 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 6b36418 
>   giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
95e029d 
>   giraph-core/src/main/java/org/apache/giraph/io/MappingInputFormat.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/MappingReader.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingInputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedMappingReader.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/iterables/MappingReaderWrapper.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/AbstractLongByteOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/DefaultEmbeddedLongByteOps.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/DefaultLongByteOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/LongByteMappingStore.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingEntry.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingStore.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/MappingStoreOps.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/package-info.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/LongByteTranslateEdge.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/TranslateEdge.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/mapping/translate/package-info.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 90dc9f3 
>   giraph-core/src/main/java/org/apache/giraph/master/MasterThread.java 15dbe07 
>   giraph-core/src/main/java/org/apache/giraph/partition/DefaultLongBytePartitionerFactory.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/partition/GraphPartitionerFactory.java
4200d79 
>   giraph-core/src/main/java/org/apache/giraph/partition/HashPartitionerFactory.java 7cc5651

>   giraph-core/src/main/java/org/apache/giraph/partition/HashRangePartitionerFactory.java
1eeece7 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleIntRangePartitionerFactory.java
8ab692f 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleLongRangePartitionerFactory.java
2989598 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimplePartitionerFactory.java
15b0756 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java
600d7a3 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java aff7084 
>   giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 828eac4

>   giraph-core/src/main/java/org/apache/giraph/worker/FullInputSplitCallable.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/worker/LocalData.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallable.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/MappingInputSplitsCallableFactory.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java e3e04d6

>   giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java
4e19cd2 
>   giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java 603910b 
>   giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java c7ad63b

>   giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveUtils.java 2388673 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/AbstractHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingInputFormat.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveMappingReader.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/HiveToMapping.java PRE-CREATION

>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/SimpleHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongByteHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/LongInt2ByteHiveToMapping.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/examples/package-info.java
PRE-CREATION 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/mapping/package-info.java PRE-CREATION

> 
> Diff: https://reviews.apache.org/r/22234/diff/
> 
> 
> Testing
> -------
> 
> ran pagerank jobs multiple times
> mvn clean verify
> 
> 
> Thanks,
> 
> Pavan Kumar Athivarapu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message