giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Re: Review Request: GIRAPH-639: Add support for multiple Vertex/Edge inputs
Date Wed, 17 Apr 2013 04:20:03 GMT


> On April 17, 2013, 12:37 a.m., Alessandro Presta wrote:
> > Can you add a test case with multiple edge input formats?

We have no tests in giraph-hive, and that's the only place where I added a way to set multiple
edge inputs. I am not sure it makes sense to add method to InternalVertexRunner which accepts
several String[], one for each edge input (and then for vertex input to).


> On April 17, 2013, 12:37 a.m., Alessandro Presta wrote:
> > giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java, line 687
> > <https://reviews.apache.org/r/10564/diff/1/?file=281712#file281712line687>
> >
> >     This is going to print "-hiveToEdgeClass=XXX" for multiple classes. I think
this is confusing, seems like the same option is overridden multiple times. Probably better
to rename the option to "-hiveToEdgeClasses" and print the comma-separated list.

I changed the logging a bit, now it will print this for each edge input:

Edge input format:
   hiveToEdgeClass=org.apache.giraph.hive.SomeInputFormat
   edgeInputTable=table_name
   edgeInputFilter="ds='2013-01-01'"

The whole HiveGiraphRunner is a bit messy, I think adding something like BenchmarkOption stuff
I added some time ago could make it much cleaner, but I'll leave that for a separate issue.


> On April 17, 2013, 12:37 a.m., Alessandro Presta wrote:
> > giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java,
line 86
> > <https://reviews.apache.org/r/10564/diff/1/?file=281713#file281713line86>
> >
> >     Is there a way you can map EdgeInputFormat classes to HiveToEdge classes without
going through the index?
> >     The index seems a bit low-level, and also forces Giraph to maintain the ordering.

Can you please give some details about the design which you have in mind?


- Maja


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10564/#review19285
-----------------------------------------------------------


On April 17, 2013, 4:19 a.m., Maja Kabiljo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10564/
> -----------------------------------------------------------
> 
> (Updated April 17, 2013, 4:19 a.m.)
> 
> 
> Review request for giraph.
> 
> 
> Description
> -------
> 
> For now, I did this only for Edge input, once I get some feedback I'll do the exactly
same thing for vertex input.
> Also, I added direct support only to HiveGiraphRunner, we can extend it later to others
as well.
> 
> 
> This addresses bug GIRAPH-639.
>     https://issues.apache.org/jira/browse/GIRAPH-639
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 178c96f 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 4a0e8f7 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 7f9e38e 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 6a5949e 
>   giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
8dfe546 
>   giraph-core/src/main/java/org/apache/giraph/io/InputFormatWithIndex.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java cc6b126

>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java d01dbb4 
>   giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 029cb5d

>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 037cdfc 
>   giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java afb636b

>   giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java
4a1705b 
>   giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java 6e40b7f 
>   giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java f8363b1

>   giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveProfiles.java 892d443 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeInputFormat.java
c482cf0 
>   giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java 09476cd

> 
> Diff: https://reviews.apache.org/r/10564/diff/
> 
> 
> Testing
> -------
> 
> mvn clean verify
> Run application with two edge input tables - verified results.
> 
> 
> Thanks,
> 
> Maja Kabiljo
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message