giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Armando" <a.mirag...@student.vu.nl>
Subject Re: Review Request 13756: GIRAPH-732: EdgeOutputFormat API
Date Fri, 23 Aug 2013 13:36:01 GMT


> On Aug. 23, 2013, 12:18 p.m., Claudio Martella wrote:
> > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java, line 216
> > <https://reviews.apache.org/r/13756/diff/2/?file=344102#file344102line216>
> >
> >     i think a default should be set

I don't. Currently users will have this in the main output path and would expect it to be
there. See next point.


> On Aug. 23, 2013, 12:18 p.m., Claudio Martella wrote:
> > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java, line 224
> > <https://reviews.apache.org/r/13756/diff/2/?file=344102#file344102line224>
> >
> >     i think a default should be set

This could be set by default. Related to vertex path, this could be more advisable. In this
situation path/ will contain the verteces and path/edges/ the edges. This can avoid clashes
and mess up the computation if the user forgets to specify the path.


> On Aug. 23, 2013, 12:18 p.m., Claudio Martella wrote:
> > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java, line 1096
> > <https://reviews.apache.org/r/13756/diff/2/?file=344112#file344112line1096>
> >
> >     form? Typo?

yes a typo, I'll fix it.


> On Aug. 23, 2013, 12:18 p.m., Claudio Martella wrote:
> > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java, line 1114
> > <https://reviews.apache.org/r/13756/diff/2/?file=344112#file344112line1114>
> >
> >     shouldn't be save-edges?

nope. I dont' have the statistics for the edges anyway and to avoid adding complexity I used
a similar statistic information since edges are outputted for each vertex.


- Armando


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13756/#review25468
-----------------------------------------------------------


On Aug. 23, 2013, 1:34 a.m., Armando wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13756/
> -----------------------------------------------------------
> 
> (Updated Aug. 23, 2013, 1:34 a.m.)
> 
> 
> Review request for giraph, Armando, Claudio Martella, and Nitay Joffe.
> 
> 
> Bugs: GIRAPH-732
>     https://issues.apache.org/jira/browse/GIRAPH-732
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> This patch is a possible solution to provide an EdgeOutputFormat API in Giraph.
> The idea is to add the same logic as the one for the vertices for edges. For this reason
I add saveEdges after saveVertices.
> This requires the user to have to different paths for the resulting files. For this reason
I have implemented TextEdgeOutputFormat
> and modified TextVertexOutputFormat so that a subdirectory can be specified.
> The code I provide should be compliant with previous versions of hadoop as well as YARN
(I was yet not able to test it on YARN).
> It is also retro-compatible since when not specified, TextVertexOutputFormat behaves
as before.
> I am also providing an actual usable implementation with the associated tests (SrcIdDstIdEdgeValueOutputFormat).
> Also very interesting is the fact that this implementation is totally transparent to
the classes implementing TextVertexOutputFormat.
> 
> passed "maven verify"
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java 1bd79b5 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 71fe885 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 23bcd32 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c276c2a 
>   giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
49a2ebc 
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeOutputFormat.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeWriter.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/GiraphTextOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/SrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/TextEdgeOutputFormat.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java
c91d543 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java 745764b 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java da1e7fb 
>   giraph-core/src/test/java/org/apache/giraph/io/TestSrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/13756/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Armando
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message