giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Armando" <a.mirag...@student.vu.nl>
Subject Re: Review Request 13756: GIRAPH-732: EdgeOutputFormat API
Date Mon, 26 Aug 2013 11:04:26 GMT


> On Aug. 25, 2013, 8:30 p.m., Nitay Joffe wrote:
> > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java, line 1099
> > <https://reviews.apache.org/r/13756/diff/2/?file=344112#file344112line1099>
> >
> >     Should we track and add something in here about number of edges saved instead
of just vertices, seeing as this is all about edge writing?

something like this is acceptable?

LoggerUtils.setStatusAndLog(getContext(), LOG, Level.INFO,         
                        "saveEdges: Saved " + edges +                                  
                        " edges out of " + partition.getEdgeCount() +                  
                        " partition edges, on partition " + partitionIndex +           
                        " out of " + numPartitions);


- Armando


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13756/#review25523
-----------------------------------------------------------


On Aug. 23, 2013, 1:34 a.m., Armando wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13756/
> -----------------------------------------------------------
> 
> (Updated Aug. 23, 2013, 1:34 a.m.)
> 
> 
> Review request for giraph, Armando, Claudio Martella, and Nitay Joffe.
> 
> 
> Bugs: GIRAPH-732
>     https://issues.apache.org/jira/browse/GIRAPH-732
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> This patch is a possible solution to provide an EdgeOutputFormat API in Giraph.
> The idea is to add the same logic as the one for the vertices for edges. For this reason
I add saveEdges after saveVertices.
> This requires the user to have to different paths for the resulting files. For this reason
I have implemented TextEdgeOutputFormat
> and modified TextVertexOutputFormat so that a subdirectory can be specified.
> The code I provide should be compliant with previous versions of hadoop as well as YARN
(I was yet not able to test it on YARN).
> It is also retro-compatible since when not specified, TextVertexOutputFormat behaves
as before.
> I am also providing an actual usable implementation with the associated tests (SrcIdDstIdEdgeValueOutputFormat).
> Also very interesting is the fact that this implementation is totally transparent to
the classes implementing TextVertexOutputFormat.
> 
> passed "maven verify"
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java 1bd79b5 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 71fe885 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 23bcd32 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java c276c2a 
>   giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java
49a2ebc 
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeOutputFormat.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeWriter.java PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/GiraphTextOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/SrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/io/formats/TextEdgeOutputFormat.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/io/formats/TextVertexOutputFormat.java
c91d543 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeOutputFormat.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java 745764b 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java da1e7fb 
>   giraph-core/src/test/java/org/apache/giraph/io/TestSrcIdDstIdEdgeValueTextOutputFormat.java
PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/13756/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Armando
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message