giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Claudio Martella (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-528) Decouple vertex implementation from edge storage
Date Thu, 07 Mar 2013 13:50:15 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595878#comment-13595878
] 

Claudio Martella commented on GIRAPH-528:
-----------------------------------------

Sorry for the delayed review, I'm swamped. Anyway, this is a lot of good work.

One question: this should allow us to avoid having Vertex being Writable. That would allow
us to avoid users to implement write() and readFields() methods, or in general to take care
of serialization (to rephrase it: to export it to the public).
This way we could actually store edges and vertices separated on disk.

Do I understand it correctly?

In that case I'd file to JIRA: one to remove the Writable interface from Vertex, and one to
decouple OOC storage of vertices and edges.
                
> Decouple vertex implementation from edge storage
> ------------------------------------------------
>
>                 Key: GIRAPH-528
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-528
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Alessandro Presta
>            Assignee: Alessandro Presta
>         Attachments: GIRAPH-528.patch, GIRAPH-528.patch, GIRAPH-528.patch, GIRAPH-528.patch,
GIRAPH-528.patch, GIRAPH-528.patch
>
>
> This is meant to address the following issues:
> 1) The Vertex hierarchy is too complex and sometimes hard to work with (Vertex, SimpleVertex,
MutableVertex, SimpleMutableVertex...).
> 2) Changing the underlying edge storage implementation for an existing algorithm requires
editing your vertex to extend a different one.
> 3) In the general case (e.g. when not using ByteArrayVertex with the current EdgeStore),
moving edges from the EdgeStore to the vertices is an additional step that can be avoided.
> My proposal is the following:
> - Make EdgeStore an interface. An implementation should deal with (concurrent) insertion
of edges during input superstep; iteration over a vertex's edges during computation; insertion/deletion
of edges during mutations (optional?); checkpointing.
> - The default EdgeStore will be the current byte-array implementation, which is generic
(works with any choice of <I, V, E, M>) and reasonably optimized.
> - Only one Vertex class, which the user extends for the sole purpose of defining compute().
I don't necessarily agree that it should be an interface, because we still want to provide
methods like getEdges(), sendMessage(), and those should delegate to the EdgeStore/MessageStore
of choice.
> - Switching edge storage implementation is done by passing the EdgeStore class as an
option. One can also define his own ad-hoc EdgeStore (e.g., backed by primitive arrays).
> I think we should also extend this idea to MessageStore, making it possible to override
that functionality too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message