giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-895) Trim the edges in Giraph
Date Tue, 06 May 2014 22:12:15 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991224#comment-13991224
] 

Pavan Kumar commented on GIRAPH-895:
------------------------------------

ArrayListEdges: trimToSize() does this:   elementData = Arrays.copyOf(elementData, size);
so during loading phase you end up using more resident memory & gc pressure increases
did u test this change for any memory bound application

ByteArrayEdges: uses ExtendedDataOutput which can have upto 2 times more memory than what
it stores
so you might want to add trim here as well with considerations written for ArrayListEdges

EdgeStore / Vertex
vertex.trimEdges => all vertex classes should now support trimEdges, this is a breaking
change

Trimmable.java - is generic so better to move it out of edges

for all changes in graph/* 
similar concern -> vertex.trimEdges

the changes made in io/formats are good. you can trim edges where even & whenever they
can be trimmed
without imposing that vertex be trimmable
 

> Trim the edges in Giraph
> ------------------------
>
>                 Key: GIRAPH-895
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-895
>             Project: Giraph
>          Issue Type: Improvement
>          Components: graph
>    Affects Versions: 1.1.0
>            Reporter: Sergey Edunov
>             Fix For: 1.1.0
>
>         Attachments: GIRAPH-895.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In many Giraph applications, graphs are immutable, but edges are never trimmed to the
proper size, after input phase. This means that on average we often use 1.5x memory for storing
them. Considering we are often memory bounded, adding an option to trim the edges after the
input phase will help reduce this excess memory usage. For mutable graphs, we can also provide
an option for the same method to be called after each superstep.
> Review request: https://reviews.apache.org/r/21119/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message