giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavan Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-895) Trim the edges in Giraph
Date Sun, 11 May 2014 17:11:15 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994610#comment-13994610
] 

Pavan Kumar commented on GIRAPH-895:
------------------------------------

Lukas what you are suggesting is unrelated to this change.
In this implementation of ByteArrayEdges he just makes an ArrayCopy, but sure anything that
you write to Byte backed DataOutput formats needs to resize dynamically, if not enough space
is present. Sure, we can make the multiplicative factor configurable. But do u see a lot of
benefit from that. Trimming factor is a trade-off between how many times we create new arrays
(increasing gc pressure in highly dynamic cases) to how much additional space we are willing
to sacrifice. In most places however we initialize the ByteArrays structures with an estimated
size so there is not much resizing. Do you find the contrary happening during actual execution?
If so could you please provide more proof of benefit attainable from another multiplicative
factor? Thanks. we can discuss this on another jira if you like.

> Trim the edges in Giraph
> ------------------------
>
>                 Key: GIRAPH-895
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-895
>             Project: Giraph
>          Issue Type: Improvement
>          Components: graph
>    Affects Versions: 1.1.0
>            Reporter: Sergey Edunov
>             Fix For: 1.1.0
>
>         Attachments: GIRAPH-895.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In many Giraph applications, graphs are immutable, but edges are never trimmed to the
proper size, after input phase. This means that on average we often use 1.5x memory for storing
them. Considering we are often memory bounded, adding an option to trim the edges after the
input phase will help reduce this excess memory usage. For mutable graphs, we can also provide
an option for the same method to be called after each superstep.
> Review request: https://reviews.apache.org/r/21119/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message