giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lukas Nalezenec (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-895) Trim the edges in Giraph
Date Sat, 10 May 2014 22:05:49 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994305#comment-13994305
] 

Lukas Nalezenec commented on GIRAPH-895:
----------------------------------------

I could be wrong - I think there is same problem in ByteArrayPartition (and other similar
classes)..
When you call  ByteArrayPartition.saveVertex() it calls  WritableUtils.writeVertexToByteArray
and then it works with UnsafeByteArrayOutputStream methods.

The outputstream allocates memory in method ensureSize() in multiples of 2:

if (pos + size > buf.length) {
      byte[] newBuf = new byte[(buf.length + size) << 1];
}

and this array is saved to map in ByteArrayPartition.

I am right ?





> Trim the edges in Giraph
> ------------------------
>
>                 Key: GIRAPH-895
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-895
>             Project: Giraph
>          Issue Type: Improvement
>          Components: graph
>    Affects Versions: 1.1.0
>            Reporter: Sergey Edunov
>             Fix For: 1.1.0
>
>         Attachments: GIRAPH-895.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In many Giraph applications, graphs are immutable, but edges are never trimmed to the
proper size, after input phase. This means that on average we often use 1.5x memory for storing
them. Considering we are often memory bounded, adding an option to trim the edges after the
input phase will help reduce this excess memory usage. For mutable graphs, we can also provide
an option for the same method to be called after each superstep.
> Review request: https://reviews.apache.org/r/21119/



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message