incubator-giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-28) Introduce new primitive-specific MutableVertex subclasses
Date Mon, 12 Sep 2011 03:45:09 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102412#comment-13102412
] 

Dmitriy V. Ryaboy commented on GIRAPH-28:
-----------------------------------------

There is something that's eating up memory in the LDFDVertex implementation that's not the
OpenLongFloatHashMap.

I changed my code to specifically test consumption of the edge map memory, and removing all
the Vertex complexity, by just putting together 4 implementations of a most basic interface:

{code}
 public static abstract class Vertex {
    public abstract void addEdge(Edge<LongWritable, FloatWritable> edge);
  }
{code}

The 4 implementations are "Current" (a map of LongWritables to Edge<LongWritable, FloatWritable>),
"Just Value" (a map of LongWritable to FloatWritable), "Primitive Map" (an OpenLongFloatHashMap),
and "Primitive" (two arrays of longs and floats, which resize and copy the whole array on
every edge add -- an obviously untenable, but least memory intensive, implementation).

The code is at https://gist.github.com/1210524

The results are much more sensible. Switching to the PrimitiveMap should be huge savings;
even without that, getting rid of the duplicated LongWritable is quite noticeable.

{code}
Current       : 0	144
Just Value    : 0	144
Primitive Map : 0	272
Primitive     : 0	56
Current       : 1	240
Just Value    : 1	216
Primitive Map : 1	272
Primitive     : 1	72
Current       : 10	1104
Just Value    : 10	864
Primitive Map : 10	528
Primitive     : 10	176
Current       : 100	10704
Just Value    : 100	8304
Primitive Map : 100	3728
Primitive     : 100	1256
Current       : 1000	104272
Just Value    : 1000	80272
Primitive Map : 1000	45976
Primitive     : 1000	12056
Current       : 10000	1025616
Just Value    : 10000	785616
Primitive Map : 10000	301192
Primitive     : 10000	120056
{code}


> Introduce new primitive-specific MutableVertex subclasses
> ---------------------------------------------------------
>
>                 Key: GIRAPH-28
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-28
>             Project: Giraph
>          Issue Type: New Feature
>          Components: graph
>    Affects Versions: 0.70.0
>            Reporter: Jake Mannix
>            Assignee: Jake Mannix
>         Attachments: GIRAPH-28.diff
>
>
> As discussed on the list, MutableVertex<LongWritable,DoubleWritable,FloatWritable,DoubleWritable>
(for example) could be highly optimized in its memory footprint if the vertex and edge data
were held in a form which minimized Java object usage.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message