giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Igor Kabiljo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-921) Create ByteValueVertex to store vertex values as bytes without object instance
Date Wed, 18 Jun 2014 20:00:25 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036277#comment-14036277
] 

Igor Kabiljo commented on GIRAPH-921:
-------------------------------------

Can you create a diff from patch? (it's easier to view)

I don't think you need any converters, because VertexValue needs to implement Writable anyways,
so you should use that for serializing/deserializing. (similarly to how ByteArrayPartition
does it).

Note that you shouldn't rely on setValue being called, i.e. it should work with flow:
vertex.getValue().set(15)
where V is IntWritable.

> Create ByteValueVertex to store vertex values as bytes without object instance
> ------------------------------------------------------------------------------
>
>                 Key: GIRAPH-921
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-921
>             Project: Giraph
>          Issue Type: Improvement
>          Components: graph
>    Affects Versions: 1.1.0
>            Reporter: Aapo Kyrola
>            Priority: Minor
>         Attachments: 0001-Created-ByteValueVertex-and-two-byte-to-value-conver.patch
>
>
> Facebook task (bootcamped) #4320492
> This modification creates an alternative to DefaultVertex that stores the vertex value
as raw bytes and only realizes the object when getValue() is called. The purpose of this change
is to reduce memory consumption when vertex values are large.
> User must define their own BytesToValueConverter class to translate the object type to
bytes and back.  I created ones for float and doubles. Using Java's serialization is bad idea
as it adds a lot of overhead (for example full class names). However, one could easily create
a generic bytevalueconverter that uses java serialization.
> Configuration -vcl is used to define the vertex class and -bc the converter. For example:
>  -vcl org.apache.giraph.graph.ByteValueVertex -bc org.apache.giraph.graph.byteconverters.DoubleConverter



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message