hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lukas Vlcek" <lukas.vl...@gmail.com>
Subject Re: Finding term vector per host using hadoop
Date Tue, 12 Dec 2006 11:43:32 GMT
Hi,

I have just found the ObjectWritable class in org.apache.hadoop.io package.
However, it does not support any type from java Collection framework. As for
tree-like data structure it is useful to use LinkedList for node childs (as
opposed to fixed size array). This is not directly supported by Haddop as of
now.

Do you think it would be hard to extend the ObjectWritable so that it
handles the Collections as well? Would this be useful feature / contribution
for Hadoop community?

Regards,
Lukas

On 12/12/06, Andrzej Bialecki <ab@getopt.org> wrote:
>
> Lukas Vlcek wrote:
> > Hi,
> >
> > Is there any good example how to wrap compex custom data structure into
> > Writable (or WritableComparable)? I haven't found anything on Wiki.
> >
> > Let's imagine that I need to wrap a tree-like structure (nodes, edges
> and
> > couple of other properties for each node). Is there any existing code in
> > hadoop where can I get inspiration?
>
> These illustrate serialization of Map-like structures:
>
> org.apache.nutch.crawl.MapWritable
> org.apache.nutch.metadata.Metadata
>
> I don't think we have examples of tree-like structures, but the
> serialization parts would look similar, you would just need to traverse
> the tree depth-first.
>
> And if you need to process values stored in several different classes
> you could use ObjectWritable to wrap them.
>
> --
> Best regards,
> Andrzej Bialecki     <><
> ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message