mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Is there anyway you could easily make a deep copy of Vector Writable class with hadoop's ReflectionUtils?
Date Wed, 19 Dec 2012 01:59:50 GMT
You can always just call clone() on the vector inside the VectorWritable.

On Tue, Dec 18, 2012 at 5:49 PM, Yunming Zhang
<zhangyunming1990@gmail.com>wrote:

> Hi,
>
> I have been trying to find a way to make a deep copy of key, value pairs
> inside SequenceFileRecordReader as I am implementing a getCurrentKeyCopy()
> and getCurrentValueCopy() method, but I am getting null pointer exception
> when calling "value.get()" in CIMapper.java, line 37,
>
> I want the record reader to create a new data object for the new key value
> pairs instead of reusing the original memory location,
>
> I found a few utility classes in Hadoop that should be able to make a deep
> copy or clone any writable class,
>
>        key = ReflectionUtils.copy(outer.getConfiguration(),
>                                    outer.getCurrentKey(), key);
>         value = ReflectionUtils.copy(conf, outer.getCurrentValue(), value);
>
> from MultiThreadedMapper and there is also WritableUtils.clone(...) method,
>
> However, both approaches seems to be failing for VectorWritable, I did
> notice that it is a bit different in that it wraps around another Mahout
> type, Vector, instead of a primitive type like int, long or string, in the
> cases of IntWritable and all that,
>
> I am not sure if this is why the copy or clone utilities in Hadoop
> designed for Writable is not working ?
>
> Currently it is giving me null pointer exception when I try to call
> value.get(), it does seem we copied something, but the vector inside
> VectorWritable might have failed to get copied,
>
> Thanks
>
> Yunming

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message