mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <>
Subject Re: Is there anyway you could easily make a deep copy of Vector Writable class with hadoop's ReflectionUtils?
Date Wed, 19 Dec 2012 01:59:50 GMT
You can always just call clone() on the vector inside the VectorWritable.

On Tue, Dec 18, 2012 at 5:49 PM, Yunming Zhang

> Hi,
> I have been trying to find a way to make a deep copy of key, value pairs
> inside SequenceFileRecordReader as I am implementing a getCurrentKeyCopy()
> and getCurrentValueCopy() method, but I am getting null pointer exception
> when calling "value.get()" in, line 37,
> I want the record reader to create a new data object for the new key value
> pairs instead of reusing the original memory location,
> I found a few utility classes in Hadoop that should be able to make a deep
> copy or clone any writable class,
>        key = ReflectionUtils.copy(outer.getConfiguration(),
>                                    outer.getCurrentKey(), key);
>         value = ReflectionUtils.copy(conf, outer.getCurrentValue(), value);
> from MultiThreadedMapper and there is also WritableUtils.clone(...) method,
> However, both approaches seems to be failing for VectorWritable, I did
> notice that it is a bit different in that it wraps around another Mahout
> type, Vector, instead of a primitive type like int, long or string, in the
> cases of IntWritable and all that,
> I am not sure if this is why the copy or clone utilities in Hadoop
> designed for Writable is not working ?
> Currently it is giving me null pointer exception when I try to call
> value.get(), it does seem we copied something, but the vector inside
> VectorWritable might have failed to get copied,
> Thanks
> Yunming

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message