mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yunming Zhang <zhangyunming1...@gmail.com>
Subject Is there anyway you could easily make a deep copy of Vector Writable class with hadoop's ReflectionUtils?
Date Wed, 19 Dec 2012 01:49:51 GMT
Hi, 

I have been trying to find a way to make a deep copy of key, value pairs inside SequenceFileRecordReader
as I am implementing a getCurrentKeyCopy() and getCurrentValueCopy() method, but I am getting
null pointer exception when calling "value.get()" in CIMapper.java, line 37,  

I want the record reader to create a new data object for the new key value pairs instead of
reusing the original memory location, 

I found a few utility classes in Hadoop that should be able to make a deep copy or clone any
writable class, 

       key = ReflectionUtils.copy(outer.getConfiguration(),
                                   outer.getCurrentKey(), key);
        value = ReflectionUtils.copy(conf, outer.getCurrentValue(), value);

from MultiThreadedMapper and there is also WritableUtils.clone(...) method, 

However, both approaches seems to be failing for VectorWritable, I did notice that it is a
bit different in that it wraps around another Mahout type, Vector, instead of a primitive
type like int, long or string, in the cases of IntWritable and all that, 

I am not sure if this is why the copy or clone utilities in Hadoop designed for Writable is
not working ?

Currently it is giving me null pointer exception when I try to call value.get(), it does seem
we copied something, but the vector inside VectorWritable might have failed to get copied,


Thanks

Yunming
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message