hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From madhu phatak <phatak....@gmail.com>
Subject Custom data structures in Hadoop
Date Mon, 05 Mar 2012 05:33:04 GMT
 I have a following issue in Hadoop 0.20.2. When i try to use inheritance
with WritableComparables the job is failing. Example If i create a base
writable called as shape

  public abstract class ShapeWritable<T> implements WritableComparable<T>


 Then extend this for a concrete class called CircleWritable . Now if my
mapper output is set as ShapeWritable in job configuration  and I write a
CircleWriable in collect.write() , the map fails with class mismatch. When
i looked into source code of MapTask.java , i saw the following code

*public** **synchronized** **void* collect(K key, V value,* **int* partition

                                     )* **throws* IOException {


     * **if* (key.getClass() != keyClass) {

       * **throw** **new* IOException("Type mismatch in key from map:
expected "

                              + keyClass.getName() + ", recieved "

                              + key.getClass().getName());


     * **if* (value.getClass() != valClass) {

       * **throw** **new* IOException("Type mismatch in value from map:
expected "

                              + valClass.getName() + ", recieved "

                              + value.getClass().getName());

Here , we are matching directly classes which means inheritance don't work.
Can anyone tell why its implemented in this way? Is it changed in newer
version of Hadoop?

Join me at http://hadoopworkshop.eventbrite.com/

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message