hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-742) Implement of Hama RPC
Date Thu, 18 Jul 2013 08:54:48 GMT

    [ https://issues.apache.org/jira/browse/HAMA-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13712140#comment-13712140
] 

Edward J. Yoon commented on HAMA-742:
-------------------------------------

Avro was used as a default IO serialization protocol[1].

{code}
java.lang.ClassNotFoundException: Class org.apache.avro.io.DatumWriter not found
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1486)
	at org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:70)
	at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:63)
	at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1173)
	at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1103)
	at org.apache.hama.bsp.SequenceFileRecordWriter.<init>(SequenceFileRecordWriter.java:39)
{code}

1. http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/serializer/SerializationFactory.java

So, we need to add avro dependency to lib, or "io.serializations" property to hama-default.xml.

{code}
  <property>
    <name>io.serializations</name>
    <value>org.apache.hadoop.io.serializer.WritableSerialization,org.apache.hadoop.io.serializer.JavaSerialization</value>
  </property>
{code}

I prefer the latter (set "io.serializations" to WritableSerialization).


{quote}why are DistributedCache.add/setLocalFiles(conf, files.toString()); commented out ?{quote}

Oh thanks, my fault! I have to fix this.
                
> Implement of Hama RPC 
> ----------------------
>
>                 Key: HAMA-742
>                 URL: https://issues.apache.org/jira/browse/HAMA-742
>             Project: Hama
>          Issue Type: Sub-task
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.6.3
>
>         Attachments: HAMA-742_v01.patch, HAMA-742_v02.patch
>
>
> To solve HDFS 2.0 compatibility issue, we have to change a lot of codes for Hadoop 2.0
RPC, moreover, yarn RPC doesn't support asynchronous call directly.
> Ultimately, we can pursue the performance and integrate more easily with hadoop multi-versions
by having our own RPC.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message