hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Hama graph on top of YARN
Date Wed, 15 May 2013 11:19:37 GMT
Hi,

YARN module is still a alpha version, you can't run a graph job on top
of Hadoop 2.0. Please use Hadoop 1.0 and Hama 0.6.1 at this time.
We'll try to support Hadoop 2.0 soon.

On Wed, May 15, 2013 at 8:13 PM, Rubén González <rubgoncab@gmail.com> wrote:
> Hi all.
>
> I'm trying to run a graph hama process on top of YARN. I suppose that I
> have something wrong but I don't know what :(
>
> I didn't found a lot of documentation about it, in fact I only have seen
> this link
>
> http://wiki.apache.org/hama/GettingStartedYARN
>
> So with this link and, the package of org.apache.hama.graph.GraphJob I
> built my job configuration:
>
>     val conf = new HamaConfiguration()
>     conf.addResource("/etc/hadoop/conf/yarn-site.xml")
>     conf.set("yarn.resourcemanager.address", "0.0.0.0:8040")
>     conf.set("yarn.ipc.rpc.class",
> "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC")
>
>     val job = new YARNBSPJob(conf)
>
>
> job.setBspClass(classOf[GraphJobRunner[LongWritable,LongWritable,LongWritable]])
> //this.setBspClass(GraphJobRunner.class)
>     job.setJarByClass(classOf[UserCompanyMain])
> //this.setJarByClass(exampleClass)
>
>     conf.setClass(GraphJob.VERTEX_CLASS_ATTR, classOf[UserCompanyVertex],
> classOf[Writable])
>     conf.setClass(GraphJob.VERTEX_ID_CLASS_ATTR, classOf[LongWritable],
> classOf[Writable]) //this.setVertexIDClass(Text.class)
>     conf.setClass(GraphJob.VERTEX_VALUE_CLASS_ATTR, classOf[LongWritable],
> classOf[Writable]) //this.setVertexValueClass(IntWritable.class);
>     conf.setClass(GraphJob.VERTEX_EDGE_VALUE_CLASS_ATTR,
> classOf[LongWritable], classOf[Writable])
> //this.setEdgeValueClass(IntWritable.class)
>
>
>     job.setJobName("social_company_network")
>     job.setMemoryUsedPerTaskInMb(50)
>     job.setNumBspTask(2)
>
>     // rest of configuration (input, reader etc…)
>
>     job.waitForCompletion(false)
>
>
>
> (Note: it is scala code for internal reason but I have not problems with it
> in other map/reduce jobs on top of YARN)
>
>
> The code that I run is
>
> -bash-4.1$ yarn jar /tmp/hdp-companies_2.10-0.1.0_SNAPSHOT.jar
> com.hdp.ds.hama.YarnGraph
> 13/05/15 05:40:27 INFO bsp.YARNBSPJob: Connecting to ResourceManager at /
> 0.0.0.0:8040
> 13/05/15 05:40:28 INFO bsp.YARNBSPJob: Starting job...
> 13/05/15 05:40:30 WARN conf.Configuration: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 13/05/15 05:40:30 INFO bsp.YARNBSPJobClient: Submitting job...
> Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162)
> at org.apache.hama.bsp.YARNBSPJobClient.launchJob(YARNBSPJobClient.java:82)
> at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:369)
> at org.apache.hama.bsp.YARNBSPJob.submit(YARNBSPJob.java:80)
> at org.apache.hama.bsp.YARNBSPJob.waitForCompletion(YARNBSPJob.java:95)
> at com.hdp.ds.hama.YarnGraph$.main(YarnSerializePrinting.scala:50)
> at com.hdp.ds.hama.YarnGraph.main(YarnSerializePrinting.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
> Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:398)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:436)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1235)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> at $Proxy7.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159)
> ... 11 more
>
>
> I know that the problem is the protocol, and it is because I added the
> configuration in the HamaConfiguration, but it is not the solution
>     conf.set("yarn.ipc.rpc.class",
> "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC")
>
>
> These are the version that I have installed:
>
> Hadoop  -> 2.0.3.22-alpha
> hama -> 0.6.1
>
>
> Thanks for your help
>   rgc



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message