hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rubén González <rubgon...@gmail.com>
Subject Hama graph on top of YARN
Date Wed, 15 May 2013 11:13:08 GMT
Hi all.

I'm trying to run a graph hama process on top of YARN. I suppose that I
have something wrong but I don't know what :(

I didn't found a lot of documentation about it, in fact I only have seen
this link

http://wiki.apache.org/hama/GettingStartedYARN

So with this link and, the package of org.apache.hama.graph.GraphJob I
built my job configuration:

    val conf = new HamaConfiguration()
    conf.addResource("/etc/hadoop/conf/yarn-site.xml")
    conf.set("yarn.resourcemanager.address", "0.0.0.0:8040")
    conf.set("yarn.ipc.rpc.class",
"org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC")

    val job = new YARNBSPJob(conf)


job.setBspClass(classOf[GraphJobRunner[LongWritable,LongWritable,LongWritable]])
//this.setBspClass(GraphJobRunner.class)
    job.setJarByClass(classOf[UserCompanyMain])
//this.setJarByClass(exampleClass)

    conf.setClass(GraphJob.VERTEX_CLASS_ATTR, classOf[UserCompanyVertex],
classOf[Writable])
    conf.setClass(GraphJob.VERTEX_ID_CLASS_ATTR, classOf[LongWritable],
classOf[Writable]) //this.setVertexIDClass(Text.class)
    conf.setClass(GraphJob.VERTEX_VALUE_CLASS_ATTR, classOf[LongWritable],
classOf[Writable]) //this.setVertexValueClass(IntWritable.class);
    conf.setClass(GraphJob.VERTEX_EDGE_VALUE_CLASS_ATTR,
classOf[LongWritable], classOf[Writable])
//this.setEdgeValueClass(IntWritable.class)


    job.setJobName("social_company_network")
    job.setMemoryUsedPerTaskInMb(50)
    job.setNumBspTask(2)

    // rest of configuration (input, reader etc…)

    job.waitForCompletion(false)



(Note: it is scala code for internal reason but I have not problems with it
in other map/reduce jobs on top of YARN)


The code that I run is

-bash-4.1$ yarn jar /tmp/hdp-companies_2.10-0.1.0_SNAPSHOT.jar
com.hdp.ds.hama.YarnGraph
13/05/15 05:40:27 INFO bsp.YARNBSPJob: Connecting to ResourceManager at /
0.0.0.0:8040
13/05/15 05:40:28 INFO bsp.YARNBSPJob: Starting job...
13/05/15 05:40:30 WARN conf.Configuration: user.name is deprecated.
Instead, use mapreduce.job.user.name
13/05/15 05:40:30 INFO bsp.YARNBSPJobClient: Submitting job...
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162)
at org.apache.hama.bsp.YARNBSPJobClient.launchJob(YARNBSPJobClient.java:82)
at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:369)
at org.apache.hama.bsp.YARNBSPJob.submit(YARNBSPJob.java:80)
at org.apache.hama.bsp.YARNBSPJob.waitForCompletion(YARNBSPJob.java:95)
at com.hdp.ds.hama.YarnGraph$.main(YarnSerializePrinting.scala:50)
at com.hdp.ds.hama.YarnGraph.main(YarnSerializePrinting.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:398)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:436)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729)

at org.apache.hadoop.ipc.Client.call(Client.java:1235)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at $Proxy7.getNewApplication(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159)
... 11 more


I know that the problem is the protocol, and it is because I added the
configuration in the HamaConfiguration, but it is not the solution
    conf.set("yarn.ipc.rpc.class",
"org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC")


These are the version that I have installed:

Hadoop  -> 2.0.3.22-alpha
hama -> 0.6.1


Thanks for your help
  rgc

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message