giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlo Sartiani <sarti...@gmail.com>
Subject Giraph 1.1.0 and Jetty 7
Date Fri, 04 Jul 2014 16:11:05 GMT
Dear All,

we are trying to run Giraph 1.1.0 RC against a distribution of Hadoop 
2.2.0 that uses Jetty 7 in place of Jetty 6. When we launch our 
application on our cluster we get the following error:

2014-07-02 08:14:23,307 INFO [main] 
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class 
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for 
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2014-07-02 08:14:23,551 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from 
hadoop-metrics2.properties
2014-07-02 08:14:23,619 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot 
period at 10 second(s).
2014-07-02 08:14:23,619 INFO [main] 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics 
system started
2014-07-02 08:14:23,630 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token 
for job_1403542528912_0632 to jobTokenSecretManager
2014-07-02 08:14:23,726 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing 
job_1403542528912_0632 because: not enabled; too much RAM;
2014-07-02 08:14:23,742 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job 
job_1403542528912_0632 = 0. Number of splits = 1
2014-07-02 08:14:23,742 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces 
for job job_1403542528912_0632 = 0
2014-07-02 08:14:23,742 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: 
job_1403542528912_0632Job Transitioned from NEW to INITED
2014-07-02 08:14:23,744 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching 
normal, non-uberized, multi-container job job_1403542528912_0632.
2014-07-02 08:14:23,782 INFO [Socket Reader #1 for port 7041] 
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 7041
2014-07-02 08:14:23,804 INFO [main] 
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding 
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2014-07-02 08:14:23,805 INFO [IPC Server Responder] 
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-07-02 08:14:23,805 INFO [IPC Server listener on 7041] 
org.apache.hadoop.ipc.Server: IPC Server listener on 7041: starting
2014-07-02 08:14:23,805 INFO [main] 
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated 
MRClientService at node0654/10.1.33.14:7041
2014-07-02 08:14:23,858 FATAL [main] 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError: 
org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
     at 
org.apache.hadoop.yarn.webapp.WebApps$Builder$2.<init>(WebApps.java:222)
     at 
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:219)
     at 
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:136)
     at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
     at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1058)
     at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
     at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:415)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
     at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
     at 
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)

The error is caused by the fact that in our distribution of Hadoop the 
HTTPServer constructor is expecting a parameter of type 
org.eclipse.jetty.Connector, but is receiving a parameter of type 
org.mortbay.jetty.Connector. Indeed, one big difference of Jetty 7 wrt 
Jetty 6 is the different naming of packages.

Similar errors arise when the application is launched in pure yarn mode 
(containers just fail).

We examined the code of Giraph 1.1.0, but we actually did not find any 
place where a Collector object is created and/or Jetty is really used.

Do you have  any idea on how to solve this issue? Please, observe that 
we are forced to use this Hadoop distribution and cannot switch to a 
plain Hadoop 2.2.0 distribution with Jetty 6.

Thanks in advance.

Carlo
-- 
"Humanity is overrated", Gregory House, MD

Mime
View raw message