giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlo Sartiani <sarti...@gmail.com>
Subject Re: Giraph 1.1.0 and Jetty 7
Date Mon, 07 Jul 2014 14:00:34 GMT
Thanks for the suggestion. Unfortunately, it does not work. Probably, 
Giraph is using a library that generates an org.mortbay.jetty.Connector 
in place of a org.eclipse.jetty.Connector one.

Carlo
> Roman Shaposhnik <mailto:roman@shaposhnik.org>
> 7 Jul 2014 00:12
>
> Here's what I'd suggest: take 1.1.0-RC0 source code and manually change
> the version of Jetty in the top level pom.xml. Rebuild everything with
> -Phadoop_2
> and also specify the exact version of your Hadoop with 
> -Dhadoop.version=X.Y.Z
> for good measure.
>
> Thanks,
> Roman.
>
> Carlo Sartiani <mailto:sartiani@gmail.com>
> 4 Jul 2014 18:11
> Dear All,
>
> we are trying to run Giraph 1.1.0 RC against a distribution of Hadoop 
> 2.2.0 that uses Jetty 7 in place of Jetty 6. When we launch our 
> application on our cluster we get the following error:
>
> 2014-07-02 08:14:23,307 INFO [main] 
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class 
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for 
> class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> 2014-07-02 08:14:23,551 INFO [main] 
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from 
> hadoop-metrics2.properties
> 2014-07-02 08:14:23,619 INFO [main] 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot 
> period at 10 second(s).
> 2014-07-02 08:14:23,619 INFO [main] 
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics 
> system started
> 2014-07-02 08:14:23,630 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token 
> for job_1403542528912_0632 to jobTokenSecretManager
> 2014-07-02 08:14:23,726 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing 
> job_1403542528912_0632 because: not enabled; too much RAM;
> 2014-07-02 08:14:23,742 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for 
> job job_1403542528912_0632 = 0. Number of splits = 1
> 2014-07-02 08:14:23,742 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces 
> for job job_1403542528912_0632 = 0
> 2014-07-02 08:14:23,742 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: 
> job_1403542528912_0632Job Transitioned from NEW to INITED
> 2014-07-02 08:14:23,744 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching 
> normal, non-uberized, multi-container job job_1403542528912_0632.
> 2014-07-02 08:14:23,782 INFO [Socket Reader #1 for port 7041] 
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 7041
> 2014-07-02 08:14:23,804 INFO [main] 
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: 
> Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB 
> to the server
> 2014-07-02 08:14:23,805 INFO [IPC Server Responder] 
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2014-07-02 08:14:23,805 INFO [IPC Server listener on 7041] 
> org.apache.hadoop.ipc.Server: IPC Server listener on 7041: starting
> 2014-07-02 08:14:23,805 INFO [main] 
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService: 
> Instantiated MRClientService at node0654/10.1.33.14:7041
> 2014-07-02 08:14:23,858 FATAL [main] 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
>     at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder$2.<init>(WebApps.java:222)
>     at 
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:219)
>     at 
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:136)
>     at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>     at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1058)
>     at 
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
>     at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>     at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
>     at 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
>
> The error is caused by the fact that in our distribution of Hadoop the 
> HTTPServer constructor is expecting a parameter of type 
> org.eclipse.jetty.Connector, but is receiving a parameter of type 
> org.mortbay.jetty.Connector. Indeed, one big difference of Jetty 7 wrt 
> Jetty 6 is the different naming of packages.
>
> Similar errors arise when the application is launched in pure yarn 
> mode (containers just fail).
>
> We examined the code of Giraph 1.1.0, but we actually did not find any 
> place where a Collector object is created and/or Jetty is really used.
>
> Do you have  any idea on how to solve this issue? Please, observe that 
> we are forced to use this Hadoop distribution and cannot switch to a 
> plain Hadoop 2.2.0 distribution with Jetty 6.
>
> Thanks in advance.
>
> Carlo
> ------------------------------------------------------------------------


-- 
"Humanity is overrated", Gregory House, MD

Mime
View raw message