it looks like the Pivotal cluster is running on YARN or trying to launch Giraph on YARN from the stack trace?


On Thu, Mar 13, 2014 at 4:34 AM, Carlo Sartiani <sartiani@gmail.com> wrote:
Hi all.

I am encountering a strange problem when running a Giraph application on a cluster with Hadoop 2.0.5alpha.gphd2.1.0 (Pivotal Hadoop distribution).

Our application is based on Giraph 1.0 and is executed in mapreduce compatibility mode  rather than in yarn mode. The application works smoothly on a standard Hadoop 2.0.5alpha distribution in our local cluster. However, when moved to an external cluster with Pivotal Hadoop 2.0.5alpha, it returns the  error described by the following log:

2014-03-12 17:33:21,489 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1388715708455_1597_000001
2014-03-12
17:33:21,722 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-03-12 17:33:21,723 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-03-12 17:33:21,989 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-03-12 17:33:21,989 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-03-12 17:33:22,005 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: jobSubmitDir=file:/data5/phd/yarn/nm-local-dir/usercache/csartiani/appcache/application_1388715708455_1597/container_1388715708455_1597_01_000001/jobSubmitDir jobTokenFile=file:/data5/phd/yarn/nm-local-dir/usercache/csartiani/appcache/application_1388715708455_1597/container_1388715708455_1597_01_000001/jobSubmitDir/appTokens
2014-03-12 17:33:22,005 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: AM Retries: 1 attempt num: 1 is last retry: true
2014-03-12 17:33:22,013 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2014-03-12 17:33:22,315 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2014-03-12 17:33:22,377 WARN [main] org.apache.giraph.bsp.BspOutputFormat: getOutputCommitter: Returning ImmutableOutputCommiter (does nothing).
2014-03-12 17:33:22,377 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.giraph.bsp.ImmutableOutputCommitter
2014-03-12 17:33:22,377 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Not starting RecoveryService: recoveryEnabled: true recoverySupportedByCommitter: false ApplicationAttemptID: 1
2014-03-12
17:33:22,394 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2014-03-12 17:33:22,395 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2014-03-12 17:33:22,396 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2014-03-12 17:33:22,396 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2014-03-12 17:33:22,397 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2014-03-12 17:33:22,401 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2014-03-12 17:33:22,402 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2014-03-12 17:33:22,403 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2014-03-12 17:33:22,403 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is inited.
2014-03-12 17:33:22,403 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:MRClientService is inited.
2014-03-12 17:33:22,403 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:CommitterEventHandler is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:TaskHeartbeatHandler is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapred.TaskAttemptListenerImpl is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter is inited.
2014-03-12 17:33:22,405 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster$StagingDirCleaningService is inited.
2014-03-12 17:33:22,411 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:JobHistoryEventHandler is inited.
2014-03-12 17:33:22,411 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster is inited.
2014-03-12 17:33:22,460 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2014-03-12 17:33:22,653 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2014-03-12 17:33:22,696 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2014-03-12 17:33:22,696 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2014-03-12 17:33:22,704 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1388715708455_1597 to jobTokenSecretManager
2014-03-12 17:33:22,785 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1388715708455_1597 because: not enabled; too much RAM;
2014-03-12 17:33:22,799 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1388715708455_1597 = 0. Number of splits = 1
2014-03-12
17:33:22,799 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1388715708455_1597 = 0
2014-03-12 17:33:22,799 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1388715708455_1597Job Transitioned from NEW to INITED
2014-03-12 17:33:22,799 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1388715708455_1597.
2014-03-12 17:33:22,799 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is started.
2014-03-12 17:33:22,801 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1388715708455_1597
2014-03-12 17:33:22,833 INFO [Socket Reader #1 for port 43578] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 43578
2014-03-12 17:33:22,846 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2014-03-12 17:33:22,847 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-03-12 17:33:22,847 INFO [IPC Server listener on 43578] org.apache.hadoop.ipc.Server: IPC Server listener on 43578: starting
2014-03-12 17:33:22,847 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at node0801/10.1.41.1:43578
2014-03-12 17:33:22,867 ERROR [main] org.apache.hadoop.yarn.service.CompositeService: Error starting services org.apache.hadoop.mapreduce.v2.app.MRAppMaster
java.lang.NoSuchMethodError: org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
    at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:196)
    at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:149)
    at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1022)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1276)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1272)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1228)
2014-03-12 17:33:22,868 INFO [main] org.apache.hadoop.ipc.Server: Stopping server on 43578
2014-03-12 17:33:22,869 INFO [IPC Server listener on 43578] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 43578
2014-03-12 17:33:22,869 INFO [main] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is stopped.
2014-03-12 17:33:22,869 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-03-12 17:33:22,869 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.YarnException: Failed to Start org.apache.hadoop.mapreduce.v2.app.MRAppMaster
    at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.start(MRAppMaster.java:1022)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1276)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1272)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1228)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
    at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:196)
    at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.start(MRClientService.java:149)
    at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
    ... 7 more
2014-03-12 17:33:22,870 INFO [Thread-1] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster received a signal. Signaling RMCommunicator and JobHistoryEventHandler.
2014-03-12 17:33:22,871 WARN [Thread-1] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook 'MRAppMasterShutdownHook' failed, java.lang.NullPointerException
java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:811)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1249)
    at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

The error seems quite strange, as the system is reporting the lack of  a given constructor for org.apache.hadoop.http.HttpServer.

Other applications that we are running on this cluster and that do not use Giraph work perfectly. We have no control on this cluster, so we cannot modify any cluster setting.

Do you any idea on how to solve this issue?

Thanks in advance.

Best Regards

Carlo Sartiani 
--
"Humanity is overrated", Gregory House, MD