hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Problems building hadoop 2.2.0 from source
Date Fri, 07 Feb 2014 03:47:12 GMT
I checked out source code from
http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and it
builds.

>From TestMRJobsWithHistoryService.txt, the test passed.
What led to this test being singled out among the 454 tests ?

Thanks


On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
christophermauricethomas@gmail.com> wrote:

> Yes well I tried 2.3, but I have found a number of problems building it. I
> had to resort to manually applying patches that I found in the bug tracking
> lists, which did not seem to have made it into all branches. So for the
> moment I am sticking with 2.2.0 which is advertised as being stable.
>
> I apologise for the confusion.
>
> Here is the contents
> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
> though perhaps not that illuminating:
>
>
>
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669 sec
>
>
>
>
> On 7 February 2014 14:12, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> The output was
>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>
>>  Can you show us the contents
>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>> ?
>>
>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>> more attention to hadoop 2.3
>>
>> Cheers
>>
>>
>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>> christophermauricethomas@gmail.com> wrote:
>>
>>> I included the last part of
>>>
>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>
>>> in the second half of my initial posting, calling it "the output from TestMRJobsWithHistoryService".
>>> Sloppy terminology I know, sorry if I wasn't very clear.
>>>
>>> Regards
>>>
>>> Chris
>>>
>>>
>>> On 7 February 2014 11:53, Ted Yu <yuzhihong@gmail.com> wrote:
>>>
>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>
>>>> What
>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>> say ?
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>> christophermauricethomas@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0 from source
>>>>> on a Macbook Pro running OS X Mavericks.
>>>>>
>>>>> I am following the 'instructions' at
>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
such
>>>>> as they are.
>>>>>
>>>>> I get the following test failure:
>>>>>
>>>>> Forking command line: /bin/sh -c cd
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>> &&
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>
>>>>> Results :
>>>>>
>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Reactor Summary:
>>>>> [INFO]
>>>>> [INFO] hadoop-mapreduce-client-jobclient ................. FAILURE
>>>>> [1:18:07.440s]
>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
>>>>> [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
>>>>> [INFO] hadoop-mapreduce .................................. SKIPPED
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD FAILURE
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1:18:07.828s
>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>> [INFO] Final Memory: 32M/94M
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [ERROR] Failed to execute goal
>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test)
>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException; nested
>>>>> exception is java.util.concurrent.ExecutionException:
>>>>> java.lang.RuntimeException: The forked VM terminated without saying
>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>
>>>>>
>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>
>>>>>
>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:00:54,531 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>> 2014-02-06 23:00:54,540 INFO  [IPC Server listener on 55131]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55131
>>>>> 2014-02-06 23:00:54,540 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:00:54,544 INFO  [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,647 INFO  [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>> 2014-02-06 23:00:54,647 INFO  [main] hs.JobHistory
>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move To
Done
>>>>> 2014-02-06 23:00:54,656 INFO  [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:00:54,759 INFO  [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328)) -
>>>>> Applications still running : [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:54,762 INFO  [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336)) - Waiting
>>>>> for Applications to be Finished
>>>>> 2014-02-06 23:00:54,762 INFO  [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from RUNNING to
>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>> 2014-02-06 23:00:54,763 INFO  [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute
path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,763 INFO  [DeletionService #0]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute
path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO  [AsyncDispatcher event handler]
>>>>> containermanager.AuxServices (AuxServices.java:handle(175)) - Got event
>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,765 INFO  [DeletionService #1]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute
path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:54,766 INFO  [AsyncDispatcher event handler]
>>>>> application.Application (ApplicationImpl.java:handle(451)) - Application
>>>>> application_1391688017341_0001 transitioned from
>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>> 2014-02-06 23:00:54,767 INFO  [AsyncDispatcher event handler]
>>>>> loghandler.NonAggregatingLogHandler
>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling Log Deletion
for
>>>>> application: application_1391688017341_0001, with delay of 10800 seconds
>>>>> 2014-02-06 23:00:54,768 INFO  [DeletionService #3]
>>>>> nodemanager.DefaultContainerExecutor
>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting absolute
path
>>>>> :
>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>> 2014-02-06 23:00:58,766 INFO  [main]
>>>>> containermanager.ContainerManagerImpl
>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353)) - Done
>>>>> waiting for Applications to be Finished. Still alive:
>>>>> [application_1391688017341_0001]
>>>>> 2014-02-06 23:00:58,768 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>> 2014-02-06 23:00:58,771 INFO  [IPC Server listener on 55125]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55125
>>>>> 2014-02-06 23:00:58,772 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,774 INFO  [main] monitor.ContainersMonitorImpl
>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin
is
>>>>> unavailable on this system.
>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>> is disabled.
>>>>> 2014-02-06 23:01:08,784 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>> 2014-02-06 23:01:08,785 INFO  [IPC Server listener on 55126]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55126
>>>>> 2014-02-06 23:01:08,785 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,785 INFO  [Public Localizer]
>>>>> localizer.ResourceLocalizationService
>>>>> (ResourceLocalizationService.java:run(728)) - Public cache exiting
>>>>> 2014-02-06 23:01:08,787 INFO  [main] mortbay.log
>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>> sleep interrupted
>>>>> 2014-02-06 23:01:08,891 INFO  [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer metrics
>>>>> system...
>>>>> 2014-02-06 23:01:08,895 INFO  [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics system
>>>>> stopped.
>>>>> 2014-02-06 23:01:08,895 INFO  [main] impl.MetricsSystemImpl
>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer metrics system
>>>>> shutdown complete.
>>>>> 2014-02-06 23:01:08,896 WARN  [ApplicationMaster Launcher]
>>>>> amlauncher.ApplicationMasterLauncher
>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>> interrupted. Returning.
>>>>> 2014-02-06 23:01:08,897 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>> 2014-02-06 23:01:08,898 INFO  [IPC Server listener on 55122]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55122
>>>>> 2014-02-06 23:01:08,898 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>> 2014-02-06 23:01:08,898 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO  [IPC Server listener on 55121]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55121
>>>>> 2014-02-06 23:01:08,908 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,902 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>> 2014-02-06 23:01:08,912 INFO  [IPC Server listener on 55120]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55120
>>>>> 2014-02-06 23:01:08,912 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,912 INFO  [main] ipc.Server
>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>> 2014-02-06 23:01:08,917 INFO  [IPC Server listener on 55119]
>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener on 55119
>>>>> 2014-02-06 23:01:08,917 INFO  [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>> NMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435)) -
>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>> 2014-02-06 23:01:08,917 INFO  [IPC Server Responder] ipc.Server
>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>> AMLivelinessMonitor thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>> thread interrupted
>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>> AMLivelinessMonitor thread interrupted
>>>>>
>>>>> Has anyone seen this error before when building hadoop, and does
>>>>> anyone have any ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Chris
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message