hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Thomas <christophermauricetho...@gmail.com>
Subject Re: Problems building hadoop 2.2.0 from source
Date Sat, 08 Feb 2014 04:23:09 GMT
Thanks, I built 2.3 yesterday (checked out from from link suggested in
earlier post of this thread) without problems apart from VM running out of
memory which was fixed with

export MAVEN_OPTS=-Xmx2048m

At least, I got a message saying successful build.

Thanks for your help.




On 8 February 2014 10:53, Ted Yu <yuzhihong@gmail.com> wrote:

> In the output for a passing test, I saw:
>
> 2014-02-06 16:48:49,722 ERROR [Thread[Thread-71,5,main]]
> delegation.AbstractDelegationTokenSecretManager
> (AbstractDelegationTokenSecretManager.java:run(557)) -
> InterruptedExcpetion recieved for ExpiredTokenRemover thread
> java.lang.InterruptedException: sleep interrupted
>
> Meaning the above was not critical.
>
> branch-2.3 is receiving attention now.
> Discovering test failure there would be more helpful.
>
> Cheers
>
>
> On Thu, Feb 6, 2014 at 9:25 PM, Christopher Thomas <
> christophermauricethomas@gmail.com> wrote:
>
>> I guess the ERROR lines in
>>
>> hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-
>> jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.
>> TestMRJobsWithHistoryService-output.txt
>>
>> led me to believe that the problem was with TestMRJobsWithHistoryService.
>> If that's not the case then what do these messages indicate?
>>
>> As I say I am a complete novice and the learning curve is very steep.
>>
>>
>> On 7 February 2014 14:47, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> I checked out source code from
>>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.3 and
>>> it builds.
>>>
>>> From TestMRJobsWithHistoryService.txt, the test passed.
>>> What led to this test being singled out among the 454 tests ?
>>>
>>> Thanks
>>>
>>>
>>> On Thu, Feb 6, 2014 at 7:26 PM, Christopher Thomas <
>>> christophermauricethomas@gmail.com> wrote:
>>>
>>>> Yes well I tried 2.3, but I have found a number of problems building
>>>> it. I had to resort to manually applying patches that I found in the bug
>>>> tracking lists, which did not seem to have made it into all branches. So
>>>> for the moment I am sticking with 2.2.0 which is advertised as being
>>>> stable.
>>>>
>>>> I apologise for the confusion.
>>>>
>>>> Here is the contents
>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt,
>>>> though perhaps not that illuminating:
>>>>
>>>>
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
>>>>
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.669
>>>> sec
>>>>
>>>>
>>>>
>>>>
>>>> On 7 February 2014 14:12, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>
>>>>> The output was
>>>>> from hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-output.txt
>>>>>
>>>>>  Can you show us the contents
>>>>> of ./hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>> ?
>>>>>
>>>>> BTW hadoop 2.3 release candidate is coming up. You may consider paying
>>>>> more attention to hadoop 2.3
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 5:33 PM, Christopher Thomas <
>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>
>>>>>> I included the last part of
>>>>>>
>>>>>> hadoop-mapreduce-project/hadoop-mapreduce-client/
>>>>>> hadoop-mapreduce-client-jobclient/target/surefire-
>>>>>> reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.
>>>>>> txt
>>>>>>
>>>>>> in the second half of my initial posting, calling it "the output
>>>>>> from TestMRJobsWithHistoryService". Sloppy terminology I know, sorry
>>>>>> if I wasn't very clear.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Chris
>>>>>>
>>>>>>
>>>>>> On 7 February 2014 11:53, Ted Yu <yuzhihong@gmail.com> wrote:
>>>>>>
>>>>>>> There isn't System.exit call in TestMRJobsWithHistoryService.java
>>>>>>>
>>>>>>> What
>>>>>>> did hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire-reports/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService.txt
>>>>>>> say ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 6, 2014 at 4:41 PM, Christopher Thomas <
>>>>>>> christophermauricethomas@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am a complete beginner to Hadoop, trying to build 2.2.0
from
>>>>>>>> source on a Macbook Pro running OS X Mavericks.
>>>>>>>>
>>>>>>>> I am following the 'instructions' at
>>>>>>>> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
such
>>>>>>>> as they are.
>>>>>>>>
>>>>>>>> I get the following test failure:
>>>>>>>>
>>>>>>>> Forking command line: /bin/sh -c cd
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
>>>>>>>> &&
>>>>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java
>>>>>>>> -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter1837947962445626736.jar
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire8537321961333093423tmp
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1668661782666182940486tmp
>>>>>>>> Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
>>>>>>>>
>>>>>>>> Results :
>>>>>>>>
>>>>>>>> Tests run: 454, Failures: 0, Errors: 0, Skipped: 11
>>>>>>>>
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Reactor Summary:
>>>>>>>> [INFO]
>>>>>>>> [INFO] hadoop-mapreduce-client-jobclient .................
FAILURE
>>>>>>>> [1:18:07.440s]
>>>>>>>> [INFO] hadoop-mapreduce-client-hs-plugins ................
SKIPPED
>>>>>>>> [INFO] Apache Hadoop MapReduce Examples ..................
SKIPPED
>>>>>>>> [INFO] hadoop-mapreduce ..................................
SKIPPED
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] BUILD FAILURE
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [INFO] Total time: 1:18:07.828s
>>>>>>>> [INFO] Finished at: Thu Feb 06 23:01:45 EST 2014
>>>>>>>> [INFO] Final Memory: 32M/94M
>>>>>>>> [INFO]
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test
(default-test)
>>>>>>>> on project hadoop-mapreduce-client-jobclient: ExecutionException;
nested
>>>>>>>> exception is java.util.concurrent.ExecutionException:
>>>>>>>> java.lang.RuntimeException: The forked VM terminated without
saying
>>>>>>>> properly goodbye. VM crash or System.exit called ?
>>>>>>>>
>>>>>>>>
>>>>>>>> and the output from TestMRJobsWithHistoryService includes:
>>>>>>>>
>>>>>>>>
>>>>>>>> 2014-02-06 23:00:54,530 ERROR [Thread[Thread-252,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:00:54,531 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO  [IPC Server listener on 55131]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55131
>>>>>>>> 2014-02-06 23:00:54,540 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:00:54,544 INFO  [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,647 INFO  [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(136)) - Stopping JobHistory
>>>>>>>> 2014-02-06 23:00:54,647 INFO  [main] hs.JobHistory
>>>>>>>> (JobHistory.java:serviceStop(138)) - Stopping History Cleaner/Move
To Done
>>>>>>>> 2014-02-06 23:00:54,656 INFO  [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:00:54,759 INFO  [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(328))
-
>>>>>>>> Applications still running : [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:54,762 INFO  [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(336))
- Waiting
>>>>>>>> for Applications to be Finished
>>>>>>>> 2014-02-06 23:00:54,762 INFO  [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451))
- Application
>>>>>>>> application_1391688017341_0001 transitioned from RUNNING
to
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP
>>>>>>>> 2014-02-06 23:00:54,763 INFO  [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting
absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_0/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,763 INFO  [DeletionService #0]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting
absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_1/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO  [AsyncDispatcher event handler]
>>>>>>>> containermanager.AuxServices (AuxServices.java:handle(175))
- Got event
>>>>>>>> APPLICATION_STOP for appId application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,765 INFO  [DeletionService #1]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting
absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_2/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:54,766 INFO  [AsyncDispatcher event handler]
>>>>>>>> application.Application (ApplicationImpl.java:handle(451))
- Application
>>>>>>>> application_1391688017341_0001 transitioned from
>>>>>>>> APPLICATION_RESOURCES_CLEANINGUP to FINISHED
>>>>>>>> 2014-02-06 23:00:54,767 INFO  [AsyncDispatcher event handler]
>>>>>>>> loghandler.NonAggregatingLogHandler
>>>>>>>> (NonAggregatingLogHandler.java:handle(118)) - Scheduling
Log Deletion for
>>>>>>>> application: application_1391688017341_0001, with delay of
10800 seconds
>>>>>>>> 2014-02-06 23:00:54,768 INFO  [DeletionService #3]
>>>>>>>> nodemanager.DefaultContainerExecutor
>>>>>>>> (DefaultContainerExecutor.java:deleteAsUser(368)) - Deleting
absolute path
>>>>>>>> :
>>>>>>>> /Users/hadoop/hadoop-2.2.0-src/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService/org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService-localDir-nm-0_3/usercache/root/appcache/application_1391688017341_0001
>>>>>>>> 2014-02-06 23:00:58,766 INFO  [main]
>>>>>>>> containermanager.ContainerManagerImpl
>>>>>>>> (ContainerManagerImpl.java:cleanUpApplicationsOnNMShutDown(353))
- Done
>>>>>>>> waiting for Applications to be Finished. Still alive:
>>>>>>>> [application_1391688017341_0001]
>>>>>>>> 2014-02-06 23:00:58,768 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55125
>>>>>>>> 2014-02-06 23:00:58,771 INFO  [IPC Server listener on 55125]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55125
>>>>>>>> 2014-02-06 23:00:58,772 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,774 INFO  [main] monitor.ContainersMonitorImpl
>>>>>>>> (ContainersMonitorImpl.java:isEnabled(159)) - ResourceCalculatorPlugin
is
>>>>>>>> unavailable on this system.
>>>>>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
>>>>>>>> is disabled.
>>>>>>>> 2014-02-06 23:01:08,784 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO  [IPC Server listener on 55126]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55126
>>>>>>>> 2014-02-06 23:01:08,785 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,785 INFO  [Public Localizer]
>>>>>>>> localizer.ResourceLocalizationService
>>>>>>>> (ResourceLocalizationService.java:run(728)) - Public cache
exiting
>>>>>>>> 2014-02-06 23:01:08,787 INFO  [main] mortbay.log
>>>>>>>> (Slf4jLog.java:info(67)) - Stopped
>>>>>>>> SelectChannelConnector@Christophers-MacBook-Pro.local:0
>>>>>>>> 2014-02-06 23:01:08,889 ERROR [Thread[Thread-17,5,main]]
>>>>>>>> delegation.AbstractDelegationTokenSecretManager
>>>>>>>> (AbstractDelegationTokenSecretManager.java:run(547)) - InterruptedExcpetion
>>>>>>>> recieved for ExpiredTokenRemover thread java.lang.InterruptedException:
>>>>>>>> sleep interrupted
>>>>>>>> 2014-02-06 23:01:08,891 INFO  [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(200)) - Stopping JobHistoryServer
metrics
>>>>>>>> system...
>>>>>>>> 2014-02-06 23:01:08,895 INFO  [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:stop(206)) - JobHistoryServer metrics
system
>>>>>>>> stopped.
>>>>>>>> 2014-02-06 23:01:08,895 INFO  [main] impl.MetricsSystemImpl
>>>>>>>> (MetricsSystemImpl.java:shutdown(572)) - JobHistoryServer
metrics system
>>>>>>>> shutdown complete.
>>>>>>>> 2014-02-06 23:01:08,896 WARN  [ApplicationMaster Launcher]
>>>>>>>> amlauncher.ApplicationMasterLauncher
>>>>>>>> (ApplicationMasterLauncher.java:run(98)) -
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.ApplicationMasterLauncher$LauncherThread
>>>>>>>> interrupted. Returning.
>>>>>>>> 2014-02-06 23:01:08,897 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO  [IPC Server listener on 55122]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55122
>>>>>>>> 2014-02-06 23:01:08,898 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55121
>>>>>>>> 2014-02-06 23:01:08,898 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO  [IPC Server listener on 55121]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55121
>>>>>>>> 2014-02-06 23:01:08,908 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,902 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO  [IPC Server listener on 55120]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55120
>>>>>>>> 2014-02-06 23:01:08,912 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,912 INFO  [main] ipc.Server
>>>>>>>> (Server.java:stop(2437)) - Stopping server on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO  [IPC Server listener on 55119]
>>>>>>>> ipc.Server (Server.java:run(720)) - Stopping IPC Server listener
on 55119
>>>>>>>> 2014-02-06 23:01:08,917 INFO  [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>>>>> NMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,917 ERROR [ResourceManager Event Processor]
>>>>>>>> resourcemanager.ResourceManager (ResourceManager.java:run(435))
-
>>>>>>>> Returning, interrupted : java.lang.InterruptedException
>>>>>>>> 2014-02-06 23:01:08,917 INFO  [IPC Server Responder] ipc.Server
>>>>>>>> (Server.java:run(866)) - Stopping IPC Server Responder
>>>>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>>>>> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.ContainerAllocationExpirer
>>>>>>>> thread interrupted
>>>>>>>> 2014-02-06 23:01:08,918 INFO  [Ping Checker]
>>>>>>>> util.AbstractLivelinessMonitor (AbstractLivelinessMonitor.java:run(127))
-
>>>>>>>> AMLivelinessMonitor thread interrupted
>>>>>>>>
>>>>>>>> Has anyone seen this error before when building hadoop, and
does
>>>>>>>> anyone have any ideas?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message