hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: About Map 100% reduce %0 issue
Date Tue, 25 Mar 2014 05:50:54 GMT
Hi,

The log you've posted is that of the AM, not of the specific reduce
attempt, i.e. not of attempt_1395753558524_0002_r_000000_0 for
example. Its not the AM failing, but individual reduce attempts.

On Tue, Mar 25, 2014 at 11:08 AM, Vincent,Wei <weikun0905@gmail.com> wrote:
> Thanks both of you
>
> Shetty , I have checked this configure , I use the default of
> yarn-site.xml, and I can see the configure is the value you have said.
>
>
> Harsh & all
>
> I have post the whole log for the task ,may be the task name have changed,
> but the result is the same.
>
>
> 2014-03-25 21:33:28,492 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
> application appattempt_1395753558524_0002_000001
> 2014-03-25 21:33:28,633 WARN [main] org.apache.hadoop.conf.Configuration:
> job.xml:an attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval;  Ignoring.
> 2014-03-25 21:33:28,634 WARN [main] org.apache.hadoop.conf.Configuration:
> job.xml:an attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts;  Ignoring.
> 2014-03-25 21:33:28,869 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
> 2014-03-25 21:33:28,869 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
> Service: , Ident:
> (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@332437f7)
> 2014-03-25 21:33:28,882 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts:
> 2 for application: 2. Attempt num: 1 is last retry: false
> 2014-03-25 21:33:28,887 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred
> newApiCommitter.
> 2014-03-25 21:33:28,940 WARN [main] org.apache.hadoop.conf.Configuration:
> job.xml:an attempt to override final parameter:
> mapreduce.job.end-notification.max.retry.interval;  Ignoring.
> 2014-03-25 21:33:28,940 WARN [main] org.apache.hadoop.conf.Configuration:
> job.xml:an attempt to override final parameter:
> mapreduce.job.end-notification.max.attempts;  Ignoring.
> 2014-03-25 21:33:29,220 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in
> config null
> 2014-03-25 21:33:29,262 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
> 2014-03-25 21:33:29,275 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.jobhistory.EventType for class
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
> 2014-03-25 21:33:29,275 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
> 2014-03-25 21:33:29,276 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
> 2014-03-25 21:33:29,277 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
> 2014-03-25 21:33:29,277 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
> 2014-03-25 21:33:29,280 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
> 2014-03-25 21:33:29,280 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for
> class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
> 2014-03-25 21:33:29,281 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
> class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
> 2014-03-25 21:33:29,289 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Creating
> intermediate history logDir:
> [hdfs://master:9000/tmp/hadoop-yarn/staging/history/done_intermediate] +
> based on conf. Should ideally be created by the JobHistoryServer:
> yarn.app.mapreduce.am.create-intermediate-jh-base-dir
> 2014-03-25 21:33:29,310 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after
> creating 493, Expected: 1023
> 2014-03-25 21:33:29,310 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Explicitly
> setting permissions to : 1023, rwxrwxrwt
> 2014-03-25 21:33:29,342 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after
> creating 488, Expected: 504
> 2014-03-25 21:33:29,342 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Explicitly
> setting permissions to : 504, rwxrwx---
> 2014-03-25 21:33:29,384 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> 2014-03-25 21:33:29,564 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2014-03-25 21:33:29,594 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2014-03-25 21:33:29,594 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
> system started
> 2014-03-25 21:33:29,600 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
> job_1395753558524_0002 to jobTokenSecretManager
> 2014-03-25 21:33:29,684 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
> job_1395753558524_0002 because: not enabled;
> 2014-03-25 21:33:29,693 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
> job_1395753558524_0002 = 1578. Number of splits = 1
> 2014-03-25 21:33:29,693 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for
> job job_1395753558524_0002 = 1
> 2014-03-25 21:33:29,693 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1395753558524_0002Job Transitioned from NEW to INITED
> 2014-03-25 21:33:29,694 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
> normal, non-uberized, multi-container job job_1395753558524_0002.
> 2014-03-25 21:33:29,715 INFO [Socket Reader #1 for port 33413]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 33413
> 2014-03-25 21:33:29,724 INFO [main]
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
> protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
> 2014-03-25 21:33:29,724 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2014-03-25 21:33:29,724 INFO [IPC Server listener on 33413]
> org.apache.hadoop.ipc.Server: IPC Server listener on 33413: starting
> 2014-03-25 21:33:29,725 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
> MRClientService at slave1/159.99.249.203:33413
> 2014-03-25 21:33:29,746 INFO [main] org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2014-03-25 21:33:29,771 INFO [main] org.apache.hadoop.http.HttpServer:
> Added global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> 2014-03-25 21:33:29,774 INFO [main] org.apache.hadoop.http.HttpServer:
> Added filter AM_PROXY_FILTER
> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> context mapreduce
> 2014-03-25 21:33:29,774 INFO [main] org.apache.hadoop.http.HttpServer:
> Added filter AM_PROXY_FILTER
> (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to
> context static
> 2014-03-25 21:33:29,776 INFO [main] org.apache.hadoop.http.HttpServer:
> adding path spec: /mapreduce/*
> 2014-03-25 21:33:29,776 INFO [main] org.apache.hadoop.http.HttpServer:
> adding path spec: /ws/*
> 2014-03-25 21:33:29,776 INFO [main] org.apache.hadoop.http.HttpServer:
> Jetty bound to port 33346
> 2014-03-25 21:33:29,776 INFO [main] org.mortbay.log: jetty-6.1.26
> 2014-03-25 21:33:29,792 INFO [main] org.mortbay.log: Extract
> jar:file:/home/haduser/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar!/webapps/mapreduce
> to /tmp/Jetty_0_0_0_0_33346_mapreduce____d9bkg4/webapp
> 2014-03-25 21:33:29,916 INFO [main] org.mortbay.log: Started
> SelectChannelConnector@0.0.0.0:33346
> 2014-03-25 21:33:29,917 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Web app /mapreduce started at 33346
> 2014-03-25 21:33:30,113 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
> Registered webapp guice modules
> 2014-03-25 21:33:30,115 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE
> job_1395753558524_0002
> 2014-03-25 21:33:30,116 INFO [Socket Reader #1 for port 53159]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 53159
> 2014-03-25 21:33:30,119 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2014-03-25 21:33:30,119 INFO [IPC Server listener on 53159]
> org.apache.hadoop.ipc.Server: IPC Server listener on 53159: starting
> 2014-03-25 21:33:30,129 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> nodeBlacklistingEnabled:true
> 2014-03-25 21:33:30,129 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> maxTaskFailuresPerNode is 3
> 2014-03-25 21:33:30,129 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> blacklistDisablePercent is 33
> 2014-03-25 21:33:30,143 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
> Connecting to ResourceManager at master/159.99.249.63:8030
> 2014-03-25 21:33:30,174 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> maxContainerCapability: 8192
> 2014-03-25 21:33:30,175 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper
> limit on the thread pool size is 500
> 2014-03-25 21:33:30,176 INFO [main]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> yarn.client.max-nodemanagers-proxies : 500
> 2014-03-25 21:33:30,180 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1395753558524_0002Job Transitioned from INITED to SETUP
> 2014-03-25 21:33:30,182 INFO [CommitterEvent Processor #0]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: JOB_SETUP
> 2014-03-25 21:33:30,190 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1395753558524_0002Job Transitioned from SETUP to RUNNING
> 2014-03-25 21:33:30,197 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:30,198 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave2 to /default-rack
> 2014-03-25 21:33:30,198 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave3 to /default-rack
> 2014-03-25 21:33:30,199 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_m_000000 Task Transitioned from NEW to SCHEDULED
> 2014-03-25 21:33:30,199 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_r_000000 Task Transitioned from NEW to SCHEDULED
> 2014-03-25 21:33:30,200 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:30,200 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:30,200 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> mapResourceReqt:1024
> 2014-03-25 21:33:30,204 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> reduceResourceReqt:1024
> 2014-03-25 21:33:30,233 INFO [eventHandlingThread]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer
> setup for JobId: job_1395753558524_0002, File:
> hdfs://master:9000/tmp/hadoop-yarn/staging/haduser/.staging/job_1395753558524_0002/job_1395753558524_0002_1.jhist
> 2014-03-25 21:33:30,249 INFO [eventHandlingThread]
> org.apache.hadoop.conf.Configuration.deprecation: user.name is deprecated.
> Instead, use mapreduce.job.user.name
> 2014-03-25 21:33:31,175 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0
> HostLocal:0 RackLocal:0
> 2014-03-25 21:33:31,190 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=5 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:22528, vCores:0> knownNMs=3
> 2014-03-25 21:33:31,190 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=22528
> 2014-03-25 21:33:31,190 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold not met. completedMapsForReduceSlowstart 1
> 2014-03-25 21:33:32,212 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:32,213 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000002 to
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:32,214 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=21504
> 2014-03-25 21:33:32,214 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold not met. completedMapsForReduceSlowstart 1
> 2014-03-25 21:33:32,214 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:32,226 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:32,237 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar
> file on the remote FS is
> hdfs://master:9000/tmp/hadoop-yarn/staging/haduser/.staging/job_1395753558524_0002/job.jar
> 2014-03-25 21:33:32,243 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf
> file on the remote FS is
> /tmp/hadoop-yarn/staging/haduser/.staging/job_1395753558524_0002/job.xml
> 2014-03-25 21:33:32,244 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0
> tokens and #1 secret keys for NM use for launching container
> 2014-03-25 21:33:32,244 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of
> containertokens_dob is 1
> 2014-03-25 21:33:32,244 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting
> shuffle token in serviceData
> 2014-03-25 21:33:32,255 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:32,257 INFO [ContainerLauncher #0]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000002 taskAttempt
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:32,258 INFO [ContainerLauncher #0]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:32,258 INFO [ContainerLauncher #0]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : slave1:8041
> 2014-03-25 21:33:32,285 INFO [ContainerLauncher #0]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_m_000000_0
> : 13562
> 2014-03-25 21:33:32,286 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_m_000000_0] using containerId:
> [container_1395753558524_0002_01_000002 on NM: [slave1:8041]
> 2014-03-25 21:33:32,288 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:32,288 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_m_000000
> 2014-03-25 21:33:32,288 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_m_000000 Task Transitioned from SCHEDULED to RUNNING
> 2014-03-25 21:33:33,180 INFO [Socket Reader #1 for port 53159]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1395753558524_0002 (auth:SIMPLE)
> 2014-03-25 21:33:33,191 INFO [IPC Server handler 0 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1395753558524_0002_m_000002 asked for a task
> 2014-03-25 21:33:33,191 INFO [IPC Server handler 0 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1395753558524_0002_m_000002 given task:
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,219 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=5 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:33,866 INFO [IPC Server handler 1 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,866 INFO [IPC Server handler 1 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1395753558524_0002_m_000000_0 is : 0.0
> 2014-03-25 21:33:33,925 INFO [IPC Server handler 2 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,925 INFO [IPC Server handler 2 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1395753558524_0002_m_000000_0 is : 1.0
> 2014-03-25 21:33:33,928 INFO [IPC Server handler 3 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,929 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from RUNNING
> to SUCCESS_CONTAINER_CLEANUP
> 2014-03-25 21:33:33,929 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1395753558524_0002_01_000002 taskAttempt
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,929 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,935 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from
> SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
> 2014-03-25 21:33:33,939 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with
> attempt attempt_1395753558524_0002_m_000000_0
> 2014-03-25 21:33:33,940 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_m_000000 Task Transitioned from RUNNING to SUCCEEDED
> 2014-03-25 21:33:33,941 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
> 2014-03-25 21:33:34,219 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:34,225 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=21504
> 2014-03-25 21:33:34,225 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold reached. Scheduling reduces.
> 2014-03-25 21:33:34,225 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps
> assigned. Ramping up all remaining reduces:1
> 2014-03-25 21:33:34,226 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:1
> AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:35,230 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=1 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:35,230 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1395753558524_0002_01_000002
> 2014-03-25 21:33:35,230 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0
> AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:35,230 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_m_000000_0: Container killed by the
> ApplicationMaster.
> Container killed on request. Exit code is 143
>
> 2014-03-25 21:33:36,238 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:36,238 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to
> reduce
> 2014-03-25 21:33:36,239 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000003 to
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:36,239 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:36,253 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:36,254 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:36,255 INFO [ContainerLauncher #2]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000003 taskAttempt
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:36,255 INFO [ContainerLauncher #2]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:36,257 INFO [ContainerLauncher #2]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_r_000000_0
> : 13562
> 2014-03-25 21:33:36,257 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_r_000000_0] using containerId:
> [container_1395753558524_0002_01_000003 on NM: [slave1:8041]
> 2014-03-25 21:33:36,257 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:36,258 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_r_000000
> 2014-03-25 21:33:36,258 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_r_000000 Task Transitioned from SCHEDULED to RUNNING
> 2014-03-25 21:33:37,208 INFO [Socket Reader #1 for port 53159]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1395753558524_0002 (auth:SIMPLE)
> 2014-03-25 21:33:37,216 INFO [IPC Server handler 4 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1395753558524_0002_r_000003 asked for a task
> 2014-03-25 21:33:37,217 INFO [IPC Server handler 4 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1395753558524_0002_r_000003 given task:
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:37,243 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:37,746 INFO [IPC Server handler 5 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents
> request from attempt_1395753558524_0002_r_000000_0. startIndex 0 maxEvents
> 10000
> 2014-03-25 21:33:37,871 INFO [IPC Server handler 6 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:37,871 INFO [IPC Server handler 6 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1395753558524_0002_r_000000_0 is : 0.0
> 2014-03-25 21:33:37,886 FATAL [IPC Server handler 7 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1395753558524_0002_r_000000_0 - exited :
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:37,886 INFO [IPC Server handler 7 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1395753558524_0002_r_000000_0: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:37,886 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_0: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:37,887 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from RUNNING
> to FAIL_CONTAINER_CLEANUP
> 2014-03-25 21:33:37,887 INFO [ContainerLauncher #3]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1395753558524_0002_01_000003 taskAttempt
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:37,888 INFO [ContainerLauncher #3]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:37,891 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from
> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
> 2014-03-25 21:33:37,892 INFO [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: TASK_ABORT
> 2014-03-25 21:33:37,902 WARN [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
> delete
> hdfs://master:9000/output/12/_temporary/1/_temporary/attempt_1395753558524_0002_r_000000_0
> 2014-03-25 21:33:37,904 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_0 TaskAttempt Transitioned from
> FAIL_TASK_CLEANUP to FAILED
> 2014-03-25 21:33:37,908 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on
> node slave1
> 2014-03-25 21:33:37,908 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:38,244 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:38,247 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=21504
> 2014-03-25 21:33:38,247 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps
> assigned. Ramping up all remaining reduces:1
> 2014-03-25 21:33:38,248 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:39,253 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=1 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:39,254 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1395753558524_0002_01_000003
> 2014-03-25 21:33:39,254 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0
> AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:39,255 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_0: Container killed by the
> ApplicationMaster.
> Container killed on request. Exit code is 143
>
> 2014-03-25 21:33:40,262 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:40,262 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to
> reduce
> 2014-03-25 21:33:40,263 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000004 to
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:40,263 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:3 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:40,264 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:40,265 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:40,267 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000004 taskAttempt
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:40,267 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:40,275 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_r_000000_1
> : 13562
> 2014-03-25 21:33:40,276 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_r_000000_1] using containerId:
> [container_1395753558524_0002_01_000004 on NM: [slave1:8041]
> 2014-03-25 21:33:40,277 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:40,277 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_r_000000
> 2014-03-25 21:33:41,255 INFO [Socket Reader #1 for port 53159]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1395753558524_0002 (auth:SIMPLE)
> 2014-03-25 21:33:41,262 INFO [IPC Server handler 8 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1395753558524_0002_r_000004 asked for a task
> 2014-03-25 21:33:41,263 INFO [IPC Server handler 8 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1395753558524_0002_r_000004 given task:
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:41,267 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:41,789 INFO [IPC Server handler 9 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents
> request from attempt_1395753558524_0002_r_000000_1. startIndex 0 maxEvents
> 10000
> 2014-03-25 21:33:41,850 INFO [IPC Server handler 10 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:41,850 INFO [IPC Server handler 10 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1395753558524_0002_r_000000_1 is : 0.0
> 2014-03-25 21:33:41,865 FATAL [IPC Server handler 11 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1395753558524_0002_r_000000_1 - exited :
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:41,865 INFO [IPC Server handler 11 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1395753558524_0002_r_000000_1: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:41,866 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_1: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:41,867 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from RUNNING
> to FAIL_CONTAINER_CLEANUP
> 2014-03-25 21:33:41,867 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1395753558524_0002_01_000004 taskAttempt
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:41,869 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:41,870 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from
> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
> 2014-03-25 21:33:41,871 INFO [CommitterEvent Processor #2]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: TASK_ABORT
> 2014-03-25 21:33:41,873 WARN [CommitterEvent Processor #2]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
> delete
> hdfs://master:9000/output/12/_temporary/1/_temporary/attempt_1395753558524_0002_r_000000_1
> 2014-03-25 21:33:41,874 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_1 TaskAttempt Transitioned from
> FAIL_TASK_CLEANUP to FAILED
> 2014-03-25 21:33:41,874 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on
> node slave1
> 2014-03-25 21:33:41,874 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:42,267 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:3 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:42,272 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=21504
> 2014-03-25 21:33:42,272 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps
> assigned. Ramping up all remaining reduces:1
> 2014-03-25 21:33:42,273 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:3 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:43,278 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=1 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:43,279 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1395753558524_0002_01_000004
> 2014-03-25 21:33:43,279 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0
> AssignedReds:0 CompletedMaps:1 CompletedReds:0 ContAlloc:3 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:43,279 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_1: Container killed by the
> ApplicationMaster.
> Container killed on request. Exit code is 143
>
> 2014-03-25 21:33:44,285 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:44,285 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to
> reduce
> 2014-03-25 21:33:44,285 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000005 to
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:44,285 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:1 CompletedReds:0 ContAlloc:4 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:44,286 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:44,286 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:44,287 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000005 taskAttempt
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:44,287 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:44,289 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_r_000000_2
> : 13562
> 2014-03-25 21:33:44,289 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_r_000000_2] using containerId:
> [container_1395753558524_0002_01_000005 on NM: [slave1:8041]
> 2014-03-25 21:33:44,289 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:44,289 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_r_000000
> 2014-03-25 21:33:45,289 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:45,397 INFO [Socket Reader #1 for port 53159]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1395753558524_0002 (auth:SIMPLE)
> 2014-03-25 21:33:45,405 INFO [IPC Server handler 12 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1395753558524_0002_r_000005 asked for a task
> 2014-03-25 21:33:45,405 INFO [IPC Server handler 12 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1395753558524_0002_r_000005 given task:
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:45,932 INFO [IPC Server handler 13 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents
> request from attempt_1395753558524_0002_r_000000_2. startIndex 0 maxEvents
> 10000
> 2014-03-25 21:33:45,996 INFO [IPC Server handler 14 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:45,996 INFO [IPC Server handler 14 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1395753558524_0002_r_000000_2 is : 0.0
> 2014-03-25 21:33:45,998 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many
> fetch-failures for output of task attempt:
> attempt_1395753558524_0002_m_000000_0 ... raising fetch failure to map
> 2014-03-25 21:33:45,998 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_0 TaskAttempt Transitioned from
> SUCCEEDED to FAILED
> 2014-03-25 21:33:45,999 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:45,999 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave2 to /default-rack
> 2014-03-25 21:33:45,999 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave3 to /default-rack
> 2014-03-25 21:33:45,999 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_m_000000 Task Transitioned from SUCCEEDED to
> SCHEDULED
> 2014-03-25 21:33:45,999 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on
> node slave1
> 2014-03-25 21:33:45,999 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted
> host slave1
> 2014-03-25 21:33:45,999 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_1 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:45,999 INFO [Thread-48]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added
> attempt_1395753558524_0002_m_000000_1 to list of failed maps
> 2014-03-25 21:33:46,014 FATAL [IPC Server handler 15 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1395753558524_0002_r_000000_2 - exited :
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:46,014 INFO [IPC Server handler 15 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1395753558524_0002_r_000000_2: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:46,015 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_2: Error:
> org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
> shuffle in fetcher#5
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
> bailing-out.
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
> at
> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>
> 2014-03-25 21:33:46,015 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from RUNNING
> to FAIL_CONTAINER_CLEANUP
> 2014-03-25 21:33:46,016 INFO [ContainerLauncher #7]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1395753558524_0002_01_000005 taskAttempt
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:46,016 INFO [ContainerLauncher #7]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:46,017 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from
> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
> 2014-03-25 21:33:46,017 INFO [CommitterEvent Processor #3]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: TASK_ABORT
> 2014-03-25 21:33:46,020 WARN [CommitterEvent Processor #3]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
> delete
> hdfs://master:9000/output/12/_temporary/1/_temporary/attempt_1395753558524_0002_r_000000_2
> 2014-03-25 21:33:46,021 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_2 TaskAttempt Transitioned from
> FAIL_TASK_CLEANUP to FAILED
> 2014-03-25 21:33:46,022 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_3 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2014-03-25 21:33:46,290 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:46,294 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:46,295 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the
> blacklist for application_1395753558524_0002: blacklistAdditions=1
> blacklistRemovals=0
> 2014-03-25 21:33:46,295 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore
> blacklisting set to true. Known: 3, Blacklisted: 1, 33%
> 2014-03-25 21:33:46,295 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=21504
> 2014-03-25 21:33:46,295 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
> completedMapPercent 0.0 totalMemLimit:22528 finalMapMemLimit:1024
> finalReduceMemLimit:21504 netScheduledMapMem:1024 netScheduledReduceMem:1024
> 2014-03-25 21:33:46,295 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Ramping up 1
> 2014-03-25 21:33:46,296 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:1 AssignedMaps:0
> AssignedReds:1 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:47,303 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=1
> finishedContainers=1 resourcelimit=<memory:21504, vCores:-1> knownNMs=3
> 2014-03-25 21:33:47,303 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the
> blacklist for application_1395753558524_0002: blacklistAdditions=0
> blacklistRemovals=1
> 2014-03-25 21:33:47,304 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1395753558524_0002_01_000005
> 2014-03-25 21:33:47,304 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:47,304 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1395753558524_0002_r_000000_2: Container killed by the
> ApplicationMaster.
> Container killed on request. Exit code is 143
>
> 2014-03-25 21:33:47,304 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning
> container Container: [ContainerId: container_1395753558524_0002_01_000006,
> NodeId: slave2:8041, NodeHttpAddress: slave2:8042, Resource: <memory:1024,
> vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service:
> 159.99.249.99:8041 }, ] to fast fail map
> 2014-03-25 21:33:47,304 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from
> earlierFailedMaps
> 2014-03-25 21:33:47,305 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000006 to
> attempt_1395753558524_0002_m_000000_1
> 2014-03-25 21:33:47,305 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:5 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:47,305 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave2 to /default-rack
> 2014-03-25 21:33:47,305 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_1 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:47,306 INFO [ContainerLauncher #8]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000006 taskAttempt
> attempt_1395753558524_0002_m_000000_1
> 2014-03-25 21:33:47,306 INFO [ContainerLauncher #8]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_m_000000_1
> 2014-03-25 21:33:47,306 INFO [ContainerLauncher #8]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : slave2:8041
> 2014-03-25 21:33:47,326 INFO [ContainerLauncher #8]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_m_000000_1
> : 13562
> 2014-03-25 21:33:47,326 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_m_000000_1] using containerId:
> [container_1395753558524_0002_01_000006 on NM: [slave2:8041]
> 2014-03-25 21:33:47,327 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_m_000000_1 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:47,327 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_m_000000
> 2014-03-25 21:33:47,327 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1395753558524_0002_m_000000 Task Transitioned from SCHEDULED to RUNNING
> 2014-03-25 21:33:48,313 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1395753558524_0002: ask=1 release= 0 newContainers=1
> finishedContainers=0 resourcelimit=<memory:20480, vCores:-2> knownNMs=3
> 2014-03-25 21:33:48,313 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2014-03-25 21:33:48,313 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to
> reduce
> 2014-03-25 21:33:48,314 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1395753558524_0002_01_000007 to
> attempt_1395753558524_0002_r_000000_3
> 2014-03-25 21:33:48,314 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:1 CompletedMaps:0 CompletedReds:0 ContAlloc:6 ContRel:0
> HostLocal:1 RackLocal:0
> 2014-03-25 21:33:48,315 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved slave1 to /default-rack
> 2014-03-25 21:33:48,316 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_3 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2014-03-25 21:33:48,318 INFO [ContainerLauncher #9]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1395753558524_0002_01_000007 taskAttempt
> attempt_1395753558524_0002_r_000000_3
> 2014-03-25 21:33:48,318 INFO [ContainerLauncher #9]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching attempt_1395753558524_0002_r_000000_3
> 2014-03-25 21:33:48,326 INFO [ContainerLauncher #9]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1395753558524_0002_r_000000_3
> : 13562
> 2014-03-25 21:33:48,326 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1395753558524_0002_r_000000_3] using containerId:
> [container_1395753558524_0002_01_000007 on NM: [slave1:8041]
> 2014-03-25 21:33:48,327 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1395753558524_0002_r_000000_3 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2014-03-25 21:33:48,328 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator:
> ATTEMPT_START task_1395753558524_0002_r_000000
> 2014-03-25 21:33:48,503 INFO [Socket Reader #1 for port 53159]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1395753558524_0002 (auth:SIMPLE)
> 2014-03-25 21:33:48,511 INFO [IPC Server handler 16 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1395753558524_0002_m_000006 asked for a task
> 2014-03-25 21:33:48,512 INFO [IPC Server handler 16 on 53159]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1395753558524_0002_m_000006 given task:
> attempt_1395753558524_0002_m_000000_1
>
>
>
> 2014-03-25 13:15 GMT+08:00 nishan shetty <nishan.shetty@huawei.com>:
>
>> Can you configure mapreduce_shuffle class name as below and check
>>
>> <property>
>> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
>> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>>
>> -----Original Message-----
>> From: Vincent,Wei [mailto:weikun0905@gmail.com]
>> Sent: 25 March 2014 10:05 AM
>> To: yarn-dev@hadoop.apache.org
>> Subject: About Map 100% reduce %0 issue
>>
>> All
>>
>> I am a new comer for Hadoop, I have run
>> the hadoop-mapreduce-examples-2.2.0.jar wordcount, but the result is that
>> it always pending at map 100% and reduce %0.
>>
>> 14/03/25 20:19:20 INFO client.RMProxy: Connecting to ResourceManager at
>> master/159.99.249.63:8032
>> 14/03/25 20:19:20 INFO input.FileInputFormat: Total input paths to process
>> : 1
>> 14/03/25 20:19:20 INFO mapreduce.JobSubmitter: number of splits:1
>> 14/03/25 20:19:20 INFO Configuration.deprecation: user.name is deprecated.
>> Instead, use mapreduce.job.user.name
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.jar is deprecated.
>> Instead, use mapreduce.job.jar
>> 14/03/25 20:19:20 INFO Configuration.deprecation:
>> mapred.output.value.class is deprecated. Instead, use
>> mapreduce.job.output.value.class
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.combine.class
>> is deprecated. Instead, use mapreduce.job.combine.class
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.map.class is
>> deprecated. Instead, use mapreduce.job.map.class
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.job.name is
>> deprecated. Instead, use mapreduce.job.name
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapreduce.reduce.class
>> is deprecated. Instead, use mapreduce.job.reduce.class
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.input.dir is
>> deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.output.dir is
>> deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.map.tasks is
>> deprecated. Instead, use mapreduce.job.maps
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.output.key.class
>> is deprecated. Instead, use mapreduce.job.output.key.class
>> 14/03/25 20:19:20 INFO Configuration.deprecation: mapred.working.dir is
>> deprecated. Instead, use mapreduce.job.working.dir
>> 14/03/25 20:19:20 INFO mapreduce.JobSubmitter: Submitting tokens for job:
>> job_1395747600383_0002
>> 14/03/25 20:19:20 INFO impl.YarnClientImpl: Submitted application
>> application_1395747600383_0002 to ResourceManager at master/
>> 159.99.249.63:8032
>> 14/03/25 20:19:20 INFO mapreduce.Job: The url to track the job:
>> http://master:8088/proxy/application_1395747600383_0002/
>> 14/03/25 20:19:20 INFO mapreduce.Job: Running job: job_1395747600383_0002
>> 14/03/25 20:19:24 INFO mapreduce.Job: Job job_1395747600383_0002 running
>> in uber mode : false
>> 14/03/25 20:19:24 INFO mapreduce.Job:  map 0% reduce 0%
>> 14/03/25 20:19:28 INFO mapreduce.Job:  map 100% reduce 0%
>> 14/03/25 20:19:31 INFO mapreduce.Job: Task Id :
>> attempt_1395747600383_0002_r_000000_0, Status : FAILED
>> Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
>> in shuffle in fetcher#5
>>         at
>> org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
>>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:380)
>>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>         at
>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
>> Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES;
>> bailing-out.
>>         at
>>
>> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.checkReducerHealth(ShuffleSchedulerImpl.java:323)
>>         at
>>
>> org.apache.hadoop.mapreduce.task.reduce.ShuffleSchedulerImpl.copyFailed(ShuffleSchedulerImpl.java:245)
>>         at
>>
>> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:347)
>>         at
>> org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:165)
>>
>> someone says that this is caused by hosts configure .I have checked my
>> /etc/hosts  on all Mater & slaves:
>> 127.0.0.1       localhost.localdomain localhost
>> 159.99.249.63   master
>> 159.99.249.203  slave1
>> 159.99.249.99   slave2
>> 159.99.249.88   slave3
>>
>> Would you please help me to fix the issue, many thanks .
>>
>> my yarn-site.xml
>>
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>
>>
>> <configuration>
>>
>> <property>
>> <description>The hostname of the RM.</description>
>> <name>yarn.resourcemanager.hostname</name>
>> <value>master</value>
>> </property>
>>
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle</value>
>> </property>
>>
>> <property>
>> <description>The address of the container manager in the NM.</description>
>> <name>yarn.nodemanager.address</name>
>> <value>${yarn.nodemanager.hostname}:8041</value>
>> </property>
>>
>> </configuration>
>>
>> my mapred-site.xml
>>
>> <configuration>
>>         <property>
>>         <name>mapreduce.framework.name</name>
>>         <value>yarn</value>
>>         </property>
>>
>> <property>
>>   <name>mapreduce.reduce.shuffle.merge.percent</name>
>>   <value>0.33</value>
>>   <description>The usage threshold at which an in-memory merge will be
>>   initiated, expressed as a percentage of the total memory allocated to
>>   storing in-memory map outputs, as defined by
>>   mapreduce.reduce.shuffle.input.buffer.percent.
>>   </description>
>> </property>
>>
>> <property>
>>   <name>mapreduce.reduce.shuffle.input.buffer.percent</name>
>>   <value>0.35</value>
>>   <description>The percentage of memory to be allocated from the maximum
>> heap
>>   size to storing map outputs during the shuffle.
>>   </description>
>> </property>
>>
>> <property>
>>   <name>mapreduce.reduce.shuffle.memory.limit.percent</name>
>>   <value>0.12</value>
>>   <description>Expert: Maximum percentage of the in-memory limit that a
>>   single shuffle can consume</description> </property>
>>
>> </configuration>
>>
>>
>> --
>> BR,
>>
>> Vincent.Wei
>>
>
>
>
> --
> BR,
>
> Vincent.Wei



-- 
Harsh J

Mime
View raw message