hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "anikad ayman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14858) Why Yarn crashes ?
Date Mon, 11 Sep 2017 15:00:09 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

anikad ayman updated HADOOP-14858:
----------------------------------
    Description: 
During MapReduce processing, Yarn did crash and the processing of jobs had stopped. I successed
to back the processing after killing the first job which was running, but after some minutes,
another crach thatI solved by killing the second job wich was running.

 

We are looking for reasons of this crach that we had several times before (between one to
two times in a month)

 

In ressource manager logs , I find this messages repeated from the beggining of the crach
until the killing of the jobs:

 

    2017-08-25 03:51:58,815 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38352
Call#33361 Retry#0
    2017-08-25 03:53:39,255 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38456
Call#33364 Retry#0
    2017-08-25 03:55:19,700 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38556
Call#33367 Retry#0
    2017-08-25 03:57:00,262 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38674
Call#33370 Retry#0
    2017-08-25 03:58:40,687 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38804
Call#33373 Retry#0
    .
    .
    .
    2017-08-25 11:02:44,086 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34159 Retry#0
    2017-08-25 11:02:47,933 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34162 Retry#0
    2017-08-25 11:03:06,800 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39814
Call#34165 Retry#0

 

NB: We still get this warning from time to another, we still wondring if it concerns a connexion
between the node manager (10.135.8.101) and the ressource manager, or something else ?

 

For the node manager logs, I find theses messages :

    2017-08-25 03:51:54,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 03:51:54,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
2.3 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 03:51:55,177 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
619.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 03:51:58,938 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    .
    .
    .
    2017-08-25 11:05:40,104 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 11:05:40,493 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 11:05:43,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 11:05:45,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
    2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_e41_1500982512144_36627_01_001644 transitioned from RUNNING to KILLING
    2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1500982512144_36627 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
    2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_e41_1500982512144_36627_01_001644


and also for the job history :

    2017-08-25 03:53:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 03:56:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 03:59:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 04:02:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 04:05:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 04:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 04:11:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files 


.
.
.

    2017-08-25 11:05:36,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
started
    2017-08-25 11:05:41,271 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
complete
    2017-08-25 11:06:04,214 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
    2017-08-25 11:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan
to move intermediate done files
    2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36793,submitTime=1503647426340,launchTime=1503651960434,firstMapTaskLaunchTime=1503651982671,firstReduceTaskLaunchTime=0,finishTime=1503651985794,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=9,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f62736973`.`buk...20170825)(Stage-1)
    2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Deleting
JobSummary file: [maprfs:/var/mapr/cluster/yarn/rm/staging/history/done_intermediate/mapr/job_1500982512144_36793.summary]
    2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36778,submitTime=1503642110785,launchTime=1503651960266,firstMapTaskLaunchTime=1503651969483,firstReduceTaskLaunchTime=0,finishTime=1503651976016,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=19,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f7662726b`.`vbe...20170825)(Stage-1)

 

Please, have you any explication or solution of this issue ?

  was:
During MapReduce processing, Yarn did crash and the processing of jobs had stopped. I successed
to back the processing after killing the first job which was running, but after some minutes,
another crach thatI solved by killing the second job wich was running.

We are looking for reasons of this crach that we had several times before (between one to
two times in a month)

In ressource manager logs , I find this messages repeated from the beggining of the crach
until the killing of the jobs:

2017-08-25 03:51:58,815 WARN org.apache.hadoop.ipc.Server: Large response size 4739374 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38352
Call#33361 Retry#0
2017-08-25 03:53:39,255 WARN org.apache.hadoop.ipc.Server: Large response size 4739374 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38456
Call#33364 Retry#0
2017-08-25 03:55:19,700 WARN org.apache.hadoop.ipc.Server: Large response size 4739374 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38556
Call#33367 Retry#0
2017-08-25 03:57:00,262 WARN org.apache.hadoop.ipc.Server: Large response size 4739374 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38674
Call#33370 Retry#0
2017-08-25 03:58:40,687 WARN org.apache.hadoop.ipc.Server: Large response size 4739374 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38804
Call#33373 Retry#0
.
.
.
2017-08-25 11:02:44,086 WARN org.apache.hadoop.ipc.Server: Large response size 4751251 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34159 Retry#0
2017-08-25 11:02:47,933 WARN org.apache.hadoop.ipc.Server: Large response size 4751251 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34162 Retry#0
2017-08-25 11:03:06,800 WARN org.apache.hadoop.ipc.Server: Large response size 4751251 for
call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39814
Call#34165 Retry#0

NB: We still get this warning from time to another, we still wondring if it concerns a connexion
between the node manager (10.135.8.101) and the ressource manager, or something else ?

For the node manager logs, I find theses messages :

2017-08-25 03:51:54,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 03:51:54,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
2.3 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 03:51:55,177 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
619.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 03:51:58,938 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
.
.
.
2017-08-25 11:05:40,104 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 11:05:40,493 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 11:05:43,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 11:05:45,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_e41_1500982512144_36627_01_001644 transitioned from RUNNING to KILLING
2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1500982512144_36627 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_e41_1500982512144_36627_01_001644

and also for the job history :

2017-08-25 03:53:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 03:56:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 03:59:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 04:02:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 04:05:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 04:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 04:11:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files 

. . .

2017-08-25 11:05:36,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
started
2017-08-25 11:05:41,271 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner
complete
2017-08-25 11:06:04,214 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
2017-08-25 11:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to
move intermediate done files
2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36793,submitTime=1503647426340,launchTime=1503651960434,firstMapTaskLaunchTime=1503651982671,firstReduceTaskLaunchTime=0,finishTime=1503651985794,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=9,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f62736973`.`buk...20170825)(Stage-1)
2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Deleting
JobSummary file: [maprfs:/var/mapr/cluster/yarn/rm/staging/history/done_intermediate/mapr/job_1500982512144_36793.summary]
2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36778,submitTime=1503642110785,launchTime=1503651960266,firstMapTaskLaunchTime=1503651969483,firstReduceTaskLaunchTime=0,finishTime=1503651976016,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=19,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f7662726b`.`vbe...20170825)(Stage-1)

Please, have you any explication or solution of this issue ?



> Why Yarn crashes ?
> ------------------
>
>                 Key: HADOOP-14858
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14858
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: Production 
>            Reporter: anikad ayman
>             Fix For: 2.7.0
>
>
> During MapReduce processing, Yarn did crash and the processing of jobs had stopped. I
successed to back the processing after killing the first job which was running, but after
some minutes, another crach thatI solved by killing the second job wich was running.
>  
> We are looking for reasons of this crach that we had several times before (between one
to two times in a month)
>  
> In ressource manager logs , I find this messages repeated from the beggining of the crach
until the killing of the jobs:
>  
>     2017-08-25 03:51:58,815 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38352
Call#33361 Retry#0
>     2017-08-25 03:53:39,255 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38456
Call#33364 Retry#0
>     2017-08-25 03:55:19,700 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38556
Call#33367 Retry#0
>     2017-08-25 03:57:00,262 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38674
Call#33370 Retry#0
>     2017-08-25 03:58:40,687 WARN org.apache.hadoop.ipc.Server: Large response size 4739374
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:38804
Call#33373 Retry#0
>     .
>     .
>     .
>     2017-08-25 11:02:44,086 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34159 Retry#0
>     2017-08-25 11:02:47,933 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39778
Call#34162 Retry#0
>     2017-08-25 11:03:06,800 WARN org.apache.hadoop.ipc.Server: Large response size 4751251
for call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplications from 10.135.8.101:39814
Call#34165 Retry#0
>  
> NB: We still get this warning from time to another, we still wondring if it concerns
a connexion between the node manager (10.135.8.101) and the ressource manager, or something
else ?
>  
> For the node manager logs, I find theses messages :
>     2017-08-25 03:51:54,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 03:51:54,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
2.3 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 03:51:55,177 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
619.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 03:51:58,938 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.4 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     .
>     .
>     .
>     2017-08-25 11:05:40,104 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 112912 for container-id container_e41_1500982512144_36679_01_000387:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 11:05:40,493 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 11:05:43,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 98201 for container-id container_e41_1500982512144_36679_01_000382:
1.1 GB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 11:05:45,040 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 105848 for container-id container_e41_1500982512144_36627_01_001644:
648.4 MB of 10 GB physical memory used; 10.1 GB of 21 GB virtual memory used
>     2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
Container container_e41_1500982512144_36627_01_001644 transitioned from RUNNING to KILLING
>     2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl:
Application application_1500982512144_36627 transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
>     2017-08-25 11:05:48,397 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_e41_1500982512144_36627_01_001644
> and also for the job history :
>     2017-08-25 03:53:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 03:56:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 03:59:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 04:02:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 04:05:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 04:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 04:11:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files 
> .
> .
> .
>     2017-08-25 11:05:36,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History
Cleaner started
>     2017-08-25 11:05:41,271 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History
Cleaner complete
>     2017-08-25 11:06:04,214 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
Updating the current master key for generating delegation tokens
>     2017-08-25 11:08:06,504 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting
scan to move intermediate done files
>     2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36793,submitTime=1503647426340,launchTime=1503651960434,firstMapTaskLaunchTime=1503651982671,firstReduceTaskLaunchTime=0,finishTime=1503651985794,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=9,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f62736973`.`buk...20170825)(Stage-1)
>     2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager:
Deleting JobSummary file: [maprfs:/var/mapr/cluster/yarn/rm/staging/history/done_intermediate/mapr/job_1500982512144_36793.summary]
>     2017-08-25 11:08:06,518 INFO org.apache.hadoop.mapreduce.jobhistory.JobSummary: jobId=job_1500982512144_36778,submitTime=1503642110785,launchTime=1503651960266,firstMapTaskLaunchTime=1503651969483,firstReduceTaskLaunchTime=0,finishTime=1503651976016,resourcesPerMap=5120,resourcesPerReduce=0,numMaps=1,numReduces=0,user=mapr,queue=default,status=SUCCEEDED,mapSlotSeconds=19,reduceSlotSeconds=0,jobName=SELECT
`C_7361705f7662726b`.`vbe...20170825)(Stage-1)
>  
> Please, have you any explication or solution of this issue ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message