hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuan Gong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
Date Fri, 13 Apr 2018 22:44:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438029#comment-16438029
] 

Xuan Gong commented on MAPREDUCE-7042:
--------------------------------------

[~rohithsharma]

The basic workflow will be:
 * When we got the JHS event, we push the event to JHS event queue.
 * We have separate JHS event handler thread periodically pick and handle the event from JHS
event queue.
 * When we start to handle the event, if ATS is enabled, we could push the event immediately
into ATS event queue.
 * We have separate ATS event handler thread periodically pick and handle the event from ATS
event queue. 

 

To separate JHS event and ATS event here, I think that I need two different queues and handler
threads. So, when we handle the event, we can simply push the event into ATS event queue and
continue to handle this as JHS event. So, even there is anything wrong with ATS, it would
not block the process of handling JHS event.

 

> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2
is enabled
> -----------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7042
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Yesha Vora
>            Assignee: Xuan Gong
>            Priority: Major
>         Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
sleep "-Dmapreduce.job.user.name=hrt_qa"   -m 10 -r 1 -mt 1000  -rt 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message