hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rand (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications
Date Fri, 14 Apr 2017 05:03:41 GMT

    [ https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968609#comment-15968609
] 

Steven Rand commented on YARN-2985:
-----------------------------------

[~jlowe], thanks for the thoughtful response. Based on that information, it seems like the
most straightforward way to proceed, at least for branch-2, is to add a configuration option
for running the deletion service in only the timeline server, and not the JHS. Something like
{{yarn.log-aggregation.run-in-timeline-server}} that defaults to {{false}} for backcompat,
but when set to {{true}}, prevents the JHS from performing retention, and tells the timeline
server to do it instead. Does that seem reasonable? If so I'll update the patch to do that,
but certainly open to alternatives if there's a better way.

For trunk, I imagine it might be worth just removing retention from the JHS and moving it
to the timeline server entirely, since my understanding is that the timeline server is supposed
to replace the JHS, even for deployments that only run MR jobs, and 3.0 seems like a reasonable
enough point at which to require the switch from JHS to timeline server. I might be misunderstanding
the relationship between the two though, so please correct me if that doesn't make sense.

> YARN should support to delete the aggregated logs for Non-MapReduce applications
> --------------------------------------------------------------------------------
>
>                 Key: YARN-2985
>                 URL: https://issues.apache.org/jira/browse/YARN-2985
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: log-aggregation, nodemanager
>    Affects Versions: 2.8.0
>            Reporter: Xu Yang
>            Assignee: Steven Rand
>         Attachments: YARN-2985-branch-2-001.patch
>
>
> Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But the AggregatedLogDeletionService
is started in mapreduce`s JobHistoryServer. Therefore, the Non-MapReduce application can aggregate
their logs to HDFS, but can not delete those logs. Need the NodeManager take over the function
of aggregated log deletion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message