hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Botong Huang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5525) Make log aggregation service class configurable
Date Fri, 23 Sep 2016 21:16:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15517594#comment-15517594
] 

Botong Huang edited comment on YARN-5525 at 9/23/16 9:16 PM:
-------------------------------------------------------------

Thank [~subru] for the review and comments. Let me provide some context why we are doing this.
By changing {{AppLogAggregator}} we can only customize the per app log aggregation behavior.
One reason that we also want to customize {{LogAggregationService}} is to allow a different
root log directory to upload logs to per app, whereas in the current implementation, logs
in all apps are uploaded to a same (configurable) location in {{LogAggregationService.remoteRootLogDir}}.
In general, if we want to make log aggregation pluggable, I think {{LogAggregationService}}
is the right place because it is the entry service class. 

Besides, I have a question regarding all the {{private}} to {{protected}} changes, where I
am sure what is the right thing to do. In general, making log aggregation service class configurable
will suffice and people can plug in any other implementation they need, without needing to
modify {{LogAggregationService}} and {{AppLogAggregator}} at all. However, in our case, we
only need to customize some features out of the current implementation. Rather copy all the
code and modify on top, I implemented mine extending the current ones, so that lots of nice
feature are inherited. The code is much less and easier to maintain, by simply override the
methods that we need to customize. This is where the {{private}} to {{protected}} changes
are needed so that the member variables and methods are visible to the subclasses. 


was (Author: botong):
Thank [~subru] for the review and comments. Let me provide some context why we are doing this.
By changing {{AppLogAggregator}} we can only customize the per app log aggregation behavior.
One reason that we also want to customize {{LogAggregationService}} is to allow a different
root log directory to upload logs to per app, whereas in the current implementation, logs
in every app are uploaded to a same (configurable) location in {{LogAggregationService.remoteRootLogDir}}.
In general, if we want to make log aggregation pluggable, I think {{LogAggregationService}}
is the right place because it is the entry service class. 

Besides, I have a question regarding all the {{private}} to {{protected}} changes, where I
am sure what is the right thing to do. In general, making log aggregation service class configurable
will suffice and people can plug in any other implementation they need, without needing to
modify {{LogAggregationService}} and {{AppLogAggregator}} at all. However, in our case, we
only need to customize some features out of the current implementation. Rather copy all the
code and modify on top, I implemented mine extending the current ones, so that lots of nice
feature are inherited. The code is much less and easier to maintain, by simply override the
methods that we need to customize. This is where the {{private}} to {{protected}} changes
are needed so that the member variables and methods are visible to the subclasses. 

> Make log aggregation service class configurable
> -----------------------------------------------
>
>                 Key: YARN-5525
>                 URL: https://issues.apache.org/jira/browse/YARN-5525
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: log-aggregation
>            Reporter: Giovanni Matteo Fumarola
>            Assignee: Botong Huang
>            Priority: Minor
>         Attachments: YARN-5525.v1.patch, YARN-5525.v2.patch, YARN-5525.v3.patch
>
>
> Make the log aggregation class configurable and extensible, so that alternative log aggregation
behaviors like app specific log aggregation directory, log aggregation format can be implemented
and plugged in.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message