hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erik Krogen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13791) Limit logging frequency of edit tail related statements
Date Fri, 31 Aug 2018 22:01:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16599311#comment-16599311

Erik Krogen commented on HDFS-13791:

[~csun] thanks for looking!

For alignment across logging locations, I considered this as well but decided to skip it because
I didn't think it was too important and at the time I couldn't think of a way to achieve it
cleanly. I am now thinking if we could pass the previous {{LogAction}} into the subsequent
statement and use this to align them... I think it should work out pretty cleanly; I'll give
it a try.

For your second point, I think there is a question of how complicated we want to make this
class to achieve generality. I guess my inclination would be to only build it out where we
actually have a use case and then modify as necessary (given it is not public, we can change
it anytime). One option would be to store the values as a [SummaryStatistics|http://commons.apache.org/proper/commons-math/apidocs/org/apache/commons/math4/stat/descriptive/SummaryStatistics.html]
and export this so that the caller has full flexibility in what information to extract.

> Limit logging frequency of edit tail related statements
> -------------------------------------------------------
>                 Key: HDFS-13791
>                 URL: https://issues.apache.org/jira/browse/HDFS-13791
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs, qjm
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Major
>         Attachments: HDFS-13791-HDFS-12943.000.patch
> There are a number of log statements that occur every time new edits are tailed by a
Standby NameNode. When edits are tailing only on the order of every tens of seconds, this
is fine. With the work in HDFS-13150, however, edits may be tailed every few milliseconds,
which can flood the logs with tailing-related statements. We should throttle it to limit it
to printing at most, say, once per 5 seconds.
> We can implement logic similar to that used in HDFS-10713. This may be slightly more
tricky since the log statements are distributed across a few classes.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message