hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3680) Allows customized audit logging in HDFS FSNamesystem
Date Wed, 01 Aug 2012 23:37:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427000#comment-13427000
] 

Marcelo Vanzin commented on HDFS-3680:
--------------------------------------

bq. Given that on the first exception we throw away a logger, a logger could miss whole bunch
of audit logs.

You're talking about letting audit loggers recover in some way. I don't think you can have
it both ways: either misbehaving loggers are removed, or they're always called, and we just
eat the exceptions and let them misbehave for as long as they want to, and potentially recover
by themselves. Anything more complicated is, well, more complicated, and I don't think it's
worth the extra complexity, even if there is something that could possibly be done.

This covers the "add / remove" audit loggers question too. I don't think that's needed. It's
a configuration option; this is really not something that should be changing during runtime.

bq. So a mechanism to bring the out of sync audit logger "in sync" will be needed. 

Again you're assuming that we should try to fix broken audit loggers, something that we can't
do. I'm treating broken audit loggers as just that: broken. If they want to not lose audit
messages, they should make sure they don't throw exceptions, and handle any unexpected situations
internally.

bq. I am not sure I understand why it creates more problems.

Well, let me rephrase. Talking about a separate daemon to serve audit logs is moot because,
well, it wouldn't be HDFS code anymore. Anyone can do that without requiring a single line
change in HDFS.

My point with submitting this patch is that I think it's worth it to have an interface closer
to the core for audit logs. It provides loggers with richer information about the context
of the logs, information that may not exist in the text messages written to a log file. It
also provides an interface that is more stable than a string in a log file (as in, if you
change it, the compiler will yell at you).

Just to reiterate my previous point, it is possible today to have a custom audit logger inside
the NameNode without my patch; you just have to implement a log appender for the log system
in use, and configure the logging system accordingly. With that, you get all the issues with
custom loggers (they run synchronously in a critical section of the NameNode, they can bring
down the whole process, etc) without any of the benefits of a proper interface (they have
to parse a string to get any context data about the log, potentially losing info). Which is
what led me to propose a cleaner way of doing it.
                
> Allows customized audit logging in HDFS FSNamesystem
> ----------------------------------------------------
>
>                 Key: HDFS-3680
>                 URL: https://issues.apache.org/jira/browse/HDFS-3680
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>         Attachments: accesslogger-v1.patch, accesslogger-v2.patch, hdfs-3680-v3.patch,
hdfs-3680-v4.patch, hdfs-3680-v5.patch
>
>
> Currently, FSNamesystem writes audit logs to a logger; that makes it easy to get audit
logs in some log file. But it makes it kinda tricky to store audit logs in any other way (let's
say a database), because it would require the code to implement a log appender (and thus know
what logging system is actually being used underneath the fa├žade), and parse the textual
log message generated by FSNamesystem.
> I'm attaching a patch that introduces a cleaner interface for this use case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message