hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3680) Allows customized audit logging in HDFS FSNamesystem
Date Fri, 05 Oct 2012 19:42:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470589#comment-13470589

Marcelo Vanzin commented on HDFS-3680:

bq. That said if you do not want to throw IOException, which I recommend, is fine. However
you need to make sure you throw a checked exception from this interface, to design it correctly,
to force the caller to handle the error condition. RTE which typically indicates programming
error is not the choice.

My main point in resisting declaring any checked exception here is that not doing that makes
it clear that loggers are not expected to throw exceptions, and thus should avoid doing so
whenever possible. From FSNamesystem's point of view, there's no point in handling checked
and unchecked exceptions differently; and audit log failure should be handled the same way
regardless of the nature of the failure, unless you want to do crazy things like have a "RetriableFailureException"
to indicate that FSNamesystem can call the method again to see if it works.

bq. I disagree. You are allowing audit logger to be pluggable in this patch. What is the impact
of making logging pluggable and how namenode must deal with is the relevant questions for
this patch. Not sure you looked at this comment from Todd, that I agree with:

I'm making it easier, but as I've mentioned before, it's already possible to use custom loggers
by doing it at the log system layer (e.g. a custom log4j appender; just use http://wiki.apache.org/logging-log4j/JDBCAppenderConfiguration,
for example, and you have many more failure cases than just writing to a local file). 

That's why I say that if you're worried about that case, that particular change should be
made in the current code regardless of this patch. But if you feel strongly about it, I can
add a try...catch that just shuts down the namenode.

(BTW, unless log4j itself shuts down the process by calling System.exit(), which my experience
says is not the case, I don't see where this "shutdown on audit log error" code is.)

bq. Other thing I have not completely thought through but bother me is, when these failures
happen, a failure response is sent to the client, though it is successful on Namenode and
is recorded on editlog. I would like others opinion on this.

That's independent of this patch (ignoring the "broken audit logger" argument). The problem
is that the audit log is written after the operation succeeds; to fix that, the permission
check and implementation of each operation should be separated, so that you could write the
audit log before executing the operation.

bq. The side effects of this configuration change should be documented for an administrator
so he understands the impact of this.

What side effects are you talking about? The only one would be "if the audit logger fails,
namenode will shut down", if that behavior is implemented. Otherwise, any side-effect is implementation-specific
and it's impossible to say anything other than "there might be side effects".

> Allows customized audit logging in HDFS FSNamesystem
> ----------------------------------------------------
>                 Key: HDFS-3680
>                 URL: https://issues.apache.org/jira/browse/HDFS-3680
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>         Attachments: accesslogger-v1.patch, accesslogger-v2.patch, hdfs-3680-v3.patch,
hdfs-3680-v4.patch, hdfs-3680-v5.patch, hdfs-3680-v6.patch, hdfs-3680-v7.patch, hdfs-3680-v8.patch
> Currently, FSNamesystem writes audit logs to a logger; that makes it easy to get audit
logs in some log file. But it makes it kinda tricky to store audit logs in any other way (let's
say a database), because it would require the code to implement a log appender (and thus know
what logging system is actually being used underneath the fa├žade), and parse the textual
log message generated by FSNamesystem.
> I'm attaching a patch that introduces a cleaner interface for this use case.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message