hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3680) Allows customized audit logging in HDFS FSNamesystem
Date Fri, 05 Oct 2012 19:24:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470579#comment-13470579

Suresh Srinivas commented on HDFS-3680:

bq. You're making assumptions about the implementation of the logger. Why would it throw IOException
and not SQLException? What if my logger doesn't do any I/O in the thread doing the logging
at all? Saying "throws IOException" would just make implementors wrap whatever is the real
exception being thrown in an IOException instead of a RuntimeException, to no benefit I can
see. If FSNamesystem should handle errors from custom loggers, it should handle all errors,
not just specific ones.
Generally this type of interface could be implemented by implementation that can connect to
local or remote service. Even though only remote implementations throw IOException, the interfaces
are defined to throw IOExceptions. See our RPC interface examples.

That said if you do not want to throw IOException, which I recommend, is fine. However you
need to make sure you throw a checked exception from this interface, to design it correctly,
to force the caller to handle the error condition. RTE which typically indicates programming
error is not the choice.

bq. We can't prevent badly written code from doing bad things (also see previous comments
from ATM that this is not an interface that people will be implementing willy nilly - people
who'll touch it are expected to know what they are doing). The reason I say it's out of the
scope of this patch is because it's a change in the current behavior that's unrelated to whether
you have a custom audit logger or not; if audit logs should cause the name node to shut down,
that's a change that needs to be made today, right now, independent of this patch going in
or not.

I disagree. You are allowing audit logger to be pluggable in this patch. What is the impact
of making logging pluggable and how namenode must deal with is the relevant questions for
this patch. Not sure you looked at this comment from Todd, that I agree with:
bq. +1. I believe this is the case with log4j – if the current log4j volume fills up, I'm
pretty sure I've seen the NN just abort itself, rather than keep going without logs. I prefer
this behavior.

Other thing I have not completely thought through but bother me is, when these failures happen,
a failure response is sent to the client, though it is successful on Namenode and is recorded
on editlog. I would like others opinion on this.

bq. Details about what the implementation is expected to do should be (and are) documented
in the interface itself, which is the interface that the person writing the implementation
will be looking at.
I think you still need this in the hdfs-default.xml. The programmers who implements this interface
and an administrator who configures it could be different. The side effects of this configuration
change should be documented for an administrator so he understands the impact of this.

> Allows customized audit logging in HDFS FSNamesystem
> ----------------------------------------------------
>                 Key: HDFS-3680
>                 URL: https://issues.apache.org/jira/browse/HDFS-3680
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>         Attachments: accesslogger-v1.patch, accesslogger-v2.patch, hdfs-3680-v3.patch,
hdfs-3680-v4.patch, hdfs-3680-v5.patch, hdfs-3680-v6.patch, hdfs-3680-v7.patch, hdfs-3680-v8.patch
> Currently, FSNamesystem writes audit logs to a logger; that makes it easy to get audit
logs in some log file. But it makes it kinda tricky to store audit logs in any other way (let's
say a database), because it would require the code to implement a log appender (and thus know
what logging system is actually being used underneath the façade), and parse the textual
log message generated by FSNamesystem.
> I'm attaching a patch that introduces a cleaner interface for this use case.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message