hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3680) Allows customized audit logging in HDFS FSNamesystem
Date Thu, 02 Aug 2012 17:24:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427470#comment-13427470

Marcelo Vanzin commented on HDFS-3680:

Hi Daryn,

bq. The NN has an option for an external logger daemon

So, this implies some sort of IPC, either to another local process or possibly even a remote
one. Which raises different questions, some of which have been asked before:

* Are audit logs written synchronously?
* If yes, is the extra latency for the RPC call acceptable?
* If no, how do we reliably know that audit logs have been written?
* What's an acceptable timeout for the RPC call?

And it doesn't really answer the question of "what to do on failure", although the consensus
there seems to be to "abort". I'm not against this solution per se, I just don't think that,
in the end, it's much different than allowing 3rd party code into the NN. I'll refer back
to ATM's comment that we expect people who are writing audit loggers to know what they're

On a different note, I played with the "let exceptions flow" approach, and it seems that the
NN catches the exception at some point and morphs it into a RemoteException. I actually like
that better than shutting down the NN; it fails the particular operation, without shutting
down the NN itself, which allows the logger to recover if it can. Of course a really broken
logger will just keep throwing exceptions, but then IMO the user is asking for it by installing
a broken custom logger.
> Allows customized audit logging in HDFS FSNamesystem
> ----------------------------------------------------
>                 Key: HDFS-3680
>                 URL: https://issues.apache.org/jira/browse/HDFS-3680
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 2.0.0-alpha
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>         Attachments: accesslogger-v1.patch, accesslogger-v2.patch, hdfs-3680-v3.patch,
hdfs-3680-v4.patch, hdfs-3680-v5.patch
> Currently, FSNamesystem writes audit logs to a logger; that makes it easy to get audit
logs in some log file. But it makes it kinda tricky to store audit logs in any other way (let's
say a database), because it would require the code to implement a log appender (and thus know
what logging system is actually being used underneath the fa├žade), and parse the textual
log message generated by FSNamesystem.
> I'm attaching a patch that introduces a cleaner interface for this use case.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message