hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12746) DataNode Audit Logger
Date Mon, 30 Oct 2017 20:01:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225651#comment-16225651

Anu Engineer commented on HDFS-12746:

[~xkrogen] It is an interesting idea. Recently I had an off-line chat with [~arpitagarwal]
and we came to the conclusion that it would be nice to have an audit log in the data nodes.
In the idea that we chatted about, we were thinking that we should log all commands that come
to data node, both via heartbeats and from the client. Would that be something that is interesting
to you? We also wanted these to be machine-readable -- so that RPC sans data can be logged.
This gives us the ability to reconstruct data node activity. I am hoping that this data is
useful for trace-based simulation also.

> DataNode Audit Logger
> ---------------------
>                 Key: HDFS-12746
>                 URL: https://issues.apache.org/jira/browse/HDFS-12746
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, logging
>            Reporter: Erik Krogen
> I would like to discuss adding in an audit logger for the Datanodes. We have audit logging
on pretty much all other components: Namenode, ResourceManager, NodeManager. It seems the
DN should have a similar concept to log, at minimum, all block reads/writes. I think all of
the interesting information does already appear in the DN logs at INFO level but it would
be nice to have a specific audit class that this gets logged through, a la {{RMAuditLogger}}
and {{NMAuditLogger}}, to enable special handling.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message