hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dinesh Chitlangia (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDDS-98) Adding Ozone Manager Audit Log
Date Fri, 24 Aug 2018 14:20:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591724#comment-16591724
] 

Dinesh Chitlangia edited comment on HDDS-98 at 8/24/18 2:19 PM:
----------------------------------------------------------------

[~jnp] Thank you for a detailed feedback.

In this implementation, we are using the StructuredDataMessage (conforms to RFC 5424) for
logging.

Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap {color:#f79232}message{color}
throwable}}
{quote}
 
 * type - Customer action types we have defined for ozone like CREATE_VOLUME, CREATE_BUCKET
etc 
 * ID - I can log a string "username IPAddress"
 * dataMap - It is a key-value map object that we prepare for logging and can thus add as
many key-value pairs as want. So this addresses the extensibility aspect. We cannot log username/IPAddress
as key-value pair in the dataMap because the dataMap will sort by key while printing and currently
there is no API to turn off sorting.
 * message - any specific message we may want to pass, it can also be left blank. We decided
to pass this with only two values SUCCESS and FAILURE depending upon the result. In future
when we build a log parser, this will be handy for querying various combinations.
 * throwable - if any exception was thrown in the actual operation, we can log it. This is
totally configurable via properties file to log - x lines of exception or error message or
not log at all.

Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit | {color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3 {color}{color:#14892c}admin="xiaoyu"
creationTime="0" owner="xiaoyu" quotaInBytes="1099511627776" volume="xmen"{color}] {color:#f79232}SUCCESS{color}
|}}
{quote}
> I can definitely change the "username/IP" to "username  IP" and ensure this is at top
level by logging it as ID.

> Extensibility has been addressed by {color:#14892c}dataMap{color}

> With this structure, the parsing will likely be easier as the actual data in enclosed
in [ ]. Even if there were nested [ ] inside this, I feel it will still be easy to parse
when we write a parser.

> Given the above structure, unfortunately, we won't be able to log like the example you
provided

Please do let me know if there are further improvements that can be made and thank you once
again for a detailed feedback. :)


was (Author: dineshchitlangia):
[~jnp] Thank you for a detailed feedback.

In this implementation, we are using the StructuredDataMessage (conforms to RFC 5424) for
logging.

Its basic structure is:
{quote}{{{color:#59afe1}type{color} {color:#654982}ID{color} dataMap {color:#f79232}message{color}
throwable}}
{quote}
 
 * type - Customer action types we have defined for ozone like CREATE_VOLUME, CREATE_BUCKET
etc 
 * ID - I can log a string "username IPAddress"
 * dataMap - It is a key-value map object that we prepare for logging and can thus add as
many key-value pairs as want. So this addresses the extensibility aspect. We cannot log username/IPAddress
as key-value pair in the dataMap because the dataMap will sort by key while printing and currently
there is no API to turn off sorting.
 * message - any specific message we may want to pass, it can also be left blank. We decided
to pass this with only two values SUCCESS and FAILURE depending upon the result. In future
when we build a log parser, this will be handy for querying various combinations.
 * throwable - if any exception was thrown in the actual operation, we can log it. This is
totally configurable via properties file to log - x lines of exception or error message or
not log at all.

Thus, we end up with a log entry like the one below.
{quote}{{2018-08-24 04:57:28,685 | INFO | OMAudit | {color:#59afe1}CREATE_VOLUME{color} [{color:#654982}xiaoyu/172.18.0.3 {color}{color:#14892c}admin="xiaoyu"
creationTime="0" owner="xiaoyu" quotaInBytes="1099511627776" volume="xmen"{color}] {color:#f79232}SUCCESS{color}
|}}
{quote}
> I can definitely change the "username/IP" to "username  IP" and ensure this is at top
level by logging it as ID.

> Extensibility has been addressed by {color:#14892c}dataMap{color}

{color:#14892c}> With this structure, the parsing will likely be easier as the actual data
in enclosed in [ ]. Even if there were nested [ ] inside this, I feel it will still be easy
to parse when we write a parser{color}

> Given the above structure, unfortunately, we won't be able to log like the example you
provided

Please do let me know if there are further improvements that can be made and thank you once
again for a detailed feedback. :)

> Adding Ozone Manager Audit Log
> ------------------------------
>
>                 Key: HDDS-98
>                 URL: https://issues.apache.org/jira/browse/HDDS-98
>             Project: Hadoop Distributed Data Store
>          Issue Type: Sub-task
>            Reporter: Xiaoyu Yao
>            Assignee: Dinesh Chitlangia
>            Priority: Major
>              Labels: Logging, audit
>             Fix For: 0.2.1
>
>         Attachments: HDDS-98.001.patch, HDDS-98.002.patch, HDDS-98.003.patch, HDDS-98.004.patch,
audit.log, log4j2.properties
>
>
> This ticket is opened to add ozone manager's audit log. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message