ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Quiroga <quirogadf4w...@gmail.com>
Subject Ranger Logging Health Check Ambari Alert
Date Fri, 22 Dec 2017 18:16:31 GMT
Hello

First some background:

We were directed to retain audit/access records "forever" (technically 7
years but that is basically forever in electronic log time).

Each Hadoop component generates local audit logs as per their log4j
settings. In our production system these logs would frequently fill up the
disk. At first we would just compress them in place but that only works for
so long and there was no redundancy with local disk storage. In others
words, no long term plan.

We started to discuss moving them to HDFS or a different storage solution.
One of our team members pointed out the Ranger plugins are already logging
the "same data" into HDFS.
Probably after several meeting with the higher-ups, using Ranger logs as
the record truth was approved. Components log4j settings were updated to
purge data automatically.

Purging local logs felt like operating with out a safety net.
Thought it we be good to check that Ranger was successful logging to HDFS
each day. Should mention this is a kerberized cluster, not that anything
ever goes wrong with kerberos.

Checking this would have certainly been possible with a shell script, but
we have been pushing to centralize warning/alerts in Ambari. And so an
Ambari alert python script to check on Ranger Logging Health was crafted.

For the most part the alert was modeled after some of the hive alerts.
At the moment it just checks that the daily /ranger/audit/<component> HDFS
directory has been created.

I am sure there is room for improvement but I was curious:

1. Has anyone run into this type of concern?
    a. Would an alert like this be helpful?
    b. Did you come up with another solution?

2. What is best way to get this out into the community (e.g. JIRA, if so
Ranger or Ambari - I am checking with both mailing list)?
  a. Any other advice on how to best share?

Thank you for your time.
-David

Mime
View raw message