eagle-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SUDHA JENSLIN <sjens...@gmail.com>
Subject Re: HDFS Data Activity Monitoring with eagle 0.5.0-SNAPSHOT
Date Wed, 22 Mar 2017 11:34:57 GMT
You can also refer : 
http://eagle.apache.org/docs/import-hdfs-auditLog.html <http://eagle.apache.org/docs/import-hdfs-auditLog.html>

Try having the log stash conf filter and output similar to this:


filter {
 grok {
   match => ["message", "ugi=(?<user>([\w\d\-]+))@|ugi=(?<user>([\w\d\-]+))/[\w\d\-.]+@|ugi=(?<user>([\w\d.\-_]+))[\s(]+"]
  }

}

output {
   stdout {
     codec => rubydebug
   }
   kafka {
     codec => plain {
                 format => "%{message}"
             }
   bootstrap_servers => "127.0.0.1:6667"
   topic_id => "TestTopic"
   retry_backoff_ms => 100
   send_buffer_bytes => 102400



 }
 }


-Sudha Jenslin



> On Mar 21, 2017, at 7:53 PM, Jean Rossier <jean@sqooba.io> wrote:
> 
> Hello,
> 
> I'm trying to setup the HDFS data activity monitoring stream (as explained here http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html
<http://eagle.apache.org/docs/hdfs-data-activity-monitoring.html>), but I don't see
any logs flowing into my Kafka topic.
> 
> Is there anything to do that is not on the quick start page (mentioned above) ?
> 
> One thing that surprised me, is that I don't have any log4jkafka folder in /usr/hdp/current/eagle/lib/
(I'm using eagle 0.5.0-SNAPSHOT, from current master branch). I'm supposed to add the libs
from this folder to the HADOOP_CLASSPATH env variable. I moved them by hand in the proper
folder, but, still, I don't see any logs flowing into my Kafka topic.
> 
> I don't see any error regarding kafka in the HDFS audit logs. There are, though, some
solr exceptions but I guess it shouldn't impact the stream I'm interested in.
> 
> Thanks
> Jean
> 


Mime
View raw message