incubator-cvs mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Incubator Wiki] Update of "EagleProposal" by ArunManoharan
Date Mon, 19 Oct 2015 06:46:27 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Incubator Wiki" for change notification.

The "EagleProposal" page has been changed by ArunManoharan:
https://wiki.apache.org/incubator/EagleProposal?action=diff&rev1=3&rev2=4

- == Eagle ==
+ = Eagle =
  
- === Abstract ===
+ == Abstract ==
  Eagle is an Open Source Monitoring solution for Hadoop to instantly identify access to sensitive
data, recognize attacks, malicious activities in hadoop and take actions. 
  
- === Proposal ===
+ == Proposal ==
  Eagle audits access to HDFS files, Hive and HBase tables in real time, enforces
  policies defined on sensitive data access and alerts or blocks user’s access to that sensitive
data in real time. Eagle also creates user profiles based on the typical access behaviour
for HDFS and Hive and sends alerts when anomalous behaviour is detected. Eagle can also import
sensitive data information classified by external classification engines to help define its
policies.
  
  === Overview of Eagle ===
  
  Eagle has 3 main parts. 
- Data collection and storage - Eagle collects data from various hadoop logs in real time
using Kafka/Yarn API and uses HDFS and HBase for storage.
+ '''Data collection and storage''' - Eagle collects data from various hadoop logs in real
time using Kafka/Yarn API and uses HDFS and HBase for storage.
- Data processing and policy engine - Eagle allows users to create policies based on various
metadata properties on HDFS, Hive and HBase data. 
+ '''Data processing and policy engine''' - Eagle allows users to create policies based on
various metadata properties on HDFS, Hive and HBase data. 
+ '''Eagle services''' - Eagle services include policy manager, query service and the visualization
component. Eagle provides intuitive user interface to administer Eagle and an alert dashboard
to respond to real time alerts. 
- Eagle services - Eagle services include policy manager, query service and the visualization
component. 
- Eagle provides intuitive user interface to administer Eagle and an alert dashboard to respond
to real time alerts. 
  
  === Eagle Architecture ===
  
- ==== Data Collection and Storage: ====
+ === Data Collection and Storage: ===
  
  Eagle provides programming API for extending Eagle to integrate any data source into Eagle
policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka
which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring
collects hive query logs from running job through YARN API, which is designed to be scalable
and fault-tolerant.
  Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational
database through configuration change.
  
- ==== Data Processing and Policy Engine: ====
+ === Data Processing and Policy Engine: ===
  
  Processing Engine: Eagle provides stream processing API which is an abstraction of Apache
Storm. It can also be extended to other streaming engines. This abstraction allows developers
to assemble data transformation, filtering, external data join etc. without physically bound
to a specific streaming platform. Eagle streaming API allows developers to easily integrate
business logic with Eagle policy engine and internally Eagle framework compiles business logic
execution DAG into program primitives of underlying stream infrastructure e.g. Apache Storm.
For example, Eagle HDFS monitoring transforms audit log from Namenode to object and joins
sensitivity metadata, security zone metadata which are generated from external programs or
configured by user. Eagle hive monitoring filters running jobs to get hive query string and
parses query string into object and then joins sensitivity metadata.
  

---------------------------------------------------------------------
To unsubscribe, e-mail: cvs-unsubscribe@incubator.apache.org
For additional commands, e-mail: cvs-help@incubator.apache.org


Mime
View raw message