hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avik Dey <aviks.em...@gmail.com>
Subject Re: ANNOUNCEMENT: Project Rhino: Enhanced Data Protection for the Apache Hadoop Ecosystem
Date Tue, 26 Feb 2013 01:18:40 GMT
[thanks appreciate your doing that, the announcement itself was
cross-posted as outreach]

Thanks Cos.

As I see the work currently, I believe most, if not all of these, will be
work against JIRAs in individual projects similar to the JIRAs posted here
https://github.com/intel-hadoop/project-rhino. If we get to a point where
some of the future work needs a home outside of the individual projects,
happy to incubate that work in Apache.


On Mon, Feb 25, 2013 at 4:18 PM, Konstantin Boudnik <cos@apache.org> wrote:

> [yanking away most of the cross-posts...]
> An interesting cross component project Avik. Any plans to incubate it in
> Apache?
> Cos
> On Mon, Feb 25, 2013 at 11:46PM, Dey, Avik wrote:
> > Project Rhino
> >
> > As the Apache Hadoop ecosystem extends into new markets and sees new use
> > cases with security and compliance challenges, the benefits of processing
> > sensitive and legally protected data with Hadoop must be coupled with
> > protection for private information that limits performance impact.
> Project
> > Rhino<https://github.com/intel-hadoop/project-rhino/> is our open source
> > effort to enhance the existing data protection capabilities of the Hadoop
> > ecosystem to address these challenges, and contribute the code back to
> > Apache.
> >
> > The core of the Apache Hadoop ecosystem as it is commonly understood is:
> >
> > - Core: A set of shared libraries
> > - HDFS: The Hadoop filesystem
> > - MapReduce: Parallel computation framework
> > - ZooKeeper: Configuration management and coordination
> > - HBase: Column-oriented database on HDFS
> > - Hive: Data warehouse on HDFS with SQL-like access
> > - Pig: Higher-level programming language for Hadoop computations
> > - Oozie: Orchestration and workflow management
> > - Mahout: A library of machine learning and data mining algorithms
> > - Flume: Collection and import of log and event data
> > - Sqoop: Imports data from relational databases
> >
> > These components are all separate projects and therefore cross cutting
> concerns like authN, authZ, a consistent security policy framework,
> consistent authorization model and audit coverage are loosely coordinated.
> Some security features expected by our customers, such as encryption, are
> simply missing. Our aim is to take a full stack view and work with the
> individual projects toward consistent concepts and capabilities, filling
> gaps as we go.
> >
> > Our initial goals are:
> >
> > 1) Framework support for encryption and key management
> >
> > There is currently no framework support for encryption or key
> management. We will add this support into Hadoop Core and integrate it
> across the ecosystem.
> >
> > 2) A common authorization framework for the Hadoop ecosystem
> >
> > Each component currently has its own authorization engine. We will
> abstract the common functions into a reusable authorization framework with
> a consistent interface. Where appropriate we will either modify an existing
> engine to work within this framework, or we will plug in a common default
> engine. Therefore we also must normalize how security policy is expressed
> and applied by each component. Core, HDFS, ZooKeeper, and HBase currently
> support simple access control lists (ACLs) composed of users and groups. We
> see this as a good starting point. Where necessary we will modify
> components so they each offer equivalent functionality, and build support
> into others.
> >
> > 3) Token based authentication and single sign on
> >
> > Core, HDFS, ZooKeeper, and HBase currently support Kerberos
> authentication at the RPC layer, via SASL. However this does not provide
> valuable attributes such as group membership, classification level,
> organizational identity, or support for user defined attributes. Hadoop
> components must interrogate external resources for discovering these
> attributes and at scale this is problematic. There is also no consistent
> delegation model. HDFS has a simple delegation capability, and only Oozie
> can take limited advantage of it. We will implement a common token based
> authentication framework to decouple internal user and service
> authentication from external mechanisms used to support it (like Kerberos).
> >
> > 4) Extend HBase support for ACLs to the cell level
> >
> > Currently HBase supports setting access controls at the table or column
> family level. However, many use cases would benefit from the additional
> capability to do this on a per cell basis. In fact for many users dealing
> with sensitive information the ability to do this is crucial.
> >
> > 5) Improve audit logging
> >
> > Audit messages from various Hadoop components do not use a unified or
> even consistently formatted format. This makes analysis of logs for
> verifying compliance or taking corrective action difficult. We will build a
> common audit logging facility as part of the common authorization framework
> work. We will also build a set of common audit log processing tools for
> transforming them to different industry standard formats, for supporting
> compliance verification, and for triggering responses to policy violations.
> >
> > Current JIRAs:
> >
> > As part of this ongoing effort we are contributing our work to-date
> against the JIRAs listed below. As you may appreciate, the goals for
> Project Rhino covers a number of different Apache projects, the scope of
> work is significant and likely to only increase as we get additional
> community input. We also appreciate that there may be others in the Apache
> community that may be working on some of this or are interested in
> contributing to it. If so, we look forward to partnering with you in Apache
> to accelerate this effort so the Apache community can see the benefits from
> our collective efforts sooner. You can also find a more detailed version of
> this announcement at Project Rhino<
> https://github.com/intel-hadoop/project-rhino/>.
> >
> > Please feel free to reach out to us by commenting on the JIRAs below:
> >
> > HBASE-6222: Add per-KeyValue Security<
> https://issues.apache.org/jira/browse/hbase-6222>
> >
> > HADOOP-9331: Hadoop crypto codec framework and crypto codec
> implementations<https://issues.apache.org/jira/browse/hadoop-9331> and
> related sub-tasks
> >
> > MAPREDUCE-5025: Key Distribution and Management for supporting crypto
> codec in Map Reduce<https://issues.apache.org/jira/browse/mapreduce-5025>
> and related JIRAs
> >
> > HBASE-7544: Transparent table/CF encryption<
> https://issues.apache.org/jira/browse/hbase-7544>
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message