Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALs3O6gKpCzg12u-6oDO28P8CtNetb+yHy+aJPh1ncBPHXE8Sw@mail.gmail.com>
References: 
 <CALs3O6iVu6mE_w7t70Ei2Vxxhs=v6ZoMAWo2dMiU9X5ii=GcrA@mail.gmail.com>
	<CAPMpPc7vTE=GhbxTERonBejfmudYohgj-Cyq-X8STmtfWNnXiA@mail.gmail.com>
	<CAL5zq9ZpeNDF8Wj6zSq+f04rCEU0pqR1MTfnCz0Gu31ib-Fa6Q@mail.gmail.com>
	<CAPMpPc6oBMokO3wrxnxEYF_GiHwcdm7G8gHbag7sTJanANG7rg@mail.gmail.com>
	<CALs3O6gKpCzg12u-6oDO28P8CtNetb+yHy+aJPh1ncBPHXE8Sw@mail.gmail.com>
Date: Mon, 2 Nov 2015 13:37:24 -0500
Message-ID: 
 <CAGUtCHpa0w+XSdpy0JEwOb_qLyCnztAY0c-PRGob6t19DkJKCA@mail.gmail.com>
Subject: Re: [DISCUSS] What to do about encryption at rest?
From: Keith Turner <keith@deenlo.com>
To: Accumulo Dev List <dev@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=001a113ce054808d9a052393119c

--001a113ce054808d9a052393119c
Content-Type: text/plain; charset=UTF-8

On Mon, Nov 2, 2015 at 12:27 PM, William Slacum <wslacum@gmail.com> wrote:

> Is "the code being 'at rest'" you making a funny about active development?
> Making sure I haven't lost my ability to get jokes :)
>
> I see two reasons why the code would be inactive: the feature is good
> enough as is or it's not interesting enough to attract attention.
> Considering it's not public API, there are no discussions to bring into the
> public API, and there's no effort to document how to use it, my intuition
> tells me that there isn't enough interest in it from a project perspective.
>
> From a user perspective, I've been getting asked about it when I work with
> Accumulo users. My recommendation, exclusively, is to use HDFS encryption
> because I can go to Hadoop's website and find documentation on it. When I
> go to find documentation on Accumulo's offerings, any usability information
> comes from vendor SlideShares. Most mentions of the feature on official
> Apache Accumulo channels echo Christopher's sentiments on the feature being
> experimental and not being officially recommended for use.
>
> I wouldn't want to rip out the feature first and then figure things out
> later. Sean already alluded to it, but a roadmap should contain something
> (tool or documentation) to help users migrate if we go down that route.
>
> What I'm trying to figure out is, when the question of "How do I do
> encryption at rest in Accumulo?" comes up, what is our community's answer?
>
> If we went down the route of using HDFS encryption zones, can we offer the
> same features? At the very least, we'd be offering the same database-level
>

Where does the decryption happen with DFS, is it in the DFS client?  If so,
using HDFS level encryption seems to offer the same functionality???

Has anyone written a tool that takes an
Accumulo-encrypted-HDFS-unencrypted-RFile and rewrites it is as an
Accumulo-unencrypted-HDFS-encrypted-RFile?  Wondering if there are any
unexpected gotchas w/ this.


> encryption scheme. I don't know the details of "more advanced key stores",
> but it seems like we could potentially take any custom implementation and
> map it to a KeyProvider [1]. I could also envision table level encryption
> being implementable via zones, but probably not down to the column family
> level.
>
> [1]
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/crypto/key/KeyProvider.html
>
>
> On Sun, Nov 1, 2015 at 10:19 AM, Adam Fuchs <afuchs@apache.org> wrote:
>
> > Responses inline.
> >
> > Adam
> >
> > On Nov 1, 2015 9:58 AM, "Christopher" <ctubbsii@apache.org> wrote:
> > >
> > > 1. I'm not sure I'd call an incomplete solution 'great'. What it does
> is
> > > provide partial encryption-at-rest protection (unless you're running
> > > without walogs, and have good integration with some external secure key
> > > management faculty, and then it's probably fine).
> >
> > The only thing that doesn't get encrypted is a temporary WAL recovery
> file.
> > That is a project we should take on, but it does not imply that the
> > existing features are not valuable. With HDFS encryption options this
> would
> > now be a much easier project to take on. Also, the users I know that use
> > encryption at rest do so with a more secure key store than the default.
> >
> > >
> > > 2. I'm concerned that anybody using Accumulo's E-A-R don't necessarily
> > > realize its current shortcomings, or its lack of upstream maintenance
> > > support (which it has not been receiving). It may be the case that
> these
> > > users have support from an intermediary, and do understand the
> > > shortcomings... I don't know, but it's a concern.
> >
> > Anybody that creates a secure system has to analyze the security of the
> > system as a whole. Accumulo's encryption at rest is one part of the
> > solution. Taking away the tool without providing an alternative does
> > nothing to improve the security of systems built on Accumulo.
> >
> > >
> > > 3. Correction: it has been an explicitly experimental feature and an
> > > incomplete one, which hasn't really been touched in two years, and has
> > been
> > > explicitly excluded by the community for being public API because of
> its
> > > incompleteness. Age doesn't determine public API status. The community
> > does.
> >
> > People are using it, so we have to consider the implications of whatever
> > changes we make and weigh against the benefits. I believe the last bug
> fix
> > was done this year, so I would argue it is being maintained. Changes to
> our
> > encryption at rest implementation will have consequences for those users.
> > There had better be a clear benefit if we break their systems.
> >
> > >
> > > 4. Has Accumulo's been evaluated for security and performance? By whom?
> > Is
> > > it published?
> >
> > Yes, there have been several talks at meetups and conferences that
> discuss
> > the security and performance of the current solution.
> >
> > >
> > > On Sun, Nov 1, 2015, 08:55 Adam Fuchs <afuchs@apache.org> wrote:
> > >
> > > > There's another way to look at the state of Accumulo's encryption at
> > rest:
> > > > 1. Encryption at rest works great for what it does, and the code
> being
> > "at
> > > > rest" isn't necessarily a problem
> > > > 2. Several organizations are using Accumulo's encryption at rest
> > > > effectively in operations
> > > > 3. Encryption at rest has been a supported configuration option for
> > over
> > > > two years with established plugin interfaces, and therefore it should
> > be
> > > > considered part of the public API
> > > > 4. Upstream alternatives (to my knowledge) have not been analyzed for
> > > > performance or security
> > > >
> > > > The given option #2 would at least require an analysis of
> alternatives,
> > and
> > > > we would have to decide what to do about backwards compatibility for
> > users
> > > > using custom key stores and encryption strategies that may or may not
> > be
> > > > supported by upstream alternatives.
> > > >
> > > > As far as option #1 goes, I can get behind encouraging people to take
> > up
> > > > projects to improve Accumulo's encryption. I think we're already
> going
> > down
> > > > this path, but without having identified resources to do the
> > improvements.
> > > > Any volunteers?
> > > >
> > > > Adam
> > > >
> > > >
> > > > On Fri, Oct 30, 2015 at 4:22 PM, William Slacum <wslacum@gmail.com>
> > wrote:
> > > >
> > > > > So I've been looking into options for providing encryption at rest,
> > and
> > > > it
> > > > > seems like what Accumulo has is abandonware from a project
> > perspective.
> > > > > There is no official documentation on how to perform encryption at
> > rest,
> > > > > and the best information from its status comes from year (or
> greater)
> > old
> > > > > ticket comments about how the feature is still experimental.
> Recently
> > > > there
> > > > > was a talk that described using HDFS encryption zones as an
> > alternative.
> > > > >
> > > > > From my perspective, this is what I see as the current situation:
> > > > >
> > > > > 1- Encryption at rest in Accumulo isn't actively being worked on
> > > > > 2- Encryption at rest in Accumulo isn't part of the public API or
> > > > marketed
> > > > > capabilities
> > > > > 3- Documentation for what does exist is scattered throughout Jira
> > > > comments
> > > > > or presentations
> > > > > 4- A viable alternative exists that appears to have feature parity
> in
> > > > HDFS
> > > > > encryption
> > > > > 5- HBase has finer grained encryption capabilities that extend
> beyond
> > > > what
> > > > > HDFS provides
> > > > >
> > > > > Moving forward, what's the consensus for supporting this feature?
> > > > > Personally, I see two options:
> > > > >
> > > > > 1- Start going down a path to bring the feature into the forefront
> > and
> > > > > start providing feature parity with HBase
> > > > >
> > > > > or
> > > > >
> > > > > 2- Remove the feature and place emphasis on upstream encryption
> > offerings
> > > > >
> > > > > Any input is welcomed & appreciated!
> > > > >
> > > >
> >
>

--001a113ce054808d9a052393119c--