Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of lmccay@hortonworks.com
 designates 209.85.128.171 as permitted sender)
From: Larry McCay <lmccay@hortonworks.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_72636A44-E390-4E1E-9927-AC33D0189A0E"
Message-Id: <CE4BA42B-D347-4F09-B0FE-2CC84C9F516E@hortonworks.com>
Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1485\))
Subject: Re: Hadoop Summit: Security Design Lounge Session
Date: Tue, 9 Jul 2013 12:39:12 -0400
References: <5F781862-2DB5-4937-8F5C-E108E1144EA2@hortonworks.com>
To: common-dev@hadoop.apache.org
In-Reply-To: <5F781862-2DB5-4937-8F5C-E108E1144EA2@hortonworks.com>

--Apple-Mail=_72636A44-E390-4E1E-9927-AC33D0189A0E
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

Adding additional takeaways that were articulated by Alejandro and =
expanded by me in another thread - so that we have it all in one =
place=85thanks again, Alejandro!

++++++++++++++++++++++++++++++++++++++++++++++++

Hi Alejandro -

I missed your #4 in my summary and takeaways of the session in another =
thread on this list.

I believe that the points of discussion were along the lines of:

* put common security libraries into common much the same way as =
hadoop-auth is today making each available as separate maven modules to =
be used across the ecosystem
* the was a concern raised that we need to be cognizant of not using =
common as a "dumping grounds"
	- I believe this to mean that we need to ensure that the =
libraries that are added there are truly cross cutting and can be used =
by the other projects across Hadoop
	- I think that security related things will largely be of that =
nature but we need to keep it in mind

I'm not sure whether #3 is represented in the other summary or not=85

There was certainly discussions around the emerging work from Daryn =
related to pluggable authentication mechanisms within that layer and we =
will immediately have the options of kerberos, simple and plain. There =
was also talk of how this can be leveraged to introduce a Hadoop token =
mechanism as well.=20

At the same time, there was talk of the possibility of simply making =
kerberos easy and a non-issue for intra-cluster use. Certainly we need =
both of these approaches.
I believe someone used ApacheDS' KDC support as an example - if we could =
standup an ApacheDS based KDC and configure it and related keytabs =
easily than the end-to-end story is more palatable to a broader user =
base. That story being the choice of authentication mechanisms for user =
authentication and easy provisioning and management of kerberos for =
intra-cluster service authentication.

If you agree with this extended summary then I can update the other =
thread with that recollection.
Thanks for providing it!

--larry

On Jul 4, 2013, at 4:09 PM, Alejandro Abdelnur <tucu@cloudera.com> =
wrote:

> Leaving JIRAs and design docs aside, my recollection from the f2f =
lounge
> discussion could be summarized as:
>=20
> ------
> 1* Decouple users-services authentication from (intra) =
services-services
> authentication.
>=20
> The main motivation for this is to get pluggable authentication and
> integrated SSO experience for users.
>=20
> (we never discussed if this is needed for external-apps talking with =
Hadoop)
>=20
> 2* We should leave the Hadoop delegation tokens alone
>=20
> No need to make this pluggable as this is an internal authentication
> mechanism after the 'real' authentication happened.
>=20
> (this is independent from factoring out all classes we currently have =
into
> a common implementation for Hadoop and other projects to use)
>=20
> 3* Being able to replace kerberos with something else for (intra)
> services-services authentication.
>=20
> It was suggested that to support deployments where stock Kerberos may =
not
> be an option (i.e. cloud) we should make sure that =
UserGroupInformation and
> RPC security logic work with a pluggable GSS implementation.
>=20
> 4* Create a common security component ie 'hadoop-security' to be 'the'
> security lib for all projects to use.
>=20
> Create a component/project that would provide the common security =
pieces
> for all projects to use.
>=20
> ------
>=20
> If we agree with this, after any necessary corrections, I think we =
could
> distill clear goals from it and start from there.
>=20
> Thanks.
>=20
> Tucu & Alejandro


On Jul 1, 2013, at 5:40 PM, Larry McCay <lmccay@hortonworks.com> wrote:

> All -
>=20
> Last week at Hadoop Summit there was a room dedicated as the summit =
Design Lounge.
> This was a place where like folks could get together and talk about =
design issues with other contributors with a simple flip board and some =
beanbag chairs.
> We used this as an opportunity to bootstrap some discussions within =
common-dev for security related topics. I'd like to summarize the =
security session and takeaways here for everyone.
>=20
> This summary and set of takeaways are largely from memory.=20
> Please - anyone that attended - feel free to correct anything that is =
inaccurate or omitted.
>=20
> Pretty well attended - companies represented:
> * Yahoo!
> * Microsoft
> * Hortonworks
> * Cloudera
> * Intel
> * eBay
> * Voltage Security
> * Flying Penguins
> * EMC
> * others...
>=20
> Most folks were pretty engaged throughout the session.
> We set expectations as a meet and greet/project kickoff - project =
being the emerging security development community.
>=20
> In order to keep the scope of conversations manageable we tried to =
keep focused on authentication and the ideas around SSO and tokens.
>=20
> We discussed kerberos as:
> 1. major pain point and barrier to entry for some
> 2. seemingly perfect for others
> 	a. obviously requiring backward compatibility
>=20
> It seemed to be consensus that:
> 1. user authentication should be easily integrated with alternative =
enterprise identity solutions
> 2. that service identity issues should not require thousands of =
service identities added to enterprise user repositories
> 3. that customers should not be forced to install/deploy and manage a =
KDC for services - this implies a couple options:
> 	a. alternatives to kerberos for service identities
> 	b. hadoop KDC implementation - ie. ApacheDS?
>=20
> There was active discussion around:
> 1. Hadoop SSO server
> 	a. acknowledgement of Hadoop SSO tokens as something that can be =
standardized for representing both the identity and authentication event =
data as well and access tokens representing a verifiable means for the =
authenticated identity to access resources or services
> 	b. a general understanding of Hadoop SSO as being an analogue =
and alternative for the kerberos KDC and the related tokens being =
analogous to TGTs and service tickets
> 	c. an agreement that there are interesting attributes about the =
authentication event that may be useful in cross cluster trust for SSO - =
such as a rating of authentication strength and number of factors, etc
> 	d. that existing Hadoop tokens - ie. delegation, job, block =
access - will all continue to work and that we are initially looking at =
alternatives to the KDC, TGTs and service tickets
> 2. authentication mechanism discovery by clients - Daryn Sharp has =
done a bunch of work around this and our SSO solution may want to =
consider a similar mechanism for discovering trusted IDPs and service =
endpoints
> 3. backward compatibility - kerberos shops need to just continue to =
work
> 4. some insight into where/how folks believe that token based =
authentication can be accomplished within existing contracts - =
SASL/GSSAPI, REST, web ui
> 5. what the establishment of a cross cutting concern community around =
security and what that means in terms of the Apache way - email lists, =
wiki, Jiras across projects, etc
> 6. dependencies, rolling updates, patching and how it related to =
hadoop projects versus packaging
> 7. collaboration road ahead
>=20
> A number of breakout discussions were had outside of the designated =
design lounge session as well.
>=20
> Takeaways for the immediate road ahead:
> 1. common-dev may be sufficient to discuss security related topics
> 	a. many developers are already subscribed to it
> 	b. there is not that much traffic there anyway
> 	c. we can discuss a more security focused list if we like
> 2. we will discuss the establishment of a wiki space for a holistic =
view of security model, patterns, approaches, etc
> 3. we will begin discussion on common-dev in near-term for the =
following:
> 	a. discuss and agree on the high level moving parts required for =
our goals for authentication: SSO service, tokens, token validation =
handlers, credential management tools, etc
> 	b. discuss and agree on the natural seams across these moving =
parts and agree on collaboration by tackling various pieces in a divide =
and conquer approach
> 	c. more than likely - the first piece that will need some =
immediate discussion will be the shape and form of the tokens
> 	d. we will follow up or supplement discussions with POC code =
patches and/or specs attached to jiras
>=20
> Overall, design lounge was rather effective for what we wanted to do - =
which was to bootstrap discussions and collaboration within the =
community at large. As always, no specific decisions have been made =
during this session and we can discuss any or all of this within =
common-dev and on related jiras.
>=20
> Jiras related to the security development group and these discussions:
>=20
> Centralized SSO/Token Server =
https://issues.apache.org/jira/browse/HADOOP-9533
> Token based authentication and SSO =
https://issues.apache.org/jira/browse/HADOOP-9392
> Document/analyze current Hadoop security model =
https://issues.apache.org/jira/browse/HADOOP-9621
> Improve Hadoop security - Use cases =
https://issues.apache.org/jira/browse/HADOOP-9671
>=20
> thanks,
>=20
> --larry
>=20


--Apple-Mail=_72636A44-E390-4E1E-9927-AC33D0189A0E--