directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng, Kai" <kai.zh...@intel.com>
Subject RE: About Haox project
Date Sat, 27 Dec 2014 01:36:25 GMT
Hi Emmanuel,

Thanks for the history and background. Very interesting and helpful for me to understand ApacheDS
project and the community. I would like to contribute and develop in either codes or community,
particularly from the point of view interested and inspired by Hadoop ecosystem.

>> Thanks Kai. At this point, I think we could launch a vote to accept the contribution.
Thanks for holding this. Meanwhile, I welcome any review and feedback for the project codes,
if any issue I should resolve first I will. Currently I'm working on the cleanup and clearance
to prepare for this.

>> I must admit that the ASN/1 codec you are coming from is not only way simpler than
what we use, but it's also faster !
Thanks for your help tuning it even faster! With the help of the community experts, I will
be much confident about making the codec edible. Security field (Kerberos, PKI, LDAP and etc.)
are very ASN1 heavy, and a simpler one would help for new contributors. 

>> I wanted to build a ASN/1 compiler (see http://svn.apache.org/repos/asf/labs/dungeon/)
>> Implementing a new Codec requires a huge effort, even for me.
It's very interesting about the compiler. Actually it was in my mind to support such compiler
for haox-asn1 library, enabling to compile an ASN1 definition module (like krb5.asn1) and
automatically generate all the necessary Java types (like Kerberos codec classes), and we
don't even bother to code them manually.  Yes it's hard, huge effort is involved, not easy
for me either.

>> I would rather first check what Bouncy Castle has to propose in this area. I'd be
really careful here, implementing some crypto amgorithm is seriously complex, and require
a lot of care.
Sorry I'm confusing and have to be clear here. When I mentioned "implement all the encryption
types and checksum types that MIT Kerberos supports, like DES, DES3, AES, RC4, CAMELLIA",
I mean we need to implement the Kerberos encryption types and checksum types which are often
mentioned as Kerberos crypto. Such crypto isn't the one in JCE neither any in BC, where only
covers concrete encryption algorithms. Kerberos crypto is based on concrete encryption algorithms
and implementations, but would never bother to define and implement any encryption algorithms
itself.

Yes we could rely on the Kerberos crypto implemented in JRE, as existing ApacheDS Kerberos
implementation does. The problems are: 1) JRE doesn't expose the functionalities fully and
is very limited; 2) It's not complete and we can't guarantee full interoperable with MIT Kerberos
and MS-AD; 3) Relying on JRE means we'll never be able to go beyond that, if the corresponding
support isn't available in JRE, then we'll stop there. You might agree that many modern Kerberos
features are just not available yet.

In my view we have to tweak it for ourselves, that's why I spent so much time on it. The framework
originates the one from MIT Kerberos and is very flexible, say once a new encryption type
is up then it's easy to add it accordingly so we are updated! The support is complete and
all the ones provided in MIT Kerberos are done. I'm confident with it, basically I would think
it's the core part that's essential for a Kerberos implementation. Without it, maybe it's
fine for us to just replace the Kerberos codec part based on existing Kerberos implementation.
Yes we must be careful with the codes. I would be very happy for your review of it and make
it strong enough. 

>> The base idea is still to be able to deliver a global package where the Kerberos
Server is accessing ApacheDS LDAP Server without going through the network
Yes, I agree. With such an abstraction and API defined, various identity back ends, either
internal embedded or external crossing network can be supported. One thing to note is concrete
identity backend implementation would be separate from the Kerberos Server in its own pom-project
so never enforces any dependencies. In runtime, such a backend will be dynamically searched
or loaded from configuration then employed. 

>> has also to be considered is the configuration part, and integration with Studio.
Yes agree. Regarding Kerberos configuration either in client side (krb5.conf) or server side
(kdc.conf), I guess we'd better make it compatible with MIT Kerberos so that administrators
won't have to learn about it. Also in our documentation for simple we would just say "please
reference MIT Krb5 doc" as we're going the same way. A contributor is work on it and will
commit the codes once the codebase is imported.

>> we want to have something that move forward, not just to replace what we curently
have
Yes totally agree. That's why we're here. Develop the project, develop the community, making
the implementation updated and strong enough as a Kerberos distribution.

>> This is a critical part of the success !
Yes, with such possible inspiring by "Apache Kerberos", we would attract and welcome more
contributors in future. I guess when we come to the 2nd phase, we would be able to update
to the Hadoop space and it's possible we'll get more contribution and feedback from there.
Currently Hadoop becomes one of the biggest platforms that deploy Kerberos, and various challenges
are met there. Hopefully we can help them further. Note ApacheDS Kerberos was already integrated
for unit tests.

>> That sounds like a perfect proposal.
Thank you!

Regards,
Kai

-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Friday, December 26, 2014 4:39 PM
To: dev@directory.apache.org
Subject: Re: About Haox project

Le 23/12/14 18:57, Zheng, Kai a écrit :
> Hi all,

Hi Kai,

sorry for the delayed mail, I had to digest my Xmas heavy dinner and lunch ;-)
>
> I'm Kai and a guy from Intel Big Data team. I have been working on Apache Hadoop for
some while, and used to be an ApacheDS contributor and committed the feature of CredentialCahce
and a simple Kinit tool, guided by Emmanuel and Kiran. After that when I was going to implement
more Kerberos features I realized there needs important improvement space in the existing
fundamental Kerberos codes.
You bet...

>  As a prototype I initiated Haox project  and have implemented an ASN1 encoding/decoding
framework, complete encryption and checksum crypto, Kerberos codec, Keytab and CredentialCache
facilities, KrbClient API and simple KDC server. It's still in its initial phase and far from
ideal. However as strongly suggested by Emmanuel and Kiran, I should collaborate with ApacheDS
community and continue to make the effort with you together. It's great and now I'm going
to contribute the project to ApacheDS. I have discussed with Emmanuel and Kiran for quite
some time about this. Based on our discussion, we have the following proposal. It's for your
review and consider. Please kindly provide your feedback and advise. Thanks in advance.

Thanks Kai. At this point, I think we could launch a vote to accept the contribution.
>
> 1. First all our vision for the long term, we would think it makes great sense to establish
a first class Kerberos implementation in Java for the Apache world targeting today's environments
 in cloud, Hadoop and mobile. Therefore we could collaborate together in ApacheDS community
to build a "Apache Kerberos" project. That's why we're here.

This was one of the reason we developped the Kerberos Server (well, sort of). Let's say the
history of our current implementation is a bit complex. When Alex started to work on the LDAP
server, he designed a server that would be embeddable, and which could process more than just
the LDAP protocol. Kerberos was a natural choce - as was DNS, DHCP, etc... Basically, the
idea was to have what we call a DirectoryService which handles the data, and a front-end to
process the incoming requests (the Protocol layer). The current kerberos implementation is
based on this DirectoryService and is tightly coupled with the protocol layer.
This effort, sadly, staled for years before we rewrote most of the ASN.1 layer - which was
far from being perfect - and added a Kerberos client plus fixed many issues. But, still, we
have so many things on pur plate that it's hard for us to move forward fast enough.

What would your proposal change ? Well, After havinbg looked at it, I do think it's what we
should expect from a Kerberos server : it should not necessarily being tied to a LDAP server
as it is in our current code, and I must admit that the ASN/1 codec you are coming from is
not only way simpler than what we use, but it's also faster !

A digression here : the very first thing I worked on back in 2005 when I jumped into the project
was a rewrite of the existing ASN/1 codec, which was slow like death (well, it was not exactly
the very first thing, as I get accepted for my small contrinution on the LDIF parser, but
anyway).
Due to the constraints (ie, the codec had to be statefull, capable of dealing with TCP fragmentation)
I came with a slution based on a State Machine. It was also driven by the fact I wanted to
build a ASN/1 compiler (see http://svn.apache.org/repos/asf/labs/dungeon/), a staled effort.
The codec is efficient, up to a point, but extremelly complex to use. Implementing a new Codec
requires a huge effort, even for me.


>
> 2. And as the first step to start with, ApacheDS would accept Haox as 
> a sub-project, renamed as "Apache Kerberos". Then we consolidate 
> existing Kerberos implementation codes in ApacheDS into the new 
> project;

+1
>
> 3. Afterwards we will make further effort to push "Apache Kerberos" out as a top level
project, probably in a year or so.

That's a decision which will be easy, as soon as we have enough contributor to the project.
>
> We have discussed other approaches, like using the good stuffs in Haox to replace the
corresponding parts in ApacheDS Kerberos, as one may also suggest. So let's figure out what
might be good parts implemented in Haox.
>
> 1.     Haox-ASN1. It's a schema/model driven ASN1 parser and codec. Based on the framework
new Kerberos types can be easily defined following fixed patterns, without having to take
care of ASN1 encoding/decoding procedures. We see this as very important because Kerberos
itself and related extensions are very ASN1 heavy. Without the basic friendly library support
it's hard for new developers to contribute on new Kerberos evolving features. We tested the
codec performance and it outperforms a little. Emmanuel has even contributed a few optimization
improvements.
+1
>
>
>
> 2.     Kerberos crypto. The crypto framework (based on RFC3961) and library implement
all the encryption types and checksum types that MIT Kerberos supports, like DES, DES3, AES,
RC4, CAMELLIA, with extensive interoperable tests covered. It relies on JCE for encryption
but not any Kerberos facilities provided in JRE. Potentially we could port it to other non-Oracle
Java environment like Android system, assuming we can implement corresponding encryption and
hash providers which can be pluggable.

I would rather first check what Bouncy Castle has to propose in this area. I'd be really careful
here, implementing some crypto amgorithm is seriously complex, and require a lot of care.

>
>
> 3.     Kerberos codec. Based on Haox-Asn1, the codec has implemented all the necessary
Kerberos types defined in Krb5(RFC4210), Preauth/FAST(RFC6113), and PKINIT(4556). The core
part has some test cases covered already.

We should just migrate the tests we have in our implementation to test the new one. Easy,
even if might be boring.
>
>
> 4.     KDC server. The implementation abstracts an KrbIdentityService API and decouples
any specific identity backend from the Kerberos server. We can provide an LDAP based backend,
or allow vendor to implement their own backed by SQL database, external LDAP server or whatever.
For simple unit test usage, a memory HashMap is all the needs. This allows our embedded KDC
really very lightweight avoiding many unnecessary dependencies.

A decoupling must be done in ApacheDS. That should not be a big deal.
The base idea is still to be able to deliver a global package where the Kerberos Server is
accessing ApacheDS LDAP Server without going through the network, ie by using the CoreSession
API, or may be defining a higher API which uses CoreSession under the hood : this higher level
API would connect to a LDAP server (whatever it is) through this API, and we would either
go through the network, or uses CoreSession.
>
>
> 5.     Haox-event. It implements a UDP/TCP mixed network event support based on Java
NIO and reactor pattern according to Kerberos's needs, where UDP and TCP are both required,
and even in IAKERB, a KDC proxy would serve both TCP server role and UDP/TCP client role.
In Haox-event, all these roles can be done in an unified event receiving, processing and replying.
Test cases are given but it definitely need to tune and improve. Again, it doesn't rely on
any other libraries, thus based on it KrbClient could easily be supported in other Java environment.

Here, I would check if MINA could be a good substitue, or not. It's still an open question.
>
> Keeping our long term goal in mind, we thought above functionalities, properties and
considerations may serve better to establish the foundation for "Apache Kerberos" project,
and it would be much easier to consolidate the existing Kerberos implementation into it.

What has also to be considered is the configuration part, and integration with Studio.

>
> So as a first step for the collaboration, how about:
>
> 1.     Prepare, which involves:
>
> 1) Necessary cleanup;
>
> 2) All Kerberos related codes will start package names with "org.apache.kerberos", all
the non-Kerberos related codes will be moved into contrib projects or libraries, like Haox-ASN1,
Haox-event, Haox-config, which are actually not Kerberos logic specific.

We should think about that, sure.
>
> 2. Move Haox to be an ApacheDS sub-project:
> 1) Renaming Haox as "Apache Kerberos";
> 2) A separate GIT repo;
> 3) A separate coding style, subject to be discussed and determined.
+1
>
> Then as the 2nd step, consolidate the necessary parts in existing Kerberos implementation.
Below is the initial list and please complement:
>
> 1.     Apache Directory Server support. We need to create an LdapIdentityBackend using
Apache Directory Server and use it by default, embedded in the KDC server.
>
> 2.     Installation packaging and service wrapper. Haox hasn't implemented such but ApacheDS
does.
>
> 3.     Tests. All the Kerberos related tests should be evaluated and ported to the new
codebase if not obsolete.

+1
>
> In addition to above, we also need to consider the following aspects, to make it good
enough for potential users.
>
> 1.     Adding the missing Javadoc.
>
> 2.     Adding more tests.
>
> 3.     Adding the documentation.
> 4. Building the web site

+1. Part of the process anyway.
>
> With such done, then we might obsolete the AacheDS Kerberos implementation and might
move all the related codes into an history folder for existing users. We would encourage users
to use the new project.
> For the 3rd step, we would continue to implement some essential and advanced features,
which may include:
>
> 1.     Cross realm support, which means Apache Kerberos can federate with MIT Kerberos
or MS AD;
>
> 2.     Full functional client tools, like kinit, klist and kadmin, or other alternatives;
>
> 3.     Preauth/FAST framework, which allows to support other authentication mechanisms
based on the secure channel protected by an Armor TGT/key;
>
> 4.     PKINIT, authenticating user using x509 certificate other than plain password;
>
> 5.     Token preauth, authenticating user using a JWT token, to support OAuth.
>
> 6.     ...

All in all, that are the important points : we want to have something that move forward, not
just to replace what we curently have, which is somehow not moving forward at a decent pace
(and here, I'm ebing quite kind to the current satte...)
>
> Note tasks in step2 and 3 are not necessarily to be done in the order. We're likely to
work on them in parallel.
>
> We would encourage more developers to contribute to the effort and develop the community.
Without that, we may not be able to make it to have a high quality of Kerberos implementation,
with all the important Kerberos feature gaps being filled.

This is a critical part of the success !
>
> Then finally as the last step, in some appropriate time we would push Apache Kerberos
out to be an Apache top level project.
>
> All the points are just suggestions and subject to your discussion. Your feedback is
most welcome. As widely discussed in a week or so, this proposal is to be finalized and then
re-posted. Then let's collaborate and make it together.

That sounds like a perfect proposal.

Thanks Kai !


Mime
View raw message