On Wed, Jan 5, 2011 at 4:45 PM, Emmanuel Lecharny <email@example.com>
<warning, long mail ;)>
Heh yeah a monster mail indeed.
as the 2.0 release is on its (slow) way, it's probably time to check if we are ready for such a release. I see three different things we have to discuss :
1) What : in other words, what is the perimeter of such a release ? What should be put in the box ?
2) How : or is there a roadmap that can help us to fulfill this commitment, and who is takng responsability of each part of this roadmap.
3) When : usually, the answer to a user asking us about the release date is "When it's ready". However, there are probably some timeframe we can discuss.
Let's start with the 'What' part first.
(FTR, The last release we did, Apache Directory Server 1.5.7 was issued on april, 24th, 2010)
Even if we release a 2.0-RC1 in the very near future say a couple weeks from now, I'd recommend releasing a 1.5.8 so the features and changes made since 1.5.7 can be tested by users. There have been some big changes especially on the Kerberos side.
A release would take less than a weeks time (vote and all) and should not hold back the work being done to progress towards 2.0.
Hopefully, we have JIRA which stores the issues, tasks and wishes, and it's the first input we will use to determinate the 'what'. Let's move back to the past a bit. In may, 26th, 2010, a mail about the version scheme was sent and discussed on the directory ML . What was proposed back and agreed then was :
- skip ADS-1.5.8 and target a 2.0.0-RC1 instead
- skip shared-0.9.20 and target for a 1.0.0-RC1 instead
- Keep the RC in order to give us the opportunity to stabilize the code *and* the documentation
This might have been presumed thinking 2.0 was not far away. Now we're 8 months out since the last release. So we might want to re-evaluate this, and you made some points about not releasing shared 1.0.0-RC1 to avoid having to solidify those API's for 2.0 because this might slow 2.0 down.
What was also raised at this time is that once we will jump for a 2.0(server) and 1.0(api), we will be stuck with the exposed interfaces for a very long time, so we'd better be careful. This still stand.
Yeah this is my main worry. Let me though put this discussion out into another thread or two for better clarity and tracking.
We then started to move forward, cleaning some parts that needed it, following the classic #bug report :
- installers were totally rewrote 
- configuration has been completely re-factored, it's now stored in DIT
- all the 1.5.8/1.5.9 issues have been moved to 2.0.0-RC1 in JIRA
- performances of each operations were greatly improved (by a factor 2 to 4) 
- some discussions about the JDBM backend occured, we agreed we should have a MVCC backend sooner or later, but no clear decision was even stated about when we should do that. However, recollecting my memory, it seems that it's out of question for 2.0.
- Multi-Master replication has been added 
- we moved documentation from Confluence to use Dockbook instead 
- PasswordPolicy have been added
- the ACI subsystem has been reviewed 
- some refactoring of shared has been discussed and started 
- the various LDAP messages data structure have been merged (a long expected task)
- some discussion started about the shared/API merge 
- the kerberos codec has been completely rewritten 
- AdministrativeModel implementation refactoring has been started (support of IAP is at stake)
Sounds to me like we should get a 1.5.8 release out to the community fast while continuing with 2.0. There's a lot of things that could use real work tire kicking here. It will not necessarily slow us down and it will get us a bit more tidy for 2.0. We're still going to have to tidy up as usual for 2.0-RC1.
All those parts have been heavily discussed, as discussion about a 2.0 release started back in january 2009 !  This effort has been pursued and in may 2010, we started a bug parade  (reports are available on , , , , ). Then we hit some serious issues in the code that needed more than a few lines of code, bt some serious refactoring (namely, config in DIT, ACI subsystem, Kerberos refactoring, AdministrativeModel). Those huge refactoring stopped the release effort until they are done.
Right now, we are almost ready to start again to run the bug parade, once the Administrative Model is finished (still working on that).
Release early release often. If we get a 1.5.8 out and let peeps kick the tires but tell them, look 2.0 is coming soon so don't get hooked on this WDYT?
Sooo... time to review what's missing, and what's need some love now. This is the 'What' part we all are looking at, and the best point to start it will be to look at opened JIRAes. Otherwise, we still have to discuss some serious re-factoring to be done in Shared.
Back in december, Alex started to review Shared, we discussed about it, and we agreed that the work started on september was not enough. The rationals are clear :
- shared and API *are* the same thing. We want to merge the client-API and shared to provide a LdapAPI (and we even started a new project for that).
- many classes are useless, and need to be removed. Alex started to do that in a shared branch
Yeah I was doing some experiments in my branch not certain one way is better than another but after seeing some nice impact I can start solidifying a direction and express my findings here.
- some modules are to be reworked, we should probably rename some, and Alex is experimenting in the shared branch
Will report on my experiments shortly here.
- we also have to discuss about moving some classes out of shared (DnNode, and may be a few more) or moving in some others (Subentry classes in core-api). This is a part we can have some disagreement, but it's not really a big deal at the end.
Yeah there's no right or wrong. It's about tradeoffs and I'll outline what exactly I'm talking about.
There may be some other elements, but nothing is yet decided, so it's fine to add there what is missing.
Otherwise, we have to remember that we have 3 different 'release' at stake here :
- ADS 2.0
- Studio 2.0
- LdapAPI 1.0 (ex-shared)
LdapAPI is absolutely *critical*. Once it's out, there is no way for us to change it, because it will be heavily sued, as it's meant to be a replacement for JNDI, and many of the outdated LDAP API. So we have to be extraordinary cautious in this area.
Exactomundo!!! This is my massive fear. There was immense work done here and it's awesome and can save the world from JNDI hell when dealing with LDAP. So we need to do it right and adhere to API maintenance conventions.
Unfortunately though the LDAP Client API is not only restricted to the ldap client api module, the API's surface area spans across several shared modules: really almost all of them. So these API's don't have clear boundaries. I'll discuss this in another thread.
However, and that's the good point, we can spend as much time as needed to get LdapAPI 1.0 out, as it won't be a show-stopper for ADS and Studio. It doesn't matter if we continue with a shared-0.9.20 in ADS 2.0 or in Studio 2.0, because ADS will be used way more often as a standalone server, or even if it's embedded, the exposed interface will be the core-API part, which is not part of LdapAPI (except a few classes which will need some careful review).
I wanted to save this for a separate email but just as a hint. The surface area of the server's API includes shared libraries, since the LDAP access and operation model are defined within shared modules. So there's no clear API boundary to the core, it slurps in shared and it becomes one big API. So trying to lock in core-api is futile if you don't lock in shared libraries to follow the same change conventions.
That's if for the 'what', at least from my perception of the 'what'. Please feel free to add any relevant information here.
Let me pump this what response out and will hit the other points in separate responses.
The 'how' is clearly simpler, as it's a matter of defining the schedule. as of today, we have tried to start a 2.0 release at least twice, and we stopped it twice, because some areas were so badly problematic that they needed some heavy refactoring. I hope we will soon be out of this maze.
Currently, if we think about the way to schedule the path toward 2.0, here is how I see it :
- trunk-AP is a branch used to refactor the AdministrativeModel. It's a complex task, which will probably take 2 more weeks. i'm merging back any change done in the server to avoid having some issues when I'll do the merge back to the server
- alex-refactoring is a branch where Alex is doing some experiment and cleanup on shared. It will probably be done fast, and then we can also consider discussing the potential impacts of such refactoring. it's important to stress out that those change are crucial, and they *have* to be done. However, as shared is used all over the server and studio, it will probably need a staging during which we will only get the merge done.
- In the mean time, we agreed that AP refactoring should be done asap, because it depends on shared (thanks to subversion, merging a branch can be a nightmare... We should consider switching to Git at some point after 2.0 :)
Once those 3 steps are done, I do think we are ready for a 2.0-RC1 bug parade. We have :
- DIR : 13 issues 
- DIRSERVER : 136 issues 
- DIRSHARED : 8 issues 
- DIRAPI : 14 issues , 
- DIRKRB : 10 issues 
Not all of them are bugs, hopefully ! We "only" have 54 bugs out of those 181 issues.
So the 'how' part is just about discussing the timing, and organization (what should be done in branch, when should it be merged back, etc). I don't think it will be an obstacle.
The most difficult part ! The ASF is not a place were people can get a clear date, like M$ does. We don't release for a date, we release when "it's ready".
However, we can't also stay in a never ending story, with continuous addition of features. The very nature of our work make it impossible to be satisfied by what we deliver, and we should accept the idea that we won't provide the absolute perfection. We all know that...
That being said, and sorry if it sounds like a lecture, because it's not, we have to get something out soon, and we should now be able to define some milestones.
Here is how I see the planning :
- 2.0.0-RC1 should be ready by end of january, end of february. Enough said if I estimate the effort to something around 4 to 8 weeks.
- 2.0.0-RC2 could take as long a 3 months. We haven't yet mentioned 2 things that are absolutely essentials : documentation and migration from 1.5.7. I think that those 3 months will be a perfect delay to get those two things done, plus of course fixing bugs, and such tasks. This leads us to june.
- 2.0-GA could be ready by september. This is another 3 months, and we can use this time to conduct heavy tests, and improve the Disaster Recovery System, side components (like an HTTP admin based component), shaking the Multi-Master replication, etc.
- 2.0.0-RC1 can be out at the same time or slightly after 2.0.0 is out. i'm not the expert in this area, I would let Stefan and Pierre-Arnaud giving us a better perspective on this
***LDAPApi*** (ex shared/clientAPI)
- 0.9.20 delivered when ADS 2.0.0-RC1 is out
- as many 0.9.2x as needed to get a stabilized version of the API,
- 1.0-RC1 probably by september too. I may be a bit pessimistic here, and be happy to be demonstrated pessimistic, but here I share Alex' concern about the API freeze. Once 1.0 is out, we will have to live with it for *years*
- 1.0-GA maybe by the end of this year.
Pfff... that was a long mail. I have tried to add as much as factual elements on it, so that there is no misconception about this roadmap. Of course, this is not written in marble, it was much more intended as a snapshot of the current development state.
All of these points can be discussed, the 'what', the 'how' and even the 'when'. We don't have a Poject Leader, we are just a community, and it's the only way we can get this server out.
I hope I have phrased what I have in mind in a way that is not controversial, because it was not my intention, but if you think I have mis-represented, or forgot anything, please feel free to mention it.
Last, not least, I wish all of you a Happy New year, and I hope we can get ADS 2.0 out and in good shape, because we have worked like dogs for it, and we deserve to chew a bone now :)