From dev-return-21590-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Tue Sep 25 18:46:22 2007 Return-Path: Delivered-To: apmail-directory-dev-archive@www.apache.org Received: (qmail 34947 invoked from network); 25 Sep 2007 18:46:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 25 Sep 2007 18:46:21 -0000 Received: (qmail 35437 invoked by uid 500); 25 Sep 2007 18:46:11 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 35399 invoked by uid 500); 25 Sep 2007 18:46:11 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 35387 invoked by uid 99); 25 Sep 2007 18:46:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Sep 2007 11:46:11 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of akarasulu@gmail.com designates 209.85.198.184 as permitted sender) Received: from [209.85.198.184] (HELO rv-out-0910.google.com) (209.85.198.184) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Sep 2007 18:46:12 +0000 Received: by rv-out-0910.google.com with SMTP id g11so1516144rvb for ; Tue, 25 Sep 2007 11:45:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; bh=G7Bly+H/y1pwk41YyKo5NEvJKqvDWG2JNw2Jv2jdaeU=; b=oteca3rlV9z+DXFRCeS/A/9CtItGIw1vVRRVFZ2KZASDQ/aP9v83ZwdjwXkm1SuQXLjo3lImm4ZqdkjRYZ67aCBXcHstiNggv7P8GeuSFNPRVRSXvg4jeY9lPXbMFWxDr3qVAKYHYhcSQMsYW6ZbrkhK4JzVMPs7BoohSIsflMI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:references:x-google-sender-auth; b=iLh7LngiPe3+wi4bXE+HIbOycCNbXOlA9ldy8zLqq8TuLcLlifxm2RZtGSe9w0l+onfa1bo9sJlnhe5fzVOJpGXmysEPnwNo54rg9WKCeCCgvSR9L2bWepRDg0W7W3c3HJ3ycatG/+RYsmMiTsKXmm7Rq9GL4VC8WLRa35Acoi4= Received: by 10.114.73.1 with SMTP id v1mr3353213waa.1190745952314; Tue, 25 Sep 2007 11:45:52 -0700 (PDT) Received: by 10.115.76.8 with HTTP; Tue, 25 Sep 2007 11:45:52 -0700 (PDT) Message-ID: Date: Tue, 25 Sep 2007 14:45:52 -0400 From: "Alex Karasulu" Sender: akarasulu@gmail.com To: "Apache Directory Developers List" Subject: Re: [ApacheDS] Mitosis status inquiry In-Reply-To: <46F42C7E.9070609@planetquake.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2595_13941338.1190745952315" References: <46F42C7E.9070609@planetquake.com> X-Google-Sender-Auth: c62ceae287ef7b7a X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_2595_13941338.1190745952315 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hey Martin sorry for taking so long to respond to you especially when you responded so quickly and thoroughly. More in line ... On 9/21/07, Martin Alderson wrote: > > Hi Alex, > > For my fairly limited use mitosis is performing very well. This is excellent news. I've slacked off a bit over the last few weeks but will try to get back > onto it soon. No worries me too. I keep context switching these days myself. For stable use there is still that critical issue where a change on the > same entry on multiple servers will lead to a permanently inconsistent > state. I'll make that my priority soon. Ok what I want to do is setup and environment soon and start trying to reproduce this as well as any other problems. This way I can work with you or at least help answer any questions you have while attacking this problem. I guess we can have intermediate states where the replicas are not in sync but they must converge over time so this is a critical issue. I'll try to help out as best as I can on this. I also recently came across a minor timeout issue which I think is a > problem in MINA. I'll investigate that more soon too. This is new to me but if you localize it I'm sure we can fix it. Aside from that there are niggling issues like replication of schema > changes. Yes this is something I am dreading. It's almost as if replicating a partition must force replicating the schema across that cluster but other clusters on different partitions will be all tied together for schema changes. This is why Microsoft decided to put schema into each partition for ADAM. Although painful we could do the same but I think we can work around this without one offs but it will shift the design a bit. Once these issues have been worked out I think we have a > stable replication system. Yep. There are some features that I guess we really need to get in for it to > actually be useful for the majority of users though. The main things > that come to mind are selective replication, encryption, schedules and > removing the dependency on Derby. Yes these are also my concerns especially encrypting the replication channel. We have some ideas that Ersin and I tossed around regarding all these issues. I guess we should start simple. First we should think about integrating the UUID capabilities which are part of mitosis into ApacheDS core so UUID is supported whether you turn on replication or not. Then from there I'd like to make the quartz scheduler a core service (not interceptor) that is accessible from the DirectoryService interface to be able to schedule anything. So moving these things up simplifies Mitosis a bit. Then it's reasonable to just move the replication log off of Derby into a custom store implementation based on JDBM. Eventually we'll need to expose this data via LDAP but we don't need to do it immediately. For now getting rid of the dep and having a clean store implementation is enough. I know you and Ersin are thinking about possible design changes although > to me those can be done incrementally later on as required. I doubt > that I will be a driving force behind these changes but am willing to > help. > Right I agree with you on the incremental changes. Don't sell yourself short the new design changes are things I think you can easily grok. Plus we're not moving fast with anything at the moment - we just have ideas. The problem has been where to dig in and start getting traction. I think we can do the things above in parallel with introducing design changes as well. Alex ------=_Part_2595_13941338.1190745952315 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hey Martin sorry for taking so long to respond to you especially when
you responded so quickly and thoroughly.  More in line ...

On 9/21/07, Martin Alderson <equim@planetquake.com> wrote:
Hi Alex,

For my fairly limited use mitosis is performing very well.

This is excellent news.

I've slacked off a bit over the last few weeks but will try to get back
onto it soon.

No worries me too.  I keep context switching these days myself.

For stable use there is still that critical issue where a change on the
same entry on multiple servers will lead to a permanently inconsistent
state.  I'll make that my priority soon.

Ok what I want to do is setup and environment soon and start trying to
reproduce this as well as any other problems.  This way I can work with
you or at least help answer any questions you have while attacking this
problem.  I guess we can have intermediate states where the replicas
are not in sync but they must converge over time so this is a critical issue.
I'll try to help out as best as I can on this.

I also recently came across a minor timeout issue which I think is a
problem in MINA.  I'll investigate that more soon too.

This is new to me but if you localize it I'm sure we can fix it.

Aside from that there are niggling issues like replication of schema
changes.  

Yes this is something I am dreading. It's almost as if replicating a
partition must force replicating the schema across that cluster but
other clusters on different partitions will be all tied together for schema
changes.

This is why Microsoft decided to put schema into each partition for ADAM. 
Although painful we could do the same but I think we can work around this without
one offs but it will shift the design a bit.

Once these issues have been worked out I think we have a
stable replication system.

Yep.

There are some features that I guess we really need to get in for it to
actually be useful for the majority of users though.  The main things
that come to mind are selective replication, encryption, schedules and
removing the dependency on Derby.

Yes these are also my concerns especially encrypting the replication channel.
We have some ideas that Ersin and I tossed around regarding all these issues.

I guess we should start simple.  First we should think about integrating the
UUID capabilities which are part of mitosis into ApacheDS core so UUID is
supported whether you turn on replication or not.

Then from there I'd like to make the quartz scheduler a core service (not
interceptor) that is accessible from the DirectoryService interface to be able to
schedule anything.

So moving these things up simplifies Mitosis a bit.

Then it's reasonable to just move the replication log off of Derby into a custom
store implementation based on JDBM.  Eventually we'll need to expose this data
via LDAP but we don't need to do it immediately.  For now getting rid of the dep
and having a clean store implementation is enough.

I know you and Ersin are thinking about possible design changes although
to me those can be done incrementally later on as required.  I doubt
that I will be a driving force behind these changes but am willing to help.

Right I agree with you on the incremental changes.  Don't sell yourself short the
new design changes are things I think you can easily grok.  Plus we're not moving
fast with anything at the moment - we just have ideas.  The problem has been
where to dig in and start getting traction.  I think we can do the things above in
parallel with introducing design changes as well.

Alex


------=_Part_2595_13941338.1190745952315--