Mailing-List: contact dev-help@directory.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Apache Directory Developers List" <dev@directory.apache.org>
Received-SPF: pass (herse.apache.org: domain of aok123@bellsouth.net
 designates 205.152.59.70 as permitted sender)
Message-ID: <455003AB.7050802@bellsouth.net>
Date: Mon, 06 Nov 2006 22:55:23 -0500
From: Alex Karasulu <aok123@bellsouth.net>
User-Agent: Thunderbird 1.5.0.7 (X11/20060922)
MIME-Version: 1.0
To: Apache Directory Developers List <dev@directory.apache.org>
Subject: [Mitosis] IM Session with Trustin
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,

I'd like to report on some questions, their answers and some conclusions 
reached while Trustin and I were talking on IM about Mitosis:

======================= Start =========================


1). When adding an entry that was already marked entryDeleted true, why 
does mitosis perform a modify operation on the entry instead?

Answer: I don't remember maybe I did it this way because of a lack of 
knowledge.

2). In the OperationFactory.newAdd(CSN,LdapDN,Attributes) method the 
first thing you do is call checkBeforeAdd(LdapDN) whose short body is 
printed out below:

     private void checkBeforeAdd( LdapDN newEntryName ) throws 
NamingException
     {
         if ( nexus.hasEntry( newEntryName ) )
         {
             throw new NameAlreadyBoundException( 
newEntryName.toString() + " already exists." );
         }
     }

On add operations where the entry is marked entryDeleted=true, this 
method always throws an exception and shorts the progress of the add() 
operation.  I have modified it to check if the entry is marked as 
deleted first before throwing an exception like so:

     private void checkBeforeAdd( LdapDN newEntryName ) throws 
NamingException
     {
         if ( nexus.hasEntry( newEntryName ) )
         {
             Attributes entry = nexus.lookup( newEntryName );
             Attribute deleted = entry.get( Constants.ENTRY_DELETED );
             Object value = deleted == null ? null : deleted.get();

             /*
              * Check first if the entry has been marked as deleted before
              * throwing an exception and delete the entry if so and return
              * without throwing an exception.
              */
             if ( value != null && "true".equalsIgnoreCase( 
value.toString() ) )
             {
                 return;
             }

             throw new NameAlreadyBoundException( 
newEntryName.toString() + " already exists." );
         }
     }

Now is this the right thing to do?

Answer: Yes this is good.

3).  After making this change I still have some problems.  Let me 
describe the sequence of events in terms of 2 replicas A and B and an 
add operation:

   (i) add entry (1) to replica A

Everything is fine up to now.  Replication of the add propagates just 
fine.  I can see the entry in both replica A and B.

   (ii) delete entry (1) from replica A

Everything is fine up to now.  Replication of the del propagates just 
fine.  I can not see the entry in either replica A and B.  Pulling up 
the diagnostic gui I can see that the entry is in the partition but 
marked entryDeleted = true.

   (iii) add entry (1) back to replica A with same DN and same contents

Here I get a *FAILURE*.  The error response I get from the server is:

06:06:49 PM: Failed to add new entry ou=neworganizationalUnit, 
dc=example,dc=com Root error: [LDAP: error code 80 - failed to add entry
ou=neworganizationalUnit,dc=example,dc=com: Unexpected exception.]

In fact I get this error if I try to add to either replica A or B. At 
this point I started a debug session to determine exactly what is going 
on which prevents this 2nd add operation from working.

3a). I noticed that for add you build a CompositeOperation but really 
you're only putting into it a single AddEntryOperation.  Did you intend 
to do this or is it a simple bug?

Answer: Yeah you are right we don't need to wrap it.

3b). To the entry to be added you remove any entryDeleted and entryUUID 
attributes and add a entryDelete (false), entryUUID and an entryCSN 
attribute.  Then execute() is called eventually for the 
AddEntryOperation.  Here you're checking if the entry is updateable by 
calling EntryUtil.isEntryUpdatable.   Here's the code for it:

http://svn.apache.org/repos/asf/directory/trunks/apacheds/mitosis/src/main/java/org/apache/directory/mitosis/operation/support/EntryUtil.java

Basically you lookup the entry with a call to the nexus.  Then you 
return true if the entry does not exist.  Is this valid?  Meaning if the 
entry does not exist in the store how can it be updatable?

You also check if there is no CSN attribute.  If there is no attribute 
then you return true that the entry is updatable.  What does this mean?

It would help if you clarify the meaning of these methods in EntryUtils 
with a bit more doco.  Also it would be nice if you elaborate more on 
how mitosis does not delete entries, and why it does this as well as 
when and how it cleans up later.  This is all critical stuff for the 
architecture that was omitted.

Answer:

(20:52:07) trustin: first, if there's no entryCSN attribute, it means that
(20:52:24) trustin: an entry is added before          Mitosis is in action.
(20:52:55) akarasulu: ahh ok
(20:53:02) trustin: so it should be updatable and needs to get the CSN 
assigned by modification
(20:53:09) akarasulu: ok
(20:53:23) trustin: and then it compares the CSN
(20:53:37) akarasulu: why do you return true if entry == null ?
(20:54:28) trustin: Semantics of update includes both add and modify.. 
so it's updatable if there's no such entry.
(20:54:42) akarasulu: ahhh ok so add is considered an update
(20:54:58) trustin: yeah update is all operations that affects the DIT.

3c). The code goes past these checks for this scenario since the entry 
does exist and is marked entryDeleted: true and it does have a CSN 
attribute.  Finally you compare whether this old CSN is older than the 
new CSN.  If so then you return true.

What is the meaning of this?  Is it just an undefined situation where 
you should not be changing an entry with CSN that is older than the CSN 
of the original entry?  Why then return true or false for 
isEntryUpdatable?  Should you not just throw an exception if this
is just an undefined situtation?

Answer: We need to decide between two CSN's, the existing one or the new 
operation at hand.

3d). Next you create glue entries.  This is not very clear to me.  It 
was not in your architecture document.  I think it is very important to 
define exactly what this is for to understand mitosis.

Inside EntryUtils.createGlueEntries() you are using getSuffix() to walk 
a DN.   How do you intend to walk the DN?  From the top down like so for 
example?

ou=system
ou=A,ou=system
ou=B,ou=A,ou=system

For each one you are trying to create a glue entry I think.  There was 
yet no need for a glue entry but we have to understand what it is and 
make sure they don't violate schema or we'll bomb out.  Can you 
elaborate a little on glue entries?

Answer: will put it into another iteration of the mitosis architecture doco.

3e). In an effort to modify the entry you are removing all attributes 
that the two old and new entries do not have in common.  However you are 
removing these from the cloned version of the old entry.  Then you never 
use this old entry that has attributes removed from it.

Is this another bug?

Answer: Yes

4) If I restart a server after shuting it down I get the following error 
and the server dies:

     Caused by: java.net.BindException: Address already in use

We found a way to stop this from happening with the LDAP port used by 
the server.  Can
we make the mitosis replication service port also behave similarly so 
this does not
happen?

Answer: config.setReuseAddress( true );

5). Looks like you have not finished implementing the move operation as 
is indicated
here in OperationFactory line 188 in newMove():

         if ( !deleteOldRn )
         {
             throw new OperationNotSupportedException( "deleteOldRn must 
be true." );
         }

Is it that you did not implement this or that it was not possible 
because of some issue?

6). Can you please add some internal documentation to the mitosis code?

Answer: Yep

7). Can you update the archtecture documentation to include information 
about glue entries, not deleting and cleaning up stuff?

Answer: Yep

8). Could you also elaborate more on the log and how it is used?

Answer: Yep

9). Can we detect when we are out of sync and email administrators?

(22:46:27) akarasulu: yes right ?
(22:46:33) trustin: yes.

===========================================

Notes:

(1) We need to log when we have dropped operations.
(2) We need to email administrators when we have dropped ops and/or when 
we are out
of sync.

Decided it is best to recursively delete entries marked as deleted when 
  re-adding the
entry from scratch.


Alex