From dev-return-29225-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Mon Mar 02 19:44:35 2009 Return-Path: Delivered-To: apmail-directory-dev-archive@www.apache.org Received: (qmail 49558 invoked from network); 2 Mar 2009 19:44:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Mar 2009 19:44:35 -0000 Received: (qmail 71434 invoked by uid 500); 2 Mar 2009 19:44:35 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 71394 invoked by uid 500); 2 Mar 2009 19:44:35 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 71383 invoked by uid 99); 2 Mar 2009 19:44:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 11:44:35 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of elecharny@gmail.com designates 209.85.219.170 as permitted sender) Received: from [209.85.219.170] (HELO mail-ew0-f170.google.com) (209.85.219.170) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 19:44:25 +0000 Received: by ewy18 with SMTP id 18so2156125ewy.25 for ; Mon, 02 Mar 2009 11:44:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=nqualIXx33CM+ZdC1UPQezSBbjhkMdAHeVXe0F/3bQM=; b=OsbYa4ieN8TCXJs+7jJzwYeJ3CofwQXb9AHrKX9eiw6B1NRHpVdesev2jCarMteyYd a2Iu9YL4TXD1GTm2sygXWe+vKcpEove9cT4P80plFFGyXPq1tYAXz68GCqDrT56wDodg 4RAWGjNbs3pe4068e1Iyemis3pYITdViNDNiA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; b=bUhExzYm/7rl1i+ZQFb8WG8ypihHIp0xqEvUWdFFOMd28kCIQnuJGTKXEEl/6ZJ4yo gUUDY/6gWmF+v3CVUxEAp/Lyl/D7i0ZJ1oN+B+lAokKGg2TBX3tzMAkFELpEq8Ro57N5 RW5NlzH/ILNrGiJKPowgxDm3obgs5eP8unxxA= Received: by 10.211.194.4 with SMTP id w4mr5011872ebp.42.1236023044660; Mon, 02 Mar 2009 11:44:04 -0800 (PST) Received: from ?192.168.0.11? ([78.226.4.211]) by mx.google.com with ESMTPS id 28sm459515eye.39.2009.03.02.11.44.03 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 02 Mar 2009 11:44:03 -0800 (PST) Sender: Emmanuel Lecharny Message-ID: <49AC3703.7060401@nextury.com> Date: Mon, 02 Mar 2009 20:44:03 +0100 From: Emmanuel Lecharny User-Agent: Thunderbird 2.0.0.19 (X11/20090105) MIME-Version: 1.0 To: Apache Directory Developers List Subject: Re: Journal first draft References: <49A9C767.5030101@nextury.com> <49AC318B.9030309@nextury.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Alex Karasulu wrote: > OK thanks for the clarifications. > This was the easy part :) I'm now dealing with the hard part : - how to avoid the journal to grow indefinitively - how to use the journal with the coming replication system - how to build the recovery system I have some idea about #1 and #3 : Considering that the journal is just a representation of the pending operation, we can rotate the file every N operations, or each time we sync(), if of course we have had some modifications, or evey T seconds. Once we have a rotated the file, we can check if all the operation has been acked, and if so, we can ditch them. If the server works well, we won't have operation pending for ever. When the server crash, or when we stop it and restart it (it makes no difference), we just have to check on the journal if we have pending operation. If so, we will check if the pending operations have been applied or not (it's easy for add or del operation, a bit more complicated for modifications, definitively more complex for move and rename, as the operation can have been partially applied). What if the base is screwed ? (ie, we can't restart the server because the backend files are FU). Then we have to start from a recent backup, and apply all the journal on this backup. So next question : how do we get this backup ? We have to build it : the idea is to have a local server working as a slave, and periodically applying the journal on its base. When done, the server is stopped, the partitions are saved, and we just wait for the next backup to start. So if the machine crashes, even during a backup, we can still start from the previous saved file and reapply the journal on it. Then when the journal has been applied, we just have to copy the backup partition to the real server, and we are done. Seems a bit complex, but don't worry : it is complex :) > Alex > > On Mon, Mar 2, 2009 at 2:20 PM, Emmanuel Lecharny wrote: > > >> Alex Karasulu wrote: >> >> >>> So wdyt about this first version ? What's missing ? >>> >>>> >>>> >>> (1) Why not make these comment attribute actual attributes in the LDIF >>> even >>> though they are special? >>> >>> >>> >> I was considering this as an option, but the problem is that you need to >> clone the entry in order to add those special attributes, ar add them and >> remove them immediately, which is not really elegant. >> >> >>> (2) Why is rev not a regular attribute? >>> >>> >>> >> Same reason. And we can't use the ChangeLog rev because it is not added if >> the changeLog is not activated. >> >> In any case, we don't need to store those elements in the partition within >> the entry, because they are volatile (each time you restart the server, the >> revision is set to a new value) >> >> >>> (3) What exactly is the difference between ACK and NACK and why do we need >>> a >>> NACK? >>> >>> >>> >> We will have ACK for entries which have been stored in the partition. NACK >> is for entries that we _know_ haven't been stored for any reason. If the >> entry has not been stored yet, or not been rejected yet, then we won't have >> any ACK or NACK. >> >> It's interesting to have a NACK to distinguish between those two cases. >> >> >> -- >> -- >> cordialement, regards, >> Emmanuel Lécharny >> www.iktek.com >> directory.apache.org >> >> >> >> > > -- -- cordialement, regards, Emmanuel Lécharny www.iktek.com directory.apache.org