From dev-return-40631-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Fri Mar 16 20:02:02 2012 Return-Path: X-Original-To: apmail-directory-dev-archive@www.apache.org Delivered-To: apmail-directory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CA2A695D8 for ; Fri, 16 Mar 2012 20:02:02 +0000 (UTC) Received: (qmail 91944 invoked by uid 500); 16 Mar 2012 20:02:02 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 91902 invoked by uid 500); 16 Mar 2012 20:02:02 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 91895 invoked by uid 99); 16 Mar 2012 20:02:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Mar 2012 20:02:02 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of elecharny@gmail.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Mar 2012 20:01:56 +0000 Received: by wibhq7 with SMTP id hq7so1380541wib.1 for ; Fri, 16 Mar 2012 13:01:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:reply-to:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=TRwjgc9qxMz3bll3pnKlIoTorik4KIaM9RI0N7GYbLU=; b=i3+DVFlar6B+9nkP6dkSbdP3ZqQBXW9pDhgL0rMH5RMmk/FdIXHZqX6BkuVmsO8wkQ 5u1PD4S3TW7BmCDNKtCUKfe4n0cINntIureGwOd+lsNJ+9p4eRHsp55YWygRyYZ1cxc2 tv4qPDwBE5UqRrCNYWOTQ6UlfJjT5Rzehu3YlZ56ccMPlS5Sn8DqB42m49k88Bcc9Ip0 RbMGzJLzzkQo2wgMiWgIWbwLHjM3GDTTTdKvi4FaB8O3/b8OydTSGvif5VVaGqnJyMZO rZSE6mqfy4m0pcNcUrcxpUxa67VmicwDzxBiOspidQEXDVjW+9Vl2QCusZlt7lb8X2vl 9Sjw== Received: by 10.216.133.137 with SMTP id q9mr2360281wei.12.1331928094736; Fri, 16 Mar 2012 13:01:34 -0700 (PDT) Received: from Emmanuels-MacBook-Pro.local ([90.84.146.247]) by mx.google.com with ESMTPS id df3sm1411984wib.1.2012.03.16.13.01.28 (version=SSLv3 cipher=OTHER); Fri, 16 Mar 2012 13:01:34 -0700 (PDT) Message-ID: <4F639C03.8030304@gmail.com> Date: Fri, 16 Mar 2012 21:01:07 +0100 From: =?UTF-8?B?RW1tYW51ZWwgTMOpY2hhcm55?= Reply-To: elecharny@apache.org User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: Apache Directory Developers List Subject: Re: Question about the Log file References: <4F630734.6040206@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Le 3/16/12 7:07 PM, Selcuk AYA a écrit : > On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny wrote: >> Hi, >> >> AFAICS, the og file contains a buffer which stores UserLogRecord until we >> write them on disk. This allows the Log system to be fast, and also allow a >> dedicated thread to flush the data on disk regularly. >> >> So far, so good. >> >> But I'm wondering if it would'nt be simpler to use a MemoryMappedFile >> instead, as the Log file size is fixed. We will then let the underlying OS >> to deal with flushes, not needing a dedicated thread to handle it. Also >> MemoryMappedFile are faster than RandomFile, as it's working on pages, which >> are managed by the OS (their size is depending on the OS tuning). Last, not >> least, we won't need to dedicate a 4Mb buffer to store temporary >> userRecords, as MemoryMappedFiles aren't using the JVM memory. > the current implementation is flexible to work with any underlying > file system rather than being tied to a single implementation. MemoryMappedFile in Java are not OS dependent. It's a feature that has been added in Java 4, and it works on top of any FS. > Currently it is a random access file system. But memory mapped file > implementation should work as well. I am also thinking of using HDFS > files in the future.A couple of notes about some concerns: > > *I dont see 4MB being a big concern. This size can be tuned. It can > also be made zero if the underlying implementation is good to deal > with writes. > *I dont think a dedicated background thread is a problem either. If we > want, we can make user threads to do the log sync as they log the > records. > > Right now this is very low priority given the things we need to > implement to get the txns to work. Sure ! Just wanted to mention that, AFAIU, the current log with the buffer could probably be switched to a MMF with some extra benefits in the future. There are some area that need to be checked, like the file locks and how we should manage them. I'll create a JIRA mentionning that this is an area which may need to be investigated later, as it's not urgent. We discussed a lot with Alex about using MMF instead of other kind of file IO especially if we are to rewrite JDBM at some point. One more question, relative to the LogBuffer : I see it's flushed in two cases : - when the buffer is full (obviously !) - if a thread is calling the Log.sync() method. My understanding is that there will be a thread somewhere which will call this Log.sync() method periodically, right ? One last thing : we will need to expose the configurable size so that it can be changed. ALl our configuration is stored in the cn=config branch, I will try to draft something on the wiki to gather all the configurable elements plus the suggested AT/OC needed to manipulate those elements. I'll be mostly offline this week-end, because I don't have any credit on my 3G phone, and also forgot to bring my phone charger. I'll be back on monday afternoon. Have a good week-end ! -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com