directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lécharny <>
Subject Re: Question about the Log file
Date Fri, 16 Mar 2012 20:01:07 GMT
Le 3/16/12 7:07 PM, Selcuk AYA a écrit :
> On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny<>  wrote:
>> Hi,
>> AFAICS, the og file contains a buffer which stores UserLogRecord until we
>> write them on disk. This allows the Log system to be fast, and also allow a
>> dedicated thread to flush the data on disk regularly.
>> So far, so good.
>> But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
>> instead, as the Log file size is fixed. We will then let the underlying OS
>> to deal with flushes, not needing a dedicated thread to handle it. Also
>> MemoryMappedFile are faster than RandomFile, as it's working on pages, which
>> are managed by the OS (their size is depending on the OS tuning). Last, not
>> least, we won't need to dedicate a 4Mb buffer to store temporary
>> userRecords, as MemoryMappedFiles aren't using the JVM memory.
> the current implementation is flexible to work with any underlying
> file system rather than being tied to a single implementation.
MemoryMappedFile in Java are not OS dependent. It's a feature that has 
been added in Java 4, and it works on top of any FS.
> Currently it is a random access file system. But memory mapped file
> implementation should work as well. I am also thinking of using HDFS
> files in the future.A couple of notes about some concerns:
> *I dont see 4MB being a big concern. This size can be tuned. It can
> also be made zero if the underlying implementation is good to deal
> with writes.
> *I dont think a dedicated background thread is a problem either. If we
> want, we can make user threads to do the log sync as they log the
> records.
> Right now this is very low priority given the things we need to
> implement to get the txns to work.
Sure ! Just wanted to mention that, AFAIU, the current log with the 
buffer could probably be switched to a MMF with some extra benefits in 
the future.

There are some area that need to be checked, like the file locks and how 
we should manage them.

I'll create a JIRA mentionning that this is an area which may need to be 
investigated later, as it's not urgent.

We discussed a lot with Alex about using MMF instead of other kind of 
file IO especially if we are to rewrite JDBM at some point.

One more question, relative to the LogBuffer : I see it's flushed in two 
cases :
- when the buffer is full (obviously !)
- if a thread is calling the Log.sync() method.

My understanding is that there will be a thread somewhere which will 
call this Log.sync() method periodically, right ?

One last thing : we will need to expose the configurable size so that it 
can be changed. ALl our configuration is stored in the cn=config branch, 
I will try to draft something on the wiki to gather all the configurable 
elements plus the suggested AT/OC needed to manipulate those elements.

I'll be mostly offline this week-end, because I don't have any credit on 
my 3G phone, and  also forgot to bring my phone charger. I'll be back on 
monday afternoon.

Have a good week-end !

Emmanuel Lécharny

View raw message