A transaction journal (a.k.a. transaction log) is used to rapidly persist write operations arriving at the server before being processed.  Local transactions for these operations are opened to apply the change. This allows the server to replay operations which did not complete due to some interruption during processing. It also allows the server eventually to apply changes and their side effects (like those from triggers), in the same transaction which allows all to rollback together or to succeed together.  We need these things eventually even though they may not be critical right now for replication.

Journal file formats are simple, with indices into these files to track those operations that have completed from those that have not.  Journals are not ideal for a search-able history retrieval system to be used for auditing and snapshotting.  The history CL requires many more indices and it's information needs to be more structured.  Conversely search-able CL is not ideal as a transaction log since organizing the information and updating all these indices requires disk operations which take too much time.

We have to be clear on what we want as a set of requirements.  If we're going to implement a transaction journal/log here's what I'd like to see:

0). Very fast write of operation information to disk including any information needed to rollback an operation.
1). The journal should be the basis for implementing local transactions and indices into it should be minimized for performance sake.
2). The transaction log should be pruned asynchronously removing operations that have been processed.  These operations can then be pumped into the CL, for audit history and snapshotting.

WDYT? Please add more, I have more thoughts which I can add later.


On Mon, Jan 26, 2009 at 11:29 AM, Emmanuel Lecharny <elecharny@gmail.com> wrote:
>> Just because I think it's easier to recover from a crashed LDIF file
>> than from a crashed JDBM file :)
> Good point.  However as you said we might want a separate journal using
> something howl or the alternative that David Jencks pointed out being using
> by the AMQ folks.

Well, we should not close doors at this point. I know nothing about
howl, so I would like to see if we can use it.