directmemory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Kotek <dis...@kotek.net>
Subject Re: WAL Implementation
Date Sun, 24 Mar 2013 18:41:08 GMT
Hi,

There is WAL implementation (called journal) in MapDB. It has an interesting 
feature that modified data written into log, are not stored in memory, but can 
be re-read directly from log. MapDB is not exactly DB, it is more like 
persistent heap. 

Here is WAL storage implementation:
https://github.com/jankotek/MapDB/blob/master/src/main/java/org/mapdb/StorageJournaled.java

There is also 'direct' (update on place) and append-only storage 
implementation. Please note that I am currently reimplementing this store to 
be lock-free. In couple of days this file will be completely replaced.

Hope it helps.
Jan


On Sunday 24 March 2013 19:13:26 Christoph Engelbert wrote:
> Hey guys,
> 
> after a few weeks heavily busy at work to bring our new game to open
> beta I finally have some time to work on lovely opensource stuff
> again :-)
> 
> Currently I'm implementing a generic WAL (Write Aheat Log / Journal)
> implementation, in first place for the persistence system at our
> company.
> 
> We collect statements in a queue to be written in a background
> thread to linearize database load.
> The problem about this approach is if db servers are busy this queue
> can take some time to be cleaned up and if the gameservers crash
> before the queue is cleared (or at least the background persister is
> killed - for whatever reason - yeah we had a bug where data weren't
> written for about 4 days) player data are lost.
> 
> The new system forced all statements to be written to disk before
> being enqueued so that journals can be replayed on gameserver
> startup. I haven't found any ready to use implementation beside
> implementations found in frameworks like Hadoop, databases (I guess
> it was derby), hornetmq, etc and so I started my own implementation.
> I'll try to make it as generic as possible to not force it to be
> used for persistency (SQL Statements) only but even for maybe
> journaling memory access (or whatever).
> 
> Do you guys think it could be interesting for DM to implement some
> thing as WAL in some place? Or do you have other interesting ideas
> what to do with it?
> 
> I'll look forward to hopefully an intensive discussion. Maybe
> someone else has found a WAL implementation that could be used /
> analysed :-)
> 
> Chris / Noc

Mime
View raw message