hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "BookKeeper" by BenjaminReed
Date Wed, 24 Jun 2009 07:33:12 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by BenjaminReed:

New page:
= BookKeeper =

BookKeeper is a system to reliably log streams of records. It is designed to store write ahead
logs, such as those found in database or database like applications. In fact, the Hadoop NameNode
inspired BookKeeper. The NameNode logs changes to the in-memory namespace data structures
to the local disk before they are applied in memory. However logging the changes locally means
that if the NameNode fails the log will be inaccessible. We found that by using BookKeeper,
the NameNode can log to distributed storage devices in a way that yields higher availability
and performance. Although it was designed for the NameNode, BookKeeper can be used for any
application that needs strong durability guarantees with high performance and has a single

In BookKeeper, servers are "bookies", log streams are "ledgers", and each unit of a log (aka
record) is a "ledger entry". BookKeeper is designed to be reliable; bookies, the servers that
store ledgers can be byzantine, which means that some subset of the bookies can fail, corrupt
data, discard data, but as long as there are enough correctly behaving servers the service
as a whole behaves correctly; the meta data for BookKeeper is stored in ZooKeeper.

BookKeeper achieves high availability and strong durability guarantees by replicating ledger
entries across multiple bookies. The ledgers themselves are striped across the bookies for
high performance.

The BookKeeper data model is a flat namespace of ledgers identified by a long. The ledgers
are append only and writable by a single client. The basic operations of BookKeeper are: createLedger
to create a new ledger available for writing, openLedger to read from an existing ledger,
addEntry, removeEntry, and closeLedger. Once a ledger is closed it becomes read-only.

''once the 3.2 release happens we will include a link to the documentation here''. 

View raw message