aurora-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxim Khutornenko (JIRA)" <>
Subject [jira] [Commented] (AURORA-942) Explore using a replicated log on top of ZooKeeper
Date Thu, 20 Nov 2014 23:31:33 GMT


Maxim Khutornenko commented on AURORA-942:

Still, using something that was not designed to be a database [1] feels like a forced approach
even considering our moderate (currently) data footprint:
ZooKeeper was not designed to be a general database or large object store. Instead, it manages
coordination data. This data can come in the form of configuration, status information, rendezvous,
etc. A common property of the various forms of coordination data is that they are relatively
small: measured in kilobytes. The ZooKeeper client and the server implementations have sanity
checks to ensure that znodes have less than 1M of data, but the data should be much less than
that on average. Operating on relatively large data sizes will cause some operations to take
much more time than others and will affect the latencies of some operations because of the
extra time needed to move more data over the network and onto storage media.

The fact that we have chunking is great but managing sequencing would be completely on us,
perhaps following the sequenced node approach [2]. Not saying it's impossible, just more/different
things to worry about, including the ZK data backup.

[1] -
[2] -

> Explore using a replicated log on top of ZooKeeper
> --------------------------------------------------
>                 Key: AURORA-942
>                 URL:
>             Project: Aurora
>          Issue Type: Task
>          Components: Scheduler
>            Reporter: Bill Farner
>            Priority: Minor
> The scheduler uses the replicated log implementation provided by mesos (native
 It would be interesting to compare this against a replacement that sllows us to:
> - shed code to implement backups and recovery
> - remove one use of a dynamically-linked native library
> - use a store that allows non-leaders to read, for faster recovery and serving from non-active
> - avoid the need for periodic failover (we currently have to do this to induce compaction
in LevelDB and minimize log replay time)
> At first glance, it seems like it would be relatively straightforward to come up with
a Log implementation \[1\] that persists transactions as nodes in ZooKeeper.  This would enable
all the above results.
> \[1\]

This message was sent by Atlassian JIRA

View raw message