incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sijie Guo <>
Subject [DISCUSS] DistributedLog Incubation Proposal
Date Thu, 09 Jun 2016 04:34:46 GMT

I would like to propose DistributedLog to be an Apache Incubator project.

DistributedLog is a high performance replicated log service.
It offers durability, replication and strong consistency, which provides
a fundamental building block for building reliable distributed systems,
e.g replicated-state-machines, general pub/sub systems, distributed
databases, distributed queues and etc.

Here's a link to the proposal in the Incubator wiki

I've also pasted the initial contents below.



= Abstract =
DistributedLog is a high-performance replicated log service. It offers
durability, replication and strong consistency, which provides a
fundamental building block for building reliable distributed systems,
e.g replicated-state-machines, general pub/sub systems, distributed
databases, distributed queues and etc.

See “Building Distributedlog - Twitter’s high performance replicated
log service” for details:

= Proposal =
We propose to contribute DistributedLog codebase and associated
artifacts (e.g. documentation, web-site content etc.) to the Apache
Software Foundation with the intent of forming a productive,
meritocratic and open community around DistributedLog’s continued
development, according to the ‘Apache Way’.

= Background =
Engineers at Twitter began developing DistributedLog in early 2013.
DistributedLog is described in a Twitter engineering blog post and
presented at the Messaging Meetup in Sep 2015. It has been released as
an Apache-licensed open-source project on GitHub in May 2016.

DistributedLog is a high-performance replicated log service, which
provides simple stream-oriented abstractions over log-segments and
offers durability, replication and strong consistency for building
reliable distributed systems. The features offered by DistributedLog
 * Simple high-level, stream oriented interface
 * Naming and metadata scheme for managing streams and other entities
 * Log data management policies, include data segmentation and data retention
 * Fast write pipeline leveraging batching and compression
 * Fast read mechanism leveraging long-poll and read-ahead caching
 * Service tiers supporting writer fan-in and reader fan-out
 * Geo-replicated logs

DistributedLog’s most important benefit is high-performance with a
strong durability guarantee, making it extremely appropriate for
running different workloads from distributed database journaling to
real-time stream computing. Its modern, layered architecture makes it
easy to run the service tiers in multi-tenant datacenter environments
such as Apache Mesos or cloud environments such as EC2.

= Rationale =
DistributedLog is designed to provide core fundamental features like
high-performance, durability and strong consistency to anyone who is
building reliable distributed systems, in a simple and efficient way.

We believe that the ASF is the right venue to foster an open-source
community around DistributedLog’s development. We expect that
DistributedLog will benefit from collaboration with related Apache
projects, and under the auspices of the ASF will attract talented
contributors who will push DistributedLog’s development forward at a
faster pace.

We believe that the timing is right for DistributedLog’s development
to move to the ASF: DistributedLog has already run in production at
Twitter for 3 years and served various workloads including a
distributed database journal, reliable cross datacenter replication,
search ingestion, andgeneral pub/sub messaging. The project is stable.
We are excited to see where an ASF-based community can take

= Current Status =
DistributedLog is a stable project that has been used in production at
Twitter for 3 years. The source code is public at,
which will seed the Apache git repository.

= Meritocracy =
We understand the central importance of meritocracy to the Apache Way.
We will work to establish a welcoming, fair and meritocratic
community. Several companies have already expressed interest in this
project, and we intend to invite additional developers to participate.
We look forward to growing a rich user and developer community.

= Community =
There is a large need for a performant replicated log service for
applications such as distributed databases, distributed transactional
systems, replicated-state-machines and pub/sub messaging/queuing. We
want to attract more developers to the project, and we believe that
the ASF’s open and meritocratic philosophy will help us with this. We
note the success of other similar projects already part of the ASF,
like Kafka.

= Core Developers =
DistributedLog is actively developed within Twitter. Most of the
developers are from Twitter. Many of them are committers or PMC
members of Apache BookKeeper. Others aren’t currently affiliated with
ASF so they will require new ICLAs.

= Alignment =
DistributedLog is related to several other Apache projects:
 * DistributedLog stores log segments as Ledgers in Apache BookKeeper.
 * DistributedLog uses Apache ZooKeeper for naming and metadata
management and tracking the ownership of logs.
 * DistributedLog uses Apache Thrift as its RPC and serialization framework.
 * In the long-term, DistributedLog’s data will be stored in Apache
Hadoop clusters powered by HDFS filesystem for archives and backup.

= Known Risks =

== Orphaned Products ==
DistributedLog is used as the fundamental messaging infrastructure at
Twitter. It has been serving production traffic for online database
systems, search ingestion and a general pub/sub system. Twitter
remains committed to developing and supporting the project. Twitter
has a strong track record in standing behind projects that were
contributed to the ASF by its employees, including Apache Mesos,
Apache Aurora, Apache BookKeeper, Apache Hadoop. There are many
companies are interested in using it in production.

== Inexperience with Open Source ==
The core developers of DistributedLog are committers of Apache
BookKeeper. Although other committers on the initial list are
committers or have less experience with the ASF, they already are
active in Apache BookKeeper community. We are confident that the
project can be run in accordance with Apache principles on an ongoing

== Homogeneous Developers ==
The initial committers are from Twitter. We hope to encourage
contributions from other developers and grow them into committers
after they have had time to continue their contributions.

== Reliance on Salaried Developers ==
Many of DistributedLog’s initial set of committers work full-time on
DistributedLog, and are paid to do so. However, as mentioned
elsewhere, we anticipate growth in the developer community which we
hope will include people from industry, hobbyists, and academics who
have an interested in distributed messaging systems.

== Relationships with Other Apache Products ==
DistributedLog uses Apache BookKeeper to store log segments and Apache
ZooKeeper to store log metadata and manage log namespaces. It provides
an end-to-end solution for replicated logs, to make building reliable
distributed systems much easier. Unlike Kafka or ActiveMQ,
DistributedLog is not a full-fledged pub/sub, queuing or messaging
system.  Instead, it is targeting on providing a fundamental building
block for other distributed systems, offering durability, replication
and consistency. So it could be used by other distributed systems,
such as transaction log for replicated state machines (e.g., HDFS
NameNode), WAL for distributed databases (e.g. HBase), Journal for
in-memory services (e.g., Kestrel) and even storage backend for a
full-fledged messaging system.

== An Excessive Fascination with the Apache Brand ==
DistributedLog builds on two existing top-level projects, Apache
BookKeeper and Apache ZooKeeper. Some of the core developers actively
participate in both projects and understand well the implications of
being hosted by Apache. We would like this project to build on the
same core values of ASF and to grow a community based on meritocracy.
Also, there are several other projects already hosted by ASF in this
space of reliable messaging and that overlap with DistributedLog in
interests and scope. Consequently, the combination of all these
observations makes us believe that DistributedLog should be hosted by
the ASF.

= Documentation =
Building DistributedLog: Twitter’s high performance replicated log
service (

Documentation located in

= Initial Source =
DistributedLog’s initial source contribution will come from

= External Dependencies =
DistributedLog depends upon a number of third-party libraries, which
we list below.
 * Apache BookKeeper (Apache Software License v2.0)
 * Apache Commons (Apache Software License v2.0)
 * Apache Maven (Apache Software License v2.0)
 * Apache Thrift (Apache Software License v2.0)
 * Apache ZooKeeper (Apache Software License v2.0)
 * Google Guava (Apache Software License v2.0)
 * Mockito (MIT License)
 * Junit (Eclipse Public License 1.0)
 * LZ4-java (Apache Software License v2.0)
 * SLF4J (MIT License)
 * Twitter Finagle (Apache Software License v2.0)
 * Twitter Scrooge (Apache Software License v2.0)
 * Twitter Util (Apache Software License v2.0)

= Required Resources =
We request that following resources be created for the project to use:

== Mailing lists ==
 * (moderated subscriptions)

== Git repository ==

== JIRA instance ==
JIRA project DLOG (DLOG or DL)

= Initial Committers =
 * Sijie Guo (Apache BookKeeper Committer, Twitter)
 * Robin Dhamankar (Apache BookKeeper Committer)
 * Leigh Stewart (Twitter)
 * Dave Rusek (Twitter)
 * Honggang Zhang (Twitter)
 * Jordan Bull (Twitter)
 * Satish Kotha (Twitter)
 * Aniruddha Laud
 * Franck Cuny (Twitter)
 * Eitan Adler (Twitter)

== Affiliations ==

Most of the initial committers are employees of Twitter, except Robin
Dhamankar and Aniruddha Laud.

= Sponsors =

== Champion ==

Flavio Junqueira

== Nominated Mentors ==

 * Flavio Junqueira
 * Chris Nauroth
 * Henry Saputra

= Sponsoring Entity =

We ask that the Apache Incubator PMC to sponsor this proposal.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message