incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Homan <jgho...@gmail.com>
Subject Re: [VOTE] Accept DistributedLog into the Apache Incubator
Date Wed, 22 Jun 2016 17:25:54 GMT
+1 (binding)
-jg


On 22 June 2016 at 09:45, Suneel Marthi <smarthi@apache.org> wrote:
> +1 (binding)
>
> On Tue, Jun 21, 2016 at 1:11 AM, Sijie Guo <sijie@apache.org> wrote:
>
>> Hello All,
>>
>> Following the discussion thread, I would like to call a VOTE on accepting
>> DistributedLog into the Apache Incubator.
>>
>> [] +1 Accept DistributedLog into the Apache Incubator
>> [] +0 Abstain.
>> [] -1 Do not accept DistributedLog into the Apache Incubator because ...
>>
>> This vote will be open for at least 72 hours.
>>
>> The proposal follows, you can also access the wiki page:
>> https://wiki.apache.org/incubator/DistributedLogProposal
>>
>> Here is my +1.
>>
>> Thanks,
>> Sijie
>>
>> = Abstract =
>> DistributedLog is a high-performance replicated log service. It offers
>> durability, replication and strong consistency, which provides a
>> fundamental building block for building reliable distributed systems, e.g
>> replicated-state-machines, general pub/sub systems, distributed databases,
>> distributed queues and etc.
>>
>> See “Building Distributedlog - Twitter’s high performance replicated log
>> service” for details:
>>
>> https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service
>>
>> = Proposal =
>> We propose to contribute DistributedLog codebase and associated artifacts
>> (e.g. documentation, web-site content etc.) to the Apache Software
>> Foundation with the intent of forming a productive, meritocratic and open
>> community around DistributedLog’s continued development, according to the
>> ‘Apache Way’.
>>
>> = Background =
>> Engineers at Twitter began developing DistributedLog in early 2013.
>> DistributedLog is described in a Twitter engineering blog post and
>> presented at the Messaging Meetup in Sep 2015. It has been released as an
>> Apache-licensed open-source project on GitHub in May 2016.
>>
>> DistributedLog is a high-performance replicated log service, which provides
>> simple stream-oriented abstractions over log-segments and offers
>> durability, replication and strong consistency for building reliable
>> distributed systems. The features offered by DistributedLog includes:
>>
>>  * Simple high-level, stream oriented interface
>>  * Naming and metadata scheme for managing streams and other entities
>>  * Log data management policies, include data segmentation and data
>> retention
>>  * Fast write pipeline leveraging batching and compression
>>  * Fast read mechanism leveraging long-poll and read-ahead caching
>>  * Service tiers supporting writer fan-in and reader fan-out
>>  * Geo-replicated logs
>>
>> DistributedLog’s most important benefit is high-performance with a strong
>> durability guarantee, making it extremely appropriate for running different
>> workloads from distributed database journaling to real-time stream
>> computing. Its modern, layered architecture makes it easy to run the
>> service tiers in multi-tenant datacenter environments such as Apache Mesos
>> or cloud environments such as EC2.
>>
>> = Rationale =
>> DistributedLog is designed to provide core fundamental features like
>> high-performance, durability and strong consistency to anyone who is
>> building reliable distributed systems, in a simple and efficient way.
>>
>> We believe that the ASF is the right venue to foster an open-source
>> community around DistributedLog’s development. We expect that
>> DistributedLog will benefit from collaboration with related Apache
>> projects, and under the auspices of the ASF will attract talented
>> contributors who will push DistributedLog’s development forward at a faster
>> pace.
>>
>> We believe that the timing is right for DistributedLog’s development to
>> move to the ASF: DistributedLog has already run in production at Twitter
>> for 3 years and served various workloads including a distributed database
>> journal, reliable cross datacenter replication, search ingestion,
>> andgeneral pub/sub messaging. The project is stable. We are excited to see
>> where an ASF-based community can take DistributedLog.
>>
>> = Current Status =
>> DistributedLog is a stable project that has been used in production at
>> Twitter for 3 years. The source code is public at github.com/twitter,
>> which
>> will seed the Apache git repository.
>>
>> = Meritocracy =
>> We understand the central importance of meritocracy to the Apache Way. We
>> will work to establish a welcoming, fair and meritocratic community.
>> Several companies have already expressed interest in this project, and we
>> intend to invite additional developers to participate. We look forward to
>> growing a rich user and developer community.
>>
>> = Community =
>> There is a large need for a performant replicated log service for
>> applications such as distributed databases, distributed transactional
>> systems, replicated-state-machines and pub/sub messaging/queuing. We want
>> to attract more developers to the project, and we believe that the ASF’s
>> open and meritocratic philosophy will help us with this. We note the
>> success of other similar projects already part of the ASF, like Kafka.
>>
>> = Core Developers =
>> DistributedLog is actively developed within Twitter. Most of the developers
>> are from Twitter. Many of them are committers or PMC members of Apache
>> BookKeeper. Others aren’t currently affiliated with ASF so they will
>> require new ICLAs.
>>
>> = Alignment =
>> DistributedLog is related to several other Apache projects:
>>
>>  * DistributedLog stores log segments as Ledgers in Apache BookKeeper.
>>  * DistributedLog uses Apache ZooKeeper for naming and metadata management
>> and tracking the ownership of logs.
>>  * DistributedLog uses Apache Thrift as its RPC and serialization
>> framework.
>>  * In the long-term, DistributedLog’s data will be stored in Apache Hadoop
>> clusters powered by HDFS filesystem for archives and backup.
>>
>> = Known Risks =
>> == Orphaned Products ==
>> DistributedLog is used as the fundamental messaging infrastructure at
>> Twitter. It has been serving production traffic for online database
>> systems, search ingestion and a general pub/sub system. Twitter remains
>> committed to developing and supporting the project. Twitter has a strong
>> track record in standing behind projects that were contributed to the ASF
>> by its employees, including Apache Mesos, Apache Aurora, Apache BookKeeper,
>> Apache Hadoop. There are many companies are interested in using it in
>> production.
>>
>> == Inexperience with Open Source ==
>> The core developers of DistributedLog are committers of Apache BookKeeper.
>> Although other committers on the initial list are committers or have less
>> experience with the ASF, they already are active in Apache BookKeeper
>> community. We are confident that the project can be run in accordance with
>> Apache principles on an ongoing basis.
>>
>> == Homogeneous Developers ==
>> The initial committers are from Twitter. We hope to encourage contributions
>> from other developers and grow them into committers after they have had
>> time to continue their contributions.
>>
>> == Reliance on Salaried Developers ==
>> Many of DistributedLog’s initial set of committers work full-time on
>> DistributedLog, and are paid to do so. However, as mentioned elsewhere, we
>> anticipate growth in the developer community which we hope will include
>> people from industry, hobbyists, and academics who have an interested in
>> distributed messaging systems.
>>
>> == Relationships with Other Apache Products ==
>> DistributedLog uses Apache BookKeeper to store log segments and Apache
>> ZooKeeper to store log metadata and manage log namespaces. It provides an
>> end-to-end solution for replicated logs, to make building reliable
>> distributed systems much easier. Unlike Kafka or ActiveMQ, DistributedLog
>> is not a full-fledged pub/sub, queuing or messaging system.  Instead, it is
>> targeting on providing a fundamental building block for other distributed
>> systems, offering durability, replication and consistency. So it could be
>> used by other distributed systems, such as transactional log for replicated
>> state machines (e.g., HDFS NameNode), WAL for distributed databases (e.g.
>> HBase), Journal for in-memory services (e.g., Kestrel) and even storage
>> backend for a full-fledged messaging system.
>>
>> == An Excessive Fascination with the Apache Brand ==
>> DistributedLog builds on two existing top-level projects, Apache BookKeeper
>> and Apache ZooKeeper. Some of the core developers actively participate in
>> both projects and understand well the implications of being hosted by
>> Apache. We would like this project to build on the same core values of ASF
>> and to grow a community based on meritocracy. Also, there are several other
>> projects already hosted by ASF in this space of reliable messaging and that
>> overlap with DistributedLog in interests and scope. Consequently, the
>> combination of all these observations makes us believe that DistributedLog
>> should be hosted by the ASF.
>>
>> = Documentation =
>> Building DistributedLog: Twitter’s high performance replicated log service
>> (
>>
>> https://blog.twitter.com/2015/building-distributedlog-twitter-s-high-performance-replicated-log-service
>> )
>>
>> Documentation located in http://distributedlog.io.
>>
>> = Initial Source =
>> DistributedLog’s initial source contribution will come from
>> http://github.com/twitter/distributedlog/.
>>
>> = External Dependencies =
>> DistributedLog depends upon a number of third-party libraries, which we
>> list below.
>>
>>  * Apache BookKeeper (Apache Software License v2.0)
>>  * Apache Commons (Apache Software License v2.0)
>>  * Apache Maven (Apache Software License v2.0)
>>  * Apache Thrift (Apache Software License v2.0)
>>  * Apache ZooKeeper (Apache Software License v2.0)
>>  * Google Guava (Apache Software License v2.0)
>>  * Mockito (MIT License)
>>  * Junit (Eclipse Public License 1.0)
>>  * LZ4-java (Apache Software License v2.0)
>>  * SLF4J (MIT License)
>>  * Twitter Finagle (Apache Software License v2.0)
>>  * Twitter Scrooge (Apache Software License v2.0)
>>  * Twitter Util (Apache Software License v2.0)
>>
>> = Required Resources =
>> We request that following resources be created for the project to use:
>>
>> == Mailing lists ==
>>  * private@distributedlog.incubator.apache.org (moderated subscriptions)
>>  * commits@distributedlog.incubator.apache.org
>>  * dev@distributedlog.incubator.apache.org
>>  * user@distributedlog.incubator.apache.org
>>
>> == Git repository ==
>> https://git.apache.org/distributedlog.git
>>
>> == JIRA instance ==
>> JIRA project DLOG (DLOG or DL)
>>
>> = Initial Committers =
>>  * Sijie Guo (Apache BookKeeper Committer, Twitter)
>>  * Robin Dhamankar (Apache BookKeeper Committer)
>>  * Leigh Stewart (Twitter)
>>  * Dave Rusek (Twitter)
>>  * Honggang Zhang (Twitter)
>>  * Jordan Bull (Twitter)
>>  * Satish Kotha (Twitter)
>>  * Aniruddha Laud
>>  * Franck Cuny (Twitter)
>>  * Eitan Adler (Twitter)
>>
>> == Affiliations ==
>> Most of the initial committers are employees of Twitter, except Robin
>> Dhamankar and Aniruddha Laud.
>>
>> = Sponsors =
>> == Champion ==
>> Flavio Junqueira
>>
>> == Nominated Mentors ==
>>  * Flavio Junqueira
>>  * Chris Nauroth
>>  * Henry Saputra
>>
>> = Sponsoring Entity =
>> We ask that the Apache Incubator PMC to sponsor this proposal.
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message