bookkeeper-distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sijie <>
Subject [GitHub] incubator-distributedlog pull request #92: DL-115 ~ DL-164
Date Thu, 29 Dec 2016 17:09:04 GMT
GitHub user sijie opened a pull request:

    DL-115 ~ DL-164

    This is cherry-pick merge twitter branch's changes for DL-115 ~ DL-164. Individual commit
is associated with a DL jira.

You can merge this pull request into a Git repository by running:

    $ git pull sijie/squash_merge_twitter_changes

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #92
commit 00be3e58d9a4a4e48989c8e45825a0c4911f9cdc
Author: Yiming Zang <>
Date:   2016-11-16T22:14:52Z

    DL-115: fix force get log segment logic
    We should remove the ledger closed check:
    - force to get log segments if the reader has been idle for a while after open the ledger
    - idleReaderWarnThreshold should be 2x larger than readLACLongPollTimeout

commit 9d467a60a8c2ad2c0acc73afbfb66c232baa0b46
Author: Yiming Zang <>
Date:   2016-11-30T22:16:14Z

    DL-116: Add tool for deleting subscriber from subscription store
    Test Plan:
    1. manually create znode for subscribers resume point
    2. use the tool to delete the subscriberId

commit b91d49a89e7c9133eb2807ced2e7dbb8aba72e02
Author: Sijie Guo <>
Date:   2016-12-01T01:14:05Z

    DL-117: Stream metadata store
    This change is to abstract the zookeeper operations into a stream metadata store, so we
can replace zookeeper with other metadata store easily.
    So the metadata operations in distributedlog now are managed by 3 classes:
    - LogMetadataStore : it is the namespace metadata store : it manages the location (uri)
mapping for streams and handle namespace operations.
    - LogStreamMetadataStore: it is the stream metadata store : it manages the metadata for
a single stream, such as managing read/write lock, retriving/creating stream metadata, deleting
metadata and such.
    - LogSegmentMetadataStore: it is the segment metadata store : it manages the log segment
metadata for individual log segment.
    LogMetadataStore and LogSegmentMetadataStore are already there. This change focus on LogStreamMetadataStore
    * abstract all the zookeeper metadata operation in log handlers to LogStreamMetadataStore
    * remove disabling max tx id santify check, as maxTxId update is part of the metadata
update transaction
    Not changed:
    the name of ZKLogMetadataForReader and ZKLogMetadataForWriter are not changed. I will
send out a change to rename these two classes as they are not related to zookeeper anymore.

commit dc4548bebcd8c8ce680fb14cfaf203dd6605fa29
Author: Sijie Guo <>
Date:   2016-12-01T02:17:46Z

    DL-118: Stream metadata store followup - rename ZKLogMetadata* to LogMetadata*
    As the followup change, this change is to remove 'ZK' from 'ZKLogMetadata*' class

commit 34fa16b1d7ab1ae887b1114e8e2aabdffe16608a
Author: Yiming Zang <>
Date:   2016-12-01T02:21:08Z

    DL-119: Fix the logging on closing readahead worker

commit 0591d067f05617bc534a662b6f9a014192cbe3a5
Author: Jordan Bull <>
Date:   2016-12-13T19:11:03Z

    DL-157: resource placement for write proxy

commit 98a29a5807fd7d42d2bb765be84527d4b2fb5dbc
Author: Yiming Zang <>
Date:   2016-12-28T22:49:50Z

    DL-158: update truncation status for all completed log segments
    update turncation status for all completed log segments, when purge truncated log segments
we will leave one complete log segment.

commit 27c04f37b1b564a06d9105acb1849a6d98449ca9
Author: Sijie Guo <>
Date:   2016-12-28T22:51:54Z

    DL-195: ReadAhead Improvement (part 1) - Interface for LogSegmentEntryReader and LogSegmentEntryWriter
    Create interface for log segment entry reader and writer to abstract data write/read operations.

commit 7a977972200da9e86f4557bb002ce16cb60d236a
Author: Sijie Guo <>
Date:   2016-12-28T23:09:38Z

    DL-159: ReadAhead Improvement (part 2) - New ReadAhead Reader using the LogSegmentEntryReader
    Provide a new ReadAhead reader using the log segment entry reader interface. It does reading
entries in a batch in parallel for batches, rather than reading entries in batch by batch.
This would help mitigate the slow bookie problem.
    The core change is the new ReadAheadEntryReader.

commit c7751804ef728c39d54b19e52f2db48d7fba9f65
Author: Sijie Guo <>
Date:   2016-12-29T00:27:38Z

    DL-160: Remove 'DLSN' suffix from async and sync readers

commit 47622a6e45a34492ec50fcae79fb052dba06c6a4
Author: Sijie Guo <>
Date:   2016-12-29T00:48:47Z

    DL-161: Rename ledger id to log segment id
    Along with providing log segment store related interfaces, clean up the naming from 'ledger
id' to 'log segment id'. So it would become clear when we can plugin different log segment

commit d871e6570fe49cab56fcec37d3a8a6bd91afbe4d
Author: Sijie Guo <>
Date:   2016-12-29T01:09:57Z

    DL-162: Use log segment entry store interface
    - Use log segment entry store interface
    - Delete the old readahead implementation

commit cfc049cd045b73ca96fb328877b0acf4e75c95ab
Author: Sijie Guo <>
Date:   2016-12-29T01:47:12Z

    DL-163: clean up direct zookeeper and bookkeeper usage and use metadata/data store abstraction
    - introduce NamespaceDriver class to manage metadata/data store abstractions
    - clean up direct zookeeper and bookkeeper usage and use metadata/data store abstraction
    - separate MetadataAccessor from DistributedLogManager
    - remove deprecated classes

commit 63d6bde1929085780171e63cc5b0c95581daa564
Author: Sijie Guo <>
Date:   2016-12-29T03:09:59Z

    DL-164: Create stream operation should not be submitted by StreamImpl


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

View raw message