distributedlog-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sijie <...@git.apache.org>
Subject [GitHub] incubator-distributedlog pull request #82: DL-99 - DL-111
Date Wed, 21 Dec 2016 17:26:01 GMT
GitHub user sijie opened a pull request:


    DL-99 - DL-111

    This is a umbrella merge for DL-99 to DL-111. Individual commit in this pull request is
associated with one DL jira.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sijie/incubator-distributedlog squashed_merge/DL-111

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #82
commit 9ff22f4239754d25fb3c7238cf25cf453f0ebbd6
Author: Dave Rusek <drusek@apache.org>
Date:   2016-07-18T19:23:49Z

    DL-99: Enable thriftmux for distributedlog smoketest reader

commit 785ff1cdfb515882f934d4bd79fe34bb39cd95a9
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-07-29T04:10:33Z

    DL-101: Improve session expire handling on fetching log segments for readers
    This change focuses on improving the session expire handling on fetching log segments
for readers.
    The log segment management in DL is now done by 3 parts.
    - a LogSegmentMetadataStore (one per namespace instance): it is used for fetching the
log segments from log segment metadata store (ZooKeeper). it doesn't do any caching.
    - a LogSegmentMetadataCache (one per namespace instance): it is a guava cache based metadata
cache. it maintains a mapping between log segment metadata path and the log segment metadata.
it manages the cache for the log segments that will be accessed in this namespace instance.
it doesn't manage the sequence of the log segments for streams.
    - a PerStreamLogSegmentCache for each BKLogHandler. the log segment cache is per stream.
it maintains the sequence of the log segments.
    BKLogWriteHandler doesn't watch the log segment changes. It fetches minimal number of
log segments when it is created and fetches the full list of log segments for truncations.
New log segments will be added to the per stream log segment cache with log segment rolling.
    BKLogReadHandler watch the log segments changes and only notify when the list of log segments
is changed. the session handling which is specific to the metadata store is hidden to the
implementations of LogSegmentMetadataStore.
    The change tries to cleanup bunch of unused methods in BKLog{Read,Write}Handler too.

commit 10ff44510437d4087df5dc4775d968e174df83b0
Author: Leigh Stewart <lstewart@apache.org>
Date:   2016-07-29T04:36:49Z

    DL-102: Add routing service to write proxy server side
    this change is to add getOwner rpc in write proxy. so we can change the client side to
get owner from write proxy first for routing service. in this way, we can start experiementing
any resource placement algorithms.

commit 9f80812ab4293de2f7073642f7ef452b6a88f8f6
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-07-29T05:02:09Z

    DL-103: DL client can use server-side routing service to retrieve ownerships for streams
    If the dl client configures a routing service finagle name, the client will use this to
retrieve ownerships for streams.

commit 75774256dbaa5fcccb2e0d319e5d7d272a976901
Author: Jordan Bull <jbull@twitter.com>
Date:   2016-08-09T18:36:53Z

    DL-104: BKClientReadTimeout can be equal to getReadLACLongPollTimeout so current config

commit afe64639b871b4dbc564f904fc4f522c0de334d3
Author: Yiming Zang <yzang@twitter.com>
Date:   2016-08-10T23:18:53Z

    DL-105: Make compression stats available per stream

commit 36a07f6b1d5d3fde395687f97ee10fb9376b5d7b
Author: Yiming Zang <yzang@twitter.com>
Date:   2016-08-12T17:30:39Z

    DL-106: Use namespace after it is closed will throw AlreadyClosedException

commit d156c1d6223034fceba90750febc42d25e389d5c
Author: Phillip Su <psu@twitter.com>
Date:   2016-08-23T00:47:28Z

    DL-107: Added unregistering gauges for distributedlog-core and distributedlog-benchmark

commit e18cd48c2f9a9d03dfc7f46e96e74ebaa1471659
Author: Yiming Zang <yzang@twitter.com>
Date:   2016-09-17T00:35:51Z

    DL-108: Log rate limiting more clearly

commit a743b44f4614154c67cc4d6a98a9003d80062d70
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-10-27T17:18:52Z

    DL-109: Add a tool to find the stream containing the given ledger

commit 5f50a41b3883e2107befa95f774687deae9d62fe
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-11-22T00:17:51Z

    DL-110: Write control record if necessary when roll a log segment

commit 0d2e20c7c9189dec7909ae67f4a62443a9aa40bc
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-11-22T01:30:13Z

    DL-111: ReadAhead Cache should cache entries rather than records
    Current readahead cache cache records. So it will be a lot of callbacks (function calls)
when polling a record off the readahead cache. Most of the cpu cycles are unnecessarily spent
    on function calls on polling records off the readahead cache. It is the throughput bottleneck
for a DL reader.
    This change is to change ReadAhead cache to cache entries rather than records. Defer the
deserilization of records later on when the reader wants to access the records. It also make
    the cache more efficient to reduce the memory footprint.

commit 63b62709e982a8cdabfa0b15ae88c956e71606d0
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-11-22T01:35:40Z

    DL-112: Improve SyncReader to access readahead cache directly
    This changes the implementation of SyncReader to let it access the read ahead cache directly,
    than calling AsyncReader to fetch the entries. It would reduce the function calls that
SyncReader spends
    on reading individual records.

commit 27157ec59d30e3e519e5bdef9dfb157c5185ffea
Author: Sijie Guo <sijieg@twitter.com>
Date:   2016-11-22T01:41:26Z

    DL-113: Improve the sync reader benchmark
    Add more logging information for calcuating the stats for catching up reads


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message