kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From benstopford <...@git.apache.org>
Subject [GitHub] kafka pull request #1776: KIP-73 - Replication Quotas
Date Tue, 23 Aug 2016 21:35:24 GMT
GitHub user benstopford opened a pull request:

    https://github.com/apache/kafka/pull/1776

    KIP-73 - Replication Quotas

    This applies to Replication Quotas 
    based on KIP-73 [()](https://cwiki.apache.org/confluence/display/KAFKA/KIP-73+Replication+Quotas)
originally motivated by KAFKA-1464.
    
    **This first PR demonstrates the approach**. Main stuff left to do:
    - The quota value should be moved to a dynamic, broker-level config. Here we are making
do with a client-level override.
    - The fetch size is currently unbounded. This will be addressed in KIP-74, but we need
to ensure this ensures requests don’t go beyond the throttle window.
    - There is no specific validation or other utilities in ConfigCommand or code in ReassignPartitionsCommand
for validation etc.
    - Tests are largely happy path, the functional tests rely on a single broker-to-broker
set up. There are no system tests. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/benstopford/kafka rep-quotas-v2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1776.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1776
    
----
commit d4964b8905cef3d1e11b20a3ed8964643e214ad3
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-02T22:44:06Z

    CPKAFKA-291: Port of original POC, which used seperate replica fetcher threads.

commit a3d1a5590df491468a42a481faaa89c7951114d4
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-10T15:13:34Z

    CPKAFKA-296: MINOR: Final checkin .. includes failing test demonstrating the current linearisibility
issue

commit 300ecc43db4bda3adc5ee47e505324a4b16b15df
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-11T15:59:35Z

    CPKAFKA-305: POC of leader, bytes based throttling. This commit includes the rollback
of the previous work.

commit 3fc0444e9d77c65449c08a6c7ac482da09ebc431
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-11T16:23:08Z

    CPKAFKA-305: MINOR: rename and tidy only

commit 0c542440d41d1924a4c8d16f692cd814bcb1a460
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-11T16:23:33Z

    CPKAFKA-305: MINOR: rename and tidy only

commit 627905b4f4083dcc7ec46b1e4d25d50952f294ad
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-11T18:25:00Z

    CPKAFKA-305: Completed POC of follower throttling. Quick and Dirty.

commit 5f4c6aeb30f538b23e8aa53d5783386491648319
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-12T10:25:18Z

    CPKAFKA-305: Added test shouldReplicateThrottledAndNonThrottledPartitionsConcurrently

commit 239d6901d45518b448c90ccf9b59f5d9d3481361
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-15T16:39:03Z

    CPKAFKA-305: Stabalised tests by replacing Rate with SimpleRate. This is likely a temporary
measure, but that depends on whether we want a graduated start as defined in the current rate
implementation. Altered purgatory check. Currently not using code for proposing rate changes,
instead we validate simple whether the quota is currently exceeded (boolean)

commit 86425e44d90194ef96f3d666035b9b0977f7e510
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-15T18:52:16Z

    CPKAFKA-305: Added small test around replication quota manager

commit e26c8621ae6c550b901c2f9533960675f03cde22
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-16T07:10:14Z

    CPKAFKA-305: Altered leader algorithm to evaluate bytes per partition

commit 434b1cc3e0326f8dbc6c153b7435b1f83e661747
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-16T15:17:57Z

    CPKAFKA-305: SReplicaManagerQuotasTest - added support for multiple (subsequent) partitions

commit a6cc2400defaec16308fd3ccfc05410069d42c0b
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-16T15:36:59Z

    CPKAFKA-305: Moved things around in ClientQuotaManager, fixed bug in ReplicationQuotaTest

commit 2d3ab68b57c205775b8a065ed09e35919aa90820
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-16T16:55:27Z

    CPKAFKA-305: Added a second partition to one of the acceptance tests. Fixed bug in logging
in KafkaApis

commit ab3d846a50a6589d4b860c8afb2bda695a2547e7
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-17T16:05:59Z

    CPKAFKA-305: small changes, nothing functional

commit b8a2fcd71a792f7ae06a7aa2993141f14d0554e4
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-17T20:44:12Z

    CPKAFKA-305: Extracted ReplicationQuotaManager into its own class, and associated changes.

commit 5cd27799b4e09ece2d11c22de9f6e13f09ad5c11
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-22T12:26:06Z

    CPKAFKA-305: Moved kafka-configs format to use 0-1:1-2 format (i.e. with colon as seperator)
plus extended test

commit 3e525c141c4702615dbac6177aaa3d0704ce2394
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-22T13:35:03Z

    CPKAFKA-305: Ensure the correct config is propagated

commit efb815fde2db6f1423d70ebb701636688625aee7
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-22T19:17:07Z

    CPKAFKA-305: Changed rate to be based on fixed windows which makes it easier to test.
Added some more tests for the ReplicationQuotaManager

commit 726bd34147a7d5caa30b017f742ddc5ff973618d
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T09:51:59Z

    CPKAFKA-305: Comments only

commit 62e2b1374b384dde71de9cca7a932572d264f898
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T09:56:07Z

    CPKAFKA-305: rolled back minor changes to ClientQuotaManager

commit 0934fdd9d811fe97c941e2865b4bb23304388ccf
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T15:09:25Z

    CPKAFKA-305: Fixed bug in ReplicationQuotaManagerTest. Reverted back to SimpleRate as
it gives a better match on the rate in tests (although makes less difference long term)

commit 5e5948f58667cdbd6f195c200e612797ab574410
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T16:25:13Z

    CPKAFKA-305: roll back gradle commit (accidental)

commit cfd5a566340c97f63523ae784d9aa85b9457ac52
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T16:25:57Z

    CPKAFKA-305: roll back gradle commit (accidental)

commit 7fb65b942c66542e9ef3110cd8383d53310a2e8e
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T17:03:05Z

    CPKAFKA-305: Replaced the FetchQuota class with a Null Object pattern.

commit 9398889d151a84e451224dea20bb448083a29928
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T17:24:24Z

    CPKAFKA-305: logging only

commit 66f3f39a438af3589dad5ec04d27ad996b79a1c5
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T18:30:21Z

    CPKAFKA-305: Tidy

commit 40c5a5f446d5bd8045b653c362e3ad3d83fef8b7
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T18:33:02Z

    CPKAFKA-305: Rename only

commit c91bc40602dc6a31340d20dd558a6dd0191e19bc
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T18:48:26Z

    CPKAFKA-305: Rename only

commit bc3b460fe4d1f1dd9ca65b8bba424205d2494456
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T19:19:13Z

    CPKAFKA-305: small changes

commit 834060b6f64cf97c2e4dd54221c9e28ef10f370a
Author: Ben Stopford <benstopford@gmail.com>
Date:   2016-08-23T19:22:31Z

    CPKAFKA-305: licenses

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message