Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 41834200C3D for ; Tue, 28 Feb 2017 00:55:24 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 3FDE9160B6C; Mon, 27 Feb 2017 23:55:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 13459160B60 for ; Tue, 28 Feb 2017 00:55:22 +0100 (CET) Received: (qmail 47037 invoked by uid 500); 27 Feb 2017 23:55:22 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 47026 invoked by uid 99); 27 Feb 2017 23:55:21 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Feb 2017 23:55:21 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id B41D2DFDCF; Mon, 27 Feb 2017 23:55:21 +0000 (UTC) From: sharad-develop To: dev@kafka.apache.org Reply-To: dev@kafka.apache.org Message-ID: Subject: [GitHub] kafka pull request #2605: Kafka 4738:Remove generic type of class ClientStat... Content-Type: text/plain Date: Mon, 27 Feb 2017 23:55:21 +0000 (UTC) archived-at: Mon, 27 Feb 2017 23:55:24 -0000 GitHub user sharad-develop opened a pull request: https://github.com/apache/kafka/pull/2605 Kafka 4738:Remove generic type of class ClientState Remove generic type of class ClientState. Also removed Generic type from TaskAssignor as it referred to same generic type of ClientState You can merge this pull request into a Git repository by running: $ git pull https://github.com/sharad-develop/kafka KAFKA-4738 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/2605.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2605 ---- commit 7946b2f35c0e3811ba7d706fc9c761d73ece47bb Author: Damian Guy Date: 2017-01-16T19:40:47Z MINOR: Remove unused constructor param from ProcessorStateManager Remove applicationId parameter as it is no longer used. Author: Damian Guy Reviewers: Guozhang Wang Closes #2385 from dguy/minor-remove-unused-param commit 621dff22e79dc64b9a8748186dd985774044f91a Author: Rajini Sivaram Date: 2017-01-17T11:16:29Z KAFKA-4363; Documentation for sasl.jaas.config property Author: Rajini Sivaram Reviewers: Ismael Juma Closes #2316 from rajinisivaram/KAFKA-4363 commit e3f4cdd0e249f78a7f4e8f064533bcd15eb11cbf Author: Rajini Sivaram Date: 2017-01-17T12:55:07Z KAFKA-4590; SASL/SCRAM system tests Runs sanity test and one replication test using SASL/SCRAM. Author: Rajini Sivaram Reviewers: Ewen Cheslack-Postava , Ismael Juma Closes #2355 from rajinisivaram/KAFKA-4590 commit 2b19ad9d8c47fb0f78a6e90d2f5711df6110bf1f Author: Rajini Sivaram Date: 2017-01-17T18:42:55Z KAFKA-4580; Use sasl.jaas.config for some system tests Switched console_consumer, verifiable_consumer and verifiable_producer to use new sasl.jaas_config property instead of static JAAS configuration file when used with SASL_PLAINTEXT. Author: Rajini Sivaram Reviewers: Ewen Cheslack-Postava , Ismael Juma Closes #2323 from rajinisivaram/KAFKA-4580 (cherry picked from commit 3f6c4f63c9c17424cf717ca76c74554bcf3b2e9a) Signed-off-by: Ismael Juma commit 60d759a227087079e6fd270c68fd9e38441cb34a Author: Jason Gustafson Date: 2017-01-17T18:42:05Z MINOR: Some cleanups and additional testing for KIP-88 Author: Jason Gustafson Reviewers: Vahid Hashemian , Ismael Juma Closes #2383 from hachikuji/minor-cleanup-kip-88 commit c9b9acf6a8b5433332c2d0d825c17a4a20cf3fa5 Author: Damian Guy Date: 2017-01-17T20:33:11Z KAFKA-4588: Wait for topics to be created in QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable After debugging this i can see the times that it fails there is a race between when the topic is actually created/ready on the broker and when the assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets called with a `Cluster` with no topics. Hence the test hangs as no tasks get assigned. To fix this I added a `waitForTopics` method to `EmbeddedKafkaCluster`. This will wait until the topics have been created. Author: Damian Guy Reviewers: Matthias J. Sax, Guozhang Wang Closes #2371 from dguy/integration-test-fix (cherry picked from commit 825f225bc5706b16af8ec44ca47ee1452c11e6f3) Signed-off-by: Guozhang Wang commit 6f72a5a53c444278187fa6be58031168bcaffb26 Author: Damian Guy Date: 2017-01-17T22:13:46Z KAFKA-3452 Follow-up: Refactoring StateStore hierarchies This is a follow up of https://github.com/apache/kafka/pull/2166 - refactoring the store hierarchies as requested Author: Damian Guy Reviewers: Guozhang Wang Closes #2360 from dguy/state-store-refactor (cherry picked from commit 73b7ae0019d387407375f3865e263225c986a6ce) Signed-off-by: Guozhang Wang commit eb62e5695506ae13bd37102c3c08e8a067eca0c8 Author: Ismael Juma Date: 2017-01-18T02:43:10Z KAFKA-4591; Create Topic Policy follow-up 1. Added javadoc to public classes 2. Removed `s` from config name for consistency with interface name 3. The policy interface now implements Configurable and AutoCloseable as per the KIP 4. Use `null` instead of `-1` in `RequestMetadata` 5. Perform all broker validation before invoking the policy 6. Add tests Author: Ismael Juma Reviewers: Jason Gustafson Closes #2388 from ijuma/create-topic-policy-docs-and-config-name-change (cherry picked from commit fd6d7bcf335166a524dc9a29a50c96af8f1c1c02) Signed-off-by: Ismael Juma commit e38794e020951adec5a5d0bbfe42c57294bf67bd Author: Guozhang Wang Date: 2017-01-18T04:29:55Z KAFKA-3502; move RocksDB options construction to init() In RocksDBStore, options / wOptions / fOptions are constructed in the constructor, which needs to be dismissed in the close() call; however in some tests, the generated topology is not initialized at all, and hence the corresponding state stores are supposed to not be able to be closed as well since their `init` function is not called. This could cause the above option objects to be not released. This is fixed in this patch to move the logic out of constructor and inside `init` functions, so that no RocksDB objects will be created in the constructor only. Also some minor cleanups: 1. In KStreamTestDriver.close(), we lost the logic to close the state stores but only call `flush`; it is now changed back to call both. 2. Moved the forwarding logic from KStreamTestDriver to MockProcessorContext to remove the mutual dependency: these functions should really be in ProcessorContext, not the test driver. Author: Guozhang Wang Reviewers: Damian Guy , Matthias J. Sax , Jason Gustafson Closes #2381 from guozhangwang/K3502-pure-virtual-function-unit-tests (cherry picked from commit 1974e1b0e54abe5fdebd8ff3338df864b7ab60f3) Signed-off-by: Jason Gustafson commit 919548ba0e1d0e231008c381f824f91a8924dc6a Author: Rajini Sivaram Date: 2017-01-18T11:05:58Z KAFKA-4568; Simplify test code for multiple SASL mechanisms Remove workaround for testing multiple SASL mechanisms using sasl.jaas.config and the new support for multiple client modules within a JVM. Author: Rajini Sivaram Reviewers: Sriharsha Chintalapani , Ismael Juma Closes #2373 from rajinisivaram/KAFKA-4568 (cherry picked from commit 4c492975a05e4a646f7140c560ed403c48925331) Signed-off-by: Ismael Juma commit 9b751f0c54e0409380632c4ee4618d78daca87d5 Author: Damian Guy Date: 2017-01-18T15:17:05Z MINOR: Fix javadoc typos in KStream#process interface for `Processor` in comments incorrectly had `transform` rather than `process`. Author: Damian Guy Reviewers: Michael G. Noll, Ismael Juma Closes #2396 from dguy/minor-javadoc (cherry picked from commit ccb183f9fca20a37b9b39761dedcd197e3cf2033) Signed-off-by: Ismael Juma commit 638a82b7bbf5b375f951e5827983753e9771dda0 Author: Damian Guy Date: 2017-01-18T17:19:06Z HOTFIX: ChangeLoggingKeyValueStore.name() returns null This class doesn't need to override this method as it is handled appropriately by the super class Author: Damian Guy Reviewers: Guozhang Wang Closes #2397 from dguy/hotfix-npe-state-store (cherry picked from commit 961ebca57fdd4f8f84df8cf26a835f07ea1718c9) Signed-off-by: Guozhang Wang commit dbca4a3b60d6039558c6fec99c86400c2131f9e8 Author: Matthias J. Sax Date: 2017-01-18T19:55:23Z MINOR: refactor streams system test class hierachy Author: Matthias J. Sax Reviewers: Eno Thereska, Guozhang Wang Closes #2392 from mjsax/minor-system-test-rework (cherry picked from commit d8a77560c2fa2c209353e3ba2366ad3d4cfdf22c) Signed-off-by: Guozhang Wang commit 866b33c16496ab654d59e69cb3e83587c52da673 Author: Colin P. Mccabe Date: 2017-01-18T22:45:57Z KAFKA-4457; Add BrokerApiVersionsCommand Author: Colin P. Mccabe Reviewers: Ewen Cheslack-Postava , Apurva Mehta , Ismael Juma Closes #2184 from cmccabe/KAFKA-4457 (cherry picked from commit 4a6f2c6cc0647a08f016a2d712a01ec02630cf87) Signed-off-by: Ismael Juma commit 5562d63e8a02fdddaabe066437fe7674d28d5d01 Author: Hojjat Jafarpour Date: 2017-01-18T23:53:17Z KAFKA-4060; Follow-up: Throw exceptions when internal topics to create already exist with unexpected number of partitions Re-branched the trunk and applied the changes to the new branch to simplify commit log. Author: Hojjat Jafarpour Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang Closes #2389 from hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-followup-from-trunk Address Ismael's comments upon merging (cherry picked from commit 8e2cbae8b12cd71f99f13953b744163f68021323) Signed-off-by: Guozhang Wang commit ae5591a3af45832d9a50e2aaa31b11c039d8d389 Author: Rajini Sivaram Date: 2017-01-19T16:35:39Z KAFKA-4589; SASL/SCRAM documentation Author: Rajini Sivaram Reviewers: Ismael Juma , Gwen Shapira , Sriharsha Chintalapani , Jun Rao Closes #2369 from rajinisivaram/KAFKA-4589 (cherry picked from commit 666abafcc54f8cab64912355dba4c8ada8e44827) Signed-off-by: Jun Rao commit 770bab39916e213443aff8f696c6097b551b9408 Author: Yuto Kawamura Date: 2017-01-19T17:47:34Z KAFKA-4614; Forcefully unmap mmap of OffsetIndex to prevent long GC pause Issue: https://issues.apache.org/jira/browse/KAFKA-4614 Fixes the problem that the broker threads suffered by long GC pause. When GC thread collects mmap objects which were created for index files, it unmaps memory mapping so kernel turns to delete a file physically. This work may transparently read file's metadata from physical disk if it's not available on cache. This seems to happen typically when we're using G1GC, due to it's strategy to left a garbage for a long time if other objects in the same region are still alive. See the link for the details. Author: Yuto Kawamura Reviewers: Apurva Mehta , Guozhang Wang , Ismael Juma , Closes #2352 from kawamuray/KAFKA-4614-force-munmap-for-index (cherry picked from commit 5fc530bc483db145e0cba3b63a57d6d6a7c547f2) Signed-off-by: Jun Rao commit 17dd4db29fe7f930ca1c5d0fb5687a48c364f2d7 Author: Vahid Hashemian Date: 2017-01-20T00:35:12Z KAFKA-4547; Avoid incorrect reset of paused partitions to the committed offsets Author: Vahid Hashemian Reviewers: Jason Gustafson Closes #2341 from vahidhashemian/KAFKA-4547 commit 3e1abac4c7d298d47567d23d6390eae4cd9ac1f8 Author: Sandesh K Date: 2017-01-20T01:46:59Z KAFKA-4432; Added support to supply custom message payloads to perf-producer script. Current implementation of ProducerPerformance creates static payload. This is not very useful in testing compression or when you want to test with production/custom payloads. So, we decided to add support for providing payload file as an input to producer perf test script. We made the following changes: 1. Added support to provide a payload file which can have the list of payloads that you actually want to send. 2. Moved payload generation inside the send loop for cases when payload file is provided. Following are the changes to how the producer-performance is evoked: 1. You must provide "--record-size" or "--payload-file" but not both. This is because, record size cannot be guaranteed when you are using custom events. e.g. ./kafka-producer-perf-test.sh --topic test_topic --num-records 100000 --producer-props bootstrap.servers=127.0.0.1:9092 acks=0 buffer.memory=33554432 compression.type=gzip batch.size=10240 linger.ms=10 --throughput -1 --payload-file ./test_payloads --payload-delimiter , 2. Earlier "--record-size" was a required config, now you must provide exactly one of "--record-size" or "--payload-file". Providing both will result in an error. 3. Support for an additional parameter "--payload-delimiter" has been added which defaults to "\n" Author: Sandesh K Reviewers: dan norwood , Jun Rao Closes #2158 from SandeshKarkera/PerfProducerChanges (cherry picked from commit a37bf5fffa4a607ded14ec055276d76f19df2d50) Signed-off-by: Jun Rao commit 05237367d7a12f395c9b2a1c2b354e1564f21aae Author: Xavier Léauté Date: 2017-01-20T19:07:57Z MINOR: reduce verbosity of cache flushes This log message tends to be extremely verbose when state stores are being restored Author: Xavier Léauté Reviewers: Guozhang Wang Closes #2412 from xvrl/reduce-verbosity (cherry picked from commit 50e035dc341bb595decbac0851354a10c90b0e06) Signed-off-by: Guozhang Wang commit c3f923cbf9f686bf42adda44d34a975167a13159 Author: Shikhar Bhushan Date: 2017-01-21T00:15:15Z KAFKA-3209: KIP-66: more single message transforms Renames `HoistToStruct` SMT to `HoistField`. Adds the following SMTs: `ExtractField` `MaskField` `RegexRouter` `ReplaceField` `SetSchemaMetadata` `ValueToKey` Adds HTML doc generation and updates to `connect.html`. Author: Shikhar Bhushan Reviewers: Ewen Cheslack-Postava Closes #2374 from shikhar/more-smt commit db57c1a0c7e057e19ec165d3cf9fa35d54b97c8f Author: Matthias J. Sax Date: 2017-01-21T21:18:39Z KAFKA-4060 and KAFKA-4476 follow up: remove unnecessary InternalTopicManager branch and fixed one copartitioning bug ZK removed reveal a bug in `StreamPartitionAssigner` but did not fix it properly. This is a follow up bug fix. Issue: - If topic metadata is missing, `StreamPartitionAssigner` should not create any affected tasks that consume topics with missing metadata. - Depending downstream tasks should not be create either. - For tasks that are not created, no store changelog topics (if any) should get created - For tasks that write output to not-yet existing internal repartitioning topics, those repartitioning topics should not get created Author: Matthias J. Sax Reviewers: Damian Guy, Guozhang Wang Closes #2404 from mjsax/kafka-4060-zk-test-follow-up (cherry picked from commit 0b99bea590842018e8e97e7fd1c71b1471db4d08) Signed-off-by: Guozhang Wang commit e393c9928972c00677250ae2df3b4f061fdccd58 Author: Xavier Léauté Date: 2017-01-21T22:10:01Z KAFKA-4672; fix source compatibility for lambda expressions Variance changes introduced in KIP-100 cause compilation failures with lambda expression in Java 8. To my knowledge this only affects the following method `KStreams.transform(TransformerSupplier<...>, String...)` prior to the changes it was possible to write: `streams.transform(MyTransformer::new)` where `MyTransformer` extends `Transformer` After the changes the Java compiler is unable to infer correct return types for the lambda expressions. This change fixed this by reverting to invariant return types for transformer suppliers. please cherry-pick into 0.10.2.x Author: Xavier Léauté Reviewers: Ismael Juma, Damian Guy, Guozhang Wang Closes #2402 from xvrl/lambdas-oh-my (cherry picked from commit 20e957c9b1f389621bb8ae9e9f5554b66cd92a4f) Signed-off-by: Guozhang Wang commit f0449d3248801facca442d93552a829e8b8d06a7 Author: Eno Thereska Date: 2017-01-23T14:45:30Z MINOR: Pass RecordingLevel to MetricConfig in the broker This is a KIP-104/105 follow-up. Thanks to ijuma for pointing out. Author: Eno Thereska Reviewers: Ismael Juma Closes #2350 from enothereska/minor-broker-level-config (cherry picked from commit 1eb1e2f60ae006144757fc9fc10ab423c58970bb) Signed-off-by: Ismael Juma commit afde87804211dfea2b9ee87c5f6470e8460dcd33 Author: Matthias J. Sax Date: 2017-01-23T17:34:52Z HOTFIX: KAFKA-4060 and KAFKA-4476 follow up Author: Matthias J. Sax Reviewers: Damian Guy, Guozhang Wang Closes #2418 from mjsax/kafka-4060-zk-test-follow-up (cherry picked from commit dd897bdb2e9cc8790d1e0494fb7867a5cd09ecc6) Signed-off-by: Guozhang Wang commit 80ceb75d2acf5c3d8c21e03bf55e706c637d9fd9 Author: Hojjat Jafarpour Date: 2017-01-23T18:21:06Z KAFKA-4060 Follow-up: update docs accordingly Updated the docs with changes in KAFKA-4060. Author: Hojjat Jafarpour Author: Hojjat Jafarpour Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang Closes #2377 from hjafarpour/KAFKA-4060-docs-update (cherry picked from commit 62206de4e52dbec32b99baffa0ebf7f5303d7b7f) Signed-off-by: Guozhang Wang commit 0f87991d5dbe8741953be04628810d76a9ee7434 Author: Guozhang Wang Date: 2017-01-23T18:45:45Z KAFKA-4633; Always using regex pattern subscription in StreamThread 1. In StreamThread, always use subscribe(Pattern, ..) function in order to avoid sending MetadataRequest with specific topic names and cause brokers to possibly auto-create subscribed topics; the pattern is generated as "topic-1|topic-2..|topic-n". 2. In ConsumerCoordinator, let the leader to refresh its metadata if the generated assignment contains some topics that is not contained in the subscribed topics; also in SubscriptionState, modified the verification for regex subscription to against the regex pattern instead of the matched topics since the returned assignment may contain some topics not yet created when joining the group but existed after the rebalance; also modified some unit tests in `KafkaConsumerTest` to accommodate the above changes. 3. Minor cleanup: changed String[] to List to avoid overloaded functions. 4. Minor cleanup: enforced strong typing in SinkNodeFactory and removed unnecessary unchecked tags. 5. Minor cleanup: augmented unit test error message and fixed a potential transient failure in KafkaStreamTest. Author: Guozhang Wang Reviewers: Damian Guy , Matthias J. Sax , Jason Gustafson Closes #2379 from guozhangwang/K4633-regex-pattern (cherry picked from commit 3400d0c3cc07d5d3136bf9a19142b36cca93f92d) Signed-off-by: Jason Gustafson commit 4218c02478a213935afbb2c2d661d8a29a5f29ee Author: Matthias J. Sax Date: 2017-01-24T00:31:36Z KAFKA-4671: Fix Streams window retention policy Author: Matthias J. Sax Reviewers: Damian Guy, Eno Thereska, Guozhang Wang Closes #2401 from mjsax/kafka-4671-window-retention-policy (cherry picked from commit 79987590e3e96351ff75ce86718801ec605b2419) Signed-off-by: Guozhang Wang commit 928f69ce883c8e30050d2e6f5e72e77e6740bdb9 Author: huxi Date: 2017-01-24T10:04:42Z KAFKA-4576; Log segments close to max size break on fetch `FileChannel.read` may not fill the destination buffer even if there are enough bytes in the channel to do so. Add a couple of utility methods that ensure this and use them from all the relevant places. Author: huxi Author: amethystic Author: Ismael Juma Reviewers: Jun Rao , Jason Gustafson , Ismael Juma Closes #2304 from amethystic/kafka4576_FileChannel_read (cherry picked from commit 337f576f5979bf8924c5707b338cf4d3c76a53fd) Signed-off-by: Ismael Juma commit b676f51dd54125b950fee35fe867057778866bcb Author: Colin P. Mccabe Date: 2017-01-24T11:09:47Z KAFKA-4688; Kafka 0.10.1.1 should be available in system tests Author: Colin P. Mccabe Reviewers: Ewen Cheslack-Postava , Ismael Juma Closes #2424 from cmccabe/KAFKA-4688 (cherry picked from commit 567180605c0a8cf74b8d2cad52e2873c83065a95) Signed-off-by: Ismael Juma ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastructure@apache.org or file a JIRA ticket with INFRA. ---