Return-Path: X-Original-To: apmail-kafka-dev-archive@www.apache.org Delivered-To: apmail-kafka-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 66E861823A for ; Thu, 28 Jan 2016 19:22:40 +0000 (UTC) Received: (qmail 66715 invoked by uid 500); 28 Jan 2016 19:22:40 -0000 Delivered-To: apmail-kafka-dev-archive@kafka.apache.org Received: (qmail 66610 invoked by uid 500); 28 Jan 2016 19:22:40 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 66595 invoked by uid 99); 28 Jan 2016 19:22:40 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2016 19:22:40 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id D5A6C2C1F58 for ; Thu, 28 Jan 2016 19:22:39 +0000 (UTC) Date: Thu, 28 Jan 2016 19:22:39 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (KAFKA-3060) Refactor MeteredXXStore MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-3060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122147#comment-15122147 ] ASF GitHub Bot commented on KAFKA-3060: --------------------------------------- GitHub user guozhangwang opened a pull request: https://github.com/apache/kafka/pull/826 KAFKA-3060: Refactor MeteredStore and RockDBStore Impl Changes include: 1) Move logging logic from MeteredXXXStore to internal stores, and leave WindowedStore API clean by removed all internalPut/Get functions. 2) Wrap common logging behavior of InMemory and LRUCache stores into one class. 3) Fix a bug for StoreChangeLogger where byte arrays are not comparable in HashSet by using a specified RawStoreChangeLogger. 4) Add a caching layer on top of RocksDBStore with object caching, it relies on the object's equals and hashCode function to be consistent with serdes. You can merge this pull request into a Git repository by running: $ git pull https://github.com/guozhangwang/kafka K3060 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/826.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #826 ---- commit bf4c4cb3dbb5b4066d9c3e0ada5b7ffd98eb129a Author: Guozhang Wang Date: 2016-01-14T20:27:58Z add internal source topic for tracking commit 1485dff08a76c6ff685b0fe72226ce3b629b1d3c Author: Guozhang Wang Date: 2016-01-14T22:32:08Z minor fix for this.interSourceTopics commit 60cafd0885c41f93e408f8d89880187ddec789a1 Author: Guozhang Wang Date: 2016-01-15T01:09:00Z add KStream windowed aggregation commit 983a626008d987828deabe45d75e26e909032843 Author: Guozhang Wang Date: 2016-01-15T01:34:56Z merge from apache trunk commit 57051720de4238feb4dc3c505053096042a87d9c Author: Guozhang Wang Date: 2016-01-15T21:38:53Z v1 commit 4a49205fcab3a05ed1fd05a34c7a9a92794b992d Author: Guozhang Wang Date: 2016-01-15T22:07:17Z minor fix on HoppingWindows commit 9b4127e91c3a551fb655155d9b8e0df50132d0b7 Author: Guozhang Wang Date: 2016-01-15T22:43:14Z fix HoppingWindows commit 9649fe5c8a9b2e900e7746ae7b8745bb65694583 Author: Guozhang Wang Date: 2016-01-16T19:00:54Z add retainDuplicate option in RocksDBWindowStore commit 8a9ea02ac3f9962416defa79d16069431063eac0 Author: Guozhang Wang Date: 2016-01-16T19:06:12Z minor fixes commit 4123528cf4695b05235789ebfca3a63e8a832ffa Author: Guozhang Wang Date: 2016-01-18T17:55:02Z Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into K3104 commit 46e8c8d285c0afae6da9ec7437d082060599f3f1 Author: Guozhang Wang Date: 2016-01-18T19:15:47Z add wordcount and pipe jobs commit 582d3ac24bfe08edb1c567461971cd35c1f75a00 Author: Guozhang Wang Date: 2016-01-18T21:53:21Z merge from trunk commit 5a002fadfcf760627274ddaa016deeaed5a3199f Author: Guozhang Wang Date: 2016-01-19T00:06:34Z 1. WallClockTimestampExtractor as default; 2. remove windowMS config; 3. override state dir with jobId prefix; commit 7425673e523c42806b29a364564a747443712a53 Author: Guozhang Wang Date: 2016-01-19T01:26:11Z Add PageViewJob commit ca04ba8d18674c521ad67872562a7671cb0e2c0d Author: Guozhang Wang Date: 2016-01-19T06:23:05Z minor changes on topic names commit 563cc546b3a0dd16d586d2df33c37d2c5a5bfb18 Author: Guozhang Wang Date: 2016-01-19T21:30:11Z change config importance levels commit 4218904505363e61bb4c6b60dc5b13badfd39697 Author: Guozhang Wang Date: 2016-01-21T00:11:34Z Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into K3066 commit 26fb5f3f5a8c9b304c5b1e61778c6bc1d9d5fccb Author: Guozhang Wang Date: 2016-01-21T06:43:04Z demo examples v1 commit 6d92a55d770e058183daabb7aaef7675335fbbad Author: Guozhang Wang Date: 2016-01-22T00:41:12Z Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into K3066 commit 929e405058eb61d38510120f3f3ed50cd0cfaf47 Author: Guozhang Wang Date: 2016-01-22T01:02:04Z add RollingRocksDBStore commit 324eb584b97ed3c228347d108d697d2f5133ea99 Author: Guozhang Wang Date: 2016-01-22T01:23:32Z modify MeteredWindowStore commit 7ba2d90fe1de1ca776cea23ff1c2e8f8b3a6c3f2 Author: Guozhang Wang Date: 2016-01-22T01:35:10Z remove getter commit a4d78bac9d84dfd1c7dab4ae465b9115ddc451b3 Author: Guozhang Wang Date: 2016-01-22T01:36:51Z remove RollingRocksDB commit d0e8198ac6a25315d7ab8d21894acf0077f88fde Author: Guozhang Wang Date: 2016-01-22T17:24:32Z adding cache layer on RocksDB commit 257b53d3b6df967f8a015a06c9e178d4219d0f8c Author: Guozhang Wang Date: 2016-01-22T23:15:08Z dummy commit 25fd73107c577ac2e4b32300d4fe132ad7ff7312 Author: Guozhang Wang Date: 2016-01-22T23:21:29Z merge from trunk commit e57abf3c117d4ea7398c6157983d75443194ce9f Author: Guozhang Wang Date: 2016-01-22T23:29:59Z merge from demo example changes again commit a0caee14d4a7b77045d9813a1072143db0fc8fa1 Author: Guozhang Wang Date: 2016-01-22T23:50:19Z resolve conflicts commit 5501e6fef16fdef55cb2136d8c3f8035d394d79f Author: Guozhang Wang Date: 2016-01-26T17:12:12Z Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into K3060 commit d73b594147d1b4e25a61497550f498b9f14822a0 Author: Guozhang Wang Date: 2016-01-26T19:04:11Z move logging to rocksDBStore ---- > Refactor MeteredXXStore > ----------------------- > > Key: KAFKA-3060 > URL: https://issues.apache.org/jira/browse/KAFKA-3060 > Project: Kafka > Issue Type: Sub-task > Components: kafka streams > Affects Versions: 0.9.0.1 > Reporter: Yasuhiro Matsuda > Priority: Minor > > ** copied from a github comment by Guozhang Wang ** > The original motivation of having the MeteredXXStore is to wrap all metrics / logging semantics into one place so they do not need to be re-implemented again, but this seems to be an obstacle with the current pattern now, for example MeteredWindowStore.putAndReturnInternalKey is only used for logging, and MeteredWindowStore.putInternal / MeteredWindowStore.getInternal are never used since only its inner will trigger this function. So how about refactoring this piece as follows: > 1. WindowStore only expose two APIs: put(K, V) and get(K, long). > 2. Add a RollingRocksDBStores that does not extend any interface, but only implements putInternal, getInternal and putAndReturnInternalKey that uses underlying RocksDBStore as Segments. > 3. RocksDBWindowStore implements WindowStore with an RollingRocksDBStores inner. > 4. Let MeteredXXStore only maintain the metrics recording logic, and let different stores implement their own logging logic, since this is now different across different types and are better handled separately. Also some types of stores may not even have a loggingEnabled flag, if it will always log, or will never log. -- This message was sent by Atlassian JIRA (v6.3.4#6332)