From dev-return-95744-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Tue Jul 3 02:53:24 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BE838180626 for ; Tue, 3 Jul 2018 02:53:23 +0200 (CEST) Received: (qmail 92118 invoked by uid 500); 3 Jul 2018 00:53:22 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 92105 invoked by uid 99); 3 Jul 2018 00:53:21 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jul 2018 00:53:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 547371A4724 for ; Tue, 3 Jul 2018 00:53:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.889 X-Spam-Level: * X-Spam-Status: No, score=1.889 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id COUbyaowmGXH for ; Tue, 3 Jul 2018 00:53:17 +0000 (UTC) Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id CB6C55F430 for ; Tue, 3 Jul 2018 00:53:16 +0000 (UTC) Received: by mail-lf0-f50.google.com with SMTP id m13-v6so165425lfb.12 for ; Mon, 02 Jul 2018 17:53:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=9pFVtKSdZ8wTgZWavQQzOprwlE3bLI6RaOUF7Nt+ezk=; b=WKj01uCLexzV7HxDzZUpRdWqyb62kypCQM0mZJLvP6S2n47p9bvOIGIJanN0nQ+Y8P VQ2Wrz8uE5Nm4rfl5x89izqHcJwTtxXb2pXlrPw49jjr2QecwWZjKA/+gjKVhPLPa1v6 LgH3hyTLYPItShZSR+E60yAPW57JXNsgcZ4SSc5RG4FmufIOBT4YsL8A6lFIjgscX/fE xnaW/Eti1My/4WE12TBXSC1vx0D2L6x+t7M3LHbouwDwMxdLRDy/JrICpOY9X+3rxZZM /kVVwASOV3F+GX4SqLmxSDIDJYKsr8GEQIHGOEtK1eYtpkpheG1HUkf8HJlgiZyvgh5j THCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=9pFVtKSdZ8wTgZWavQQzOprwlE3bLI6RaOUF7Nt+ezk=; b=MEmNwP/mNo6GRN4tnXo8+vXwuSEKoianbNFTZ6t2gvcJBt2BFD6C+n6SF9G3k4DvXg 6NXYKjf1g/mz19jEreAsybRM7MlOvLn8Aq3X/pbDMItJfav3T78m6QUWFlDe2GReuAnA /gsDIfstRNdZWK5YQGG20EFv+wp+EBf+tSr58ArHcE+FqdsgyrIeZuRSOablHfUgFR3M 9UgPwm9eKh935ZwAQSgh2jR+8NpvmfWbBekqgfwqFwkU5mzPTiMgEwr20j/ja7UzX7KZ dBMROkdEx1FPqlF8c8hMlArKQLD2ZlteDRxQYl+2xzkdfiEZEa/taTFfgHJwYBswiDJd CAVA== X-Gm-Message-State: APt69E17akrKogHOC3d+eC6frMW50YHxHVv+NNoNsdEs6TauV1eKlF3Y Vjxzf7/cXJsKIlRvCV48hcRbOzBT/a/vYK/QbaZMeA== X-Google-Smtp-Source: AAOMgpc7afiVpwh8RU6mhHvJDCICYFQs7OZZy25pavvvstNqFEbrV9CkmSVW08BBN5S88maPBh3h6quG9+UMPspJ2ks= X-Received: by 2002:a19:5c06:: with SMTP id q6-v6mr15849952lfb.6.1530579195939; Mon, 02 Jul 2018 17:53:15 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a19:d555:0:0:0:0:0 with HTTP; Mon, 2 Jul 2018 17:52:35 -0700 (PDT) In-Reply-To: References: From: Ted Yu Date: Mon, 2 Jul 2018 17:52:35 -0700 Message-ID: Subject: Re: [VOTE] KIP-291: Have separate queues for control requests and data requests To: dev Content-Type: multipart/alternative; boundary="00000000000046513605700dbd6a" --00000000000046513605700dbd6a Content-Type: text/plain; charset="UTF-8" For #1, I don't know what would be good approximation for M. Maybe use max((TO / 2) / N, M / N) as default value for poll timeout ? For #2, I don't see the picture in email :-) Can you use third party website ? Thanks On Mon, Jul 2, 2018 at 5:17 PM, Lucas Wang wrote: > Hi Ted, > > 1. I'm neutral on making the poll timeout parameter configurable. > Mainly because as a config, it could be confusing for operators who try to > choose a value for it. > > To understand the implication of this value better, > let's use TO to represent the timeout value under discussion, > M to denote the processing time of data requests, > and N to be the number of io threads. > > - If the data request queue is empty and there is no incoming data > requests, > all io threads should be blocked on the data request queue, and > the average delay for a controller request is (TO / 2) / N, and the > worst case delay is TO. > - If all IO threads are busy processing data requests, then the average > latency for a controller request is M / N. > - In the worst case, a controller request can just miss the train, and IO > threads get blocked on data request queue > for TO, at the end of which they all receive a new incoming data > request, the latency for the > controller request can be TO + M. > > Given the intricacies, what do you think about choosing a relatively > meaningful value and stick with it, > rather than exposing it as a config? > > 2. Sorry for losing the format of the table, I've attached it below as a > picture > > > Regards, > Lucas > > On Fri, Jun 29, 2018 at 5:28 PM, Ted Yu wrote: > >> bq. which is hard coded to be 300 milliseconds >> >> Have you considered making the duration configurable ? >> >> The comparison at the end of your email seems to be copied where tabular >> form is lost. >> Do you mind posting that part again ? >> >> Thanks >> >> On Fri, Jun 29, 2018 at 4:53 PM, Lucas Wang >> wrote: >> >> > Hi Jun, >> > >> > Thanks for your comments. >> > 1. I just replied in the discussion thread about the positive change >> this >> > KIP can still bring >> > if implemented on the latest trunk, which includes the async ZK >> operations >> > for KAFKA-5642. >> > The evaluation is done using an integration test. >> > In production, we have not upgraded to Kafka 1.1 yet, and the code we >> are >> > currently running does >> > not include async ZK operations, therefore I don't have any real usage >> > result. >> > >> > 2. Thanks for bringing this up. I haven't considered this setting, and >> the >> > existing proposal in this KIP >> > would make data requests and controller requests share a memory poll of >> > size specified by the config >> > queued.max.request.bytes. The downside is that if there is memory >> pressure, >> > controller requests may be blocked >> > from being read from a socket and does not get prioritized at the socket >> > layer. >> > >> > If we have a separate bytes limit for the controller requests, I imagine >> > there would be a separate memory pool >> > dedicated to controller requests. Also it requires the processors to >> tell >> > connections from a controller apart >> > from connections from other brokers or clients, which would probably >> > require a dedicated port for the controller? >> > IMO, this change is mainly driven by the memory pressure, kind of an >> > orthogonal issue, and we can address it with a separate KIP >> > if desired. Please let me know what you think. >> > >> > 3. I plans to change the implementation of the method >> > receiveRequest(timeout: Long) in the RequestChannel class as follows: >> > >> > val controllerRequest = controllerRequestQueue.poll() >> > if (controllerRequest != null) { >> > controllerRequest >> > } else { >> > dataRequestQueue.poll(timeout, TimeUnit.MILLISECONDS) >> > } >> > >> > with this implementation, there is no need to explicitly choose a >> request >> > handler thread to wake up depending on >> > the types of request enqueued, and if a controller request arrives while >> > some request handler threads are blocked on an empty data request queue, >> > they will simply timeout and call the receiveRequest method again. >> > >> > In terms of performance, it means that in the worst case, for a >> controller >> > request that just missed the receiveRequest call, it can be delayed for >> as >> > long as >> > the timeout parameter, which is hard coded to be 300 milliseconds. If >> there >> > is just one request handler thread, the average delay is >> > 150 milliseconds assuming the chance of a controller request arriving at >> > any particular time is the same. With N request handler threads, >> > the average delay is 150/N milliseconds, which does not seem to be a >> > problem. >> > >> > We have considered waking up of request handler threads based on which >> > queue the request handler threads are blocked, >> > and that design was turned down because of its complexity. The design >> can >> > be found at here >> > > > controller+request+queue+design> >> > . >> > >> > If you mean a general purpose priority queue such as the >> > java.util.PriorityQueue, we also have considered it and turned down the >> > design because >> > - The readily available class java.util.PriorityQueue is unbounded and >> > we'll need to implement a bounded version >> > - We would still like to have the FIFO semantics on both the controller >> > request queue and data request queue, which conceptually does not fit >> very >> > well >> > with a general purpose priority queue, e.g. we would probably need to >> use >> > the enqueue time to enforce FIFO semantics. >> > - A typical operation on the priority queue is O(log n), whereas the >> sample >> > implementation above gives O(1) performance regardless of the size of >> both >> > queues. >> > >> > 4. For the two APIs sendRequest and receiveRequest, since we are only >> > changing their implementation, not the API itself >> > the two metrics will support two queues and the meaning of "Idle" still >> > holds: >> > >> > >> > >> > >> > >> > >> > *Before this KIPAfter this KIPNetworkProcessorAvgIdlePercentidle = >> blocked >> > on selectnot idle includes being blocked on requestQueueidle = blocked >> on >> > selectnot idle includes being blocked on either controller request >> queue or >> > data request queueRequestHandlerAvgIdlePercentidle = blocked on reading >> > from requestQueue idle = taking a request from the controller request >> > queue, or blocked on reading from the data request queue* >> > >> > Regards, >> > Lucas >> > >> > On Fri, Jun 29, 2018 at 11:22 AM, Jun Rao wrote: >> > >> > > Hi, Lucas, >> > > >> > > Thanks for the KIP. A few comments below. >> > > >> > > 1. As Eno mentioned in the discussion thread, I am wondering how much >> of >> > > this is still an issue after KAFKA-5642. With that fix, the requests >> from >> > > the controller to the brokers are batched in all the common cases. >> Have >> > you >> > > deployed Kafka 1.1? What's the request queue time and the request >> queue >> > > size that you have observed in production? >> > > >> > > 2. For the request queue, currently we can also bound it by size >> > > through queued.max.request.bytes. Should we consider the same for the >> > > control queue? >> > > >> > > 3. Implementation wise, currently the request handler threads just >> block >> > on >> > > the request queue when the queue is empty. With two queues, it seems >> that >> > > we need to wake up a request handler thread blocked on one queue, when >> > > another queue gets a request? Have we considered just making the >> request >> > > queue a priority queue? >> > > >> > > 4. Related to 3, currently we have 2 >> > > metrics NetworkProcessorAvgIdlePercent and >> RequestHandlerAvgIdlePercent >> > > that measure the utilization of the network and the request handler >> > thread >> > > pools. They are computed by measuring the amount of time waiting on >> the >> > > request queue. Will these 2 metrics be extended to support 2 request >> > > queues. >> > > >> > > Jun >> > > >> > > >> > > On Mon, Jun 18, 2018 at 1:04 PM, Lucas Wang >> > wrote: >> > > >> > > > Hi All, >> > > > >> > > > I've addressed a couple of comments in the discussion thread for >> > KIP-291, >> > > > and >> > > > got no objections after making the changes. Therefore I would like >> to >> > > start >> > > > the voting thread. >> > > > >> > > > KIP: >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-291% >> > > > 3A+Have+separate+queues+for+control+requests+and+data+requests >> > > > >> > > > Thanks for your time! >> > > > Lucas >> > > > >> > > >> > >> > > --00000000000046513605700dbd6a--