From dev-return-79010-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Sat Mar 9 00:34:46 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7CE00180626 for ; Sat, 9 Mar 2019 01:34:45 +0100 (CET) Received: (qmail 30367 invoked by uid 500); 9 Mar 2019 00:34:44 -0000 Mailing-List: contact dev-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@zookeeper.apache.org Delivered-To: mailing list dev@zookeeper.apache.org Delivered-To: moderator for dev@zookeeper.apache.org Received: (qmail 91122 invoked by uid 99); 9 Mar 2019 00:22:11 -0000 X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.798 X-Spam-Level: * X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=AlxdyvdTn+bZ3Rv4sllU+/mbCJvVx/Joz/Awn5ciimY=; b=V7v/C1dzVK6eiI9c6NWsNDfM8IVdZgZ1iw0aPpRd8npa1Nxvh1XI0rad8Lrb3ITlIg Y3wNA+doG9IQqOg0ej0tAYOeRePe9O8EJAqseM28cSC0LC9y/8afMgsSumKOxJXVLmTG 8pmczmYp96XNvhpkqLaeHsikh42PwRBxRmLtvSIw7OG/LpdDH2ro69JIe0AZaF2GB4B+ 1GTJW+eWl3xIhj9lMbItIHmw1MchkifYd+te55B3XwR6s6cT+U+YdBJc0Ou5Y2BksDbc 04XHlOEcxwaGGsUrVCac7TykHikCxYYmyDreM3RwVIop91FqOfQknNQna8XNTKur6oGu CVYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=AlxdyvdTn+bZ3Rv4sllU+/mbCJvVx/Joz/Awn5ciimY=; b=AB5wdLNgbySy9DqETR27ewoD/jExEOJPJA65B0/WTy1YsN3krPSLTV0daAp8fMWjQE VNT8JKwJQIen+5VemgZgj2UFqdSiH8n8emSzKxd3HchepHGji70J5JkZ6yx5rtb6VagJ 1UIZyMu4Mzzfk9RL3eR0BFDgl3jwLp609pVPMSM13PhAeRBruUB4/l/JhBxOnMWvcp6t N7Uwra8l3vDOt9f+OaYsbMczb+tFj9Tqs8t7aCqIZo7WgOijjUrdFdPL1JfMWJxiTHCV 6rv5LJ+Tiv9y/yJs87WwgK68QsOVH3IeFK+NzwJhd+/52czPvjbqnpQ8On7cEHjOC5C7 uHpg== X-Gm-Message-State: APjAAAXN01+wkxzULmCg5wsJZ7Hg+x0eG4Ce2HuphkqMoZA7zdqc6qPD t9WOgK9hhPCN+lmypNeODYHy6uUOdc6Rx+3E/J8= X-Google-Smtp-Source: APXvYqzNXbiXoJX+8oiBBSgMdI0ICUz2H2rN4uttvf/5QgPmkF+qIsSJWclqleRwQEu7zlljt/uJ5iPGpdZm5OoarTM= X-Received: by 2002:a81:4c13:: with SMTP id z19mr17238908ywa.57.1552090927856; Fri, 08 Mar 2019 16:22:07 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Asim Zafir Date: Fri, 8 Mar 2019 16:21:57 -0800 Message-ID: Subject: Re: Jute buffer size increasing. To: jordan@jordanzimmerman.com, dev@zookeeper.apache.org Content-Type: multipart/alternative; boundary="0000000000006a049705839e548c" --0000000000006a049705839e548c Content-Type: text/plain; charset="UTF-8" + ZK dev community. Please assist. On Fri, Mar 8, 2019 at 4:10 PM Asim Zafir wrote: > Hi Jordon, > > We are seeing constance increase in jute buffer size on our zookeeper > instance. right now it is set to 128. We are primarily using zookeeper for > HBase cluster. I want to see what is contributing to the increase of jute > buffer size but so for after investigating the code and studying the > protocol itself it appear it is a function of number of watches that gets > set on the znodes. to see how many zookeeper watch objects are on zookeeper > jvm /instance I did a jmap history:live on zookeeper pid and I got the > following output (please see below). I am not sure what is [C, [B here and > it doesn't appear its refers to any class - I don't see this on dev > instance of zookeeper. due to suspect memory leak or another issue? Please > guide me through this as I can't find a resource who can go that far to > give me any hint as to what may be happening on my end. Also is it safe for > ZK sizes to increase that much? I will greatly appreciate your feedback and > help on this. > > num #instances #bytes class name > > ---------------------------------------------- > > 1: 220810 140582448 [C > > 2: 109370 34857168 [B > > 3: 103842 7476624 org.apache.zookeeper.data.StatPersisted > > 4: 220703 5296872 java.lang.String > > 5: 28682 3783712 > > 6: 28682 3681168 > > 7: 111000 3552000 java.util.HashMap$Entry > > 8: 107569 3442208 > java.util.concurrent.ConcurrentHashMap$HashEntry > > 9: 103842 3322944 org.apache.zookeeper.server.DataNode > > 10: 2655 3179640 > > 11: 2313 2017056 > > 12: 2655 1842456 > > 13: 318 1241568 > [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; > > 14: 7526 1221504 [Ljava.util.HashMap$Entry; > > 15: 1820 812976 > > 16: 8228 394944 java.util.HashMap > > 17: 2903 348432 java.lang.Class > > 18: 4077 229688 [S > > 19: 4138 221848 [[I > > 20: 231 125664 > > 21: 7796 124736 java.util.HashSet > > 22: 6771 108336 java.util.HashMap$KeySet > > 23: 1263 62968 [Ljava.lang.Object; > > 24: 746 59680 java.lang.reflect.Method > > 25: 3570 57120 java.lang.Object > > 26: 502 36144 org.apache.zookeeper.server.Request > > 27: 649 25960 java.lang.ref.SoftReference > > 28: 501 24048 org.apache.zookeeper.txn.TxnHeader > > 29: 188 21704 [I > > 30: 861 20664 java.lang.Long > > 31: 276 19872 java.lang.reflect.Constructor > > 32: 559 17888 > java.util.concurrent.locks.ReentrantLock$NonfairSync > > 33: 422 16880 java.util.LinkedHashMap$Entry > > 34: 502 16064 > org.apache.zookeeper.server.quorum.QuorumPacket > > 35: 455 14560 java.util.Hashtable$Entry > > 36: 495 14368 [Ljava.lang.String; > > 37: 318 12720 > java.util.concurrent.ConcurrentHashMap$Segment > > 38: 3 12336 [Ljava.nio.ByteBuffer; > > 39: 514 12336 javax.management.ObjectName$Property > > 40: 505 12120 java.util.LinkedList$Node > > 41: 501 12024 > org.apache.zookeeper.server.quorum.Leader$Proposal > > 42: 619 11920 [Ljava.lang.Class; > > 43: 74 11840 > org.apache.zookeeper.server.NIOServerCnxn > > 44: 145 11672 [Ljava.util.Hashtable$Entry; > > 45: 729 11664 java.lang.Integer > > 46: 346 11072 java.lang.ref.WeakReference > > 47: 449 10776 org.apache.zookeeper.txn.SetDataTxn > > 48: 156 9984 > com.cloudera.cmf.event.shaded.org.apache.avro.Schema$Props > > 49: 266 8512 java.util.Vector > > 50: 75 8400 sun.nio.ch.SocketChannelImpl > > 51: 175 8400 java.nio.HeapByteBuffer > > 52: 247 8320 [Ljavax.management.ObjectName$Property; > > 53: 303 7272 com.cloudera.cmf.event.EventCode > > 54: 300 7200 java.util.ArrayList > > 55: 136 6528 java.util.Hashtable > > 56: 156 6240 java.util.WeakHashMap$Entry > > 57: 194 6208 > com.sun.jmx.mbeanserver.ConvertingMethod > --0000000000006a049705839e548c--