Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B0886C625 for ; Mon, 5 Jan 2015 13:22:07 +0000 (UTC) Received: (qmail 55869 invoked by uid 500); 5 Jan 2015 13:22:06 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 55797 invoked by uid 500); 5 Jan 2015 13:22:06 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 55744 invoked by uid 99); 5 Jan 2015 13:22:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2015 13:22:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yuzhihong@gmail.com designates 209.85.220.42 as permitted sender) Received: from [209.85.220.42] (HELO mail-pa0-f42.google.com) (209.85.220.42) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2015 13:21:55 +0000 Received: by mail-pa0-f42.google.com with SMTP id et14so28818185pad.29 for ; Mon, 05 Jan 2015 05:20:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:subject :message-id:date:references:in-reply-to:to; bh=YgcE1zBv8p5R/uDQqEmIEH1YJ7XM4dhSaOAuvXEHuuw=; b=HoPFhrlTahwYYwz5GyINdKRefYwOpMQxnAaHB/iraJT062iIeJslCKz/ymSwWy10wA Gi8J1yyFIOb42MudRKu0CjIIyszCN+hHY6C0kNGX0WdeuhbHfQoDRk2hyMNpTlOeq4DN xQLymfFjNGqWp/lZ3uDsZtSh2hUcJeEajQr3LTr9+ccS+VH6PsaC0iMoZBxwMo4fBBF8 AF0vq6s/jpEmv7yswZXFo0a6Yota68pjut3qFwIwc3zS5Fi12g7Fh3wh4dn5CBp4syuc RK31v9H0IwfrK0P3igxNeO6DppkyxfqGv7FhBGAqd+ordVlwKkPphGrbCfvwtnm6fIrQ 0uCw== X-Received: by 10.70.35.109 with SMTP id g13mr83300540pdj.17.1420464049548; Mon, 05 Jan 2015 05:20:49 -0800 (PST) Received: from [192.168.0.20] (c-24-130-236-83.hsd1.ca.comcast.net. [24.130.236.83]) by mx.google.com with ESMTPSA id wk2sm47198460pac.12.2015.01.05.05.20.48 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 05 Jan 2015 05:20:48 -0800 (PST) From: Ted Yu Content-Type: multipart/alternative; boundary=Apple-Mail-9083C19D-200C-4934-97DC-B885C7CF096E Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Subject: Re: Threads leaking from Apache tomcat application Message-Id: Date: Mon, 5 Jan 2015 05:20:47 -0800 References: In-Reply-To: To: "user@hbase.apache.org" X-Mailer: iPhone Mail (12B440) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-9083C19D-200C-4934-97DC-B885C7CF096E Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable There're several non-English phrases which seem to be links.=20 But when I clicked on them, there was no response.=20 Can you give the links in URL ? Cheers > On Jan 5, 2015, at 2:39 AM, Serega Sheypak wrot= e: >=20 > Hi, I'm still trying to deal with apache tomcat web-app and hbase HBase 0.= 98.6=20 > The root problem is that user threads constantly grows. I do get thousands= of live threads on tomcat instance. Then it dies of course.=20 >=20 > please see visualVM threads count dynamics >=20 >=20 > Please see selected thread. It should be related to zookeeper (because of t= hread-name suffix SendThread) >=20 >=20 >=20 > The threaddump for this thread is: >=20 > "visit-thread-27799752116280271-EventThread" - Thread t@75 > java.lang.Thread.State: WAITING > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <34671cea> (a java.util.concurrent.locks.Abst= ractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186= ) > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2043) > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 494) >=20 > Locked ownable synchronizers: > - None >=20 > Why does it live "forever"? I next 24 hours I would get ~1200 live theads.= >=20 > "visit thread" does simple put/get by key, newrelic says it takes 30-40 ms= to respond. > I just set a name for the thread inside servlet method. >=20 > Here is CPU profiling result: >=20 >=20 > Here are some Zookeeper metrics. >=20 >=20 --Apple-Mail-9083C19D-200C-4934-97DC-B885C7CF096E--