Return-Path: 
 <dev-return-7822-apmail-river-dev-archive=river.apache.org@river.apache.org>
X-Original-To: apmail-river-dev-archive@www.apache.org
Delivered-To: apmail-river-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id C9B2D96C7
	for <apmail-river-dev-archive@www.apache.org>;
 Fri, 13 Jan 2012 15:13:07 +0000 (UTC)
Received: (qmail 24791 invoked by uid 500); 13 Jan 2012 15:13:07 -0000
Delivered-To: apmail-river-dev-archive@river.apache.org
Received: (qmail 24763 invoked by uid 500); 13 Jan 2012 15:13:07 -0000
Mailing-List: contact dev-help@river.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@river.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@river.apache.org>
List-Post: <mailto:dev@river.apache.org>
List-Id: <dev.river.apache.org>
Reply-To: dev@river.apache.org
Delivered-To: mailing list dev@river.apache.org
Received: (qmail 24754 invoked by uid 99); 13 Jan 2012 15:13:07 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2012 15:13:07 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=5.0
	tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [209.85.216.43] (HELO mail-qw0-f43.google.com) (209.85.216.43)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2012 15:12:57 +0000
Received: by qadz30 with SMTP id z30so855100qad.2
        for <dev@river.apache.org>; Fri, 13 Jan 2012 07:12:36 -0800 (PST)
Received: by 10.224.111.144 with SMTP id s16mr2281322qap.5.1326467556618;
        Fri, 13 Jan 2012 07:12:36 -0800 (PST)
Received: from dhcp-1-174.qrclab.com
 (173-166-24-193-newengland.hfc.comcastbusiness.net. [173.166.24.193])
        by mx.google.com with ESMTPS id df3sm17164086qab.6.2012.01.13.07.12.35
        (version=TLSv1/SSLv3 cipher=OTHER);
        Fri, 13 Jan 2012 07:12:35 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: DGC threads issue
From: Peter Jones <pcj@roundroom.net>
In-Reply-To: 
 <DE10B00CCE0DC54883734F3060AC9ED454A33B92F4@AUSP01VMBX06.collaborationhost.net>
Date: Fri, 13 Jan 2012 10:12:33 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <923C0385-0043-448E-AF1A-440675B08C4F@roundroom.net>
References: <BLU151-W2758B4A0ACC50D1C1B544693A30@phx.gbl>
 <CAMPUXz8QuDM=LFTd7ANLLnk54-9YSa7ORzEpySXvmRV46ydTBw@mail.gmail.com>
 <BLU151-W111922896459C4283EC8E793A30@phx.gbl>
 <CAMPUXz_LH11T_50s0E7ocRqBKJmEMM-5SWedkPxqUMYKaTdWGQ@mail.gmail.com>
 <BLU151-W546F538BC0A9207A55FA5593A00@phx.gbl>
 <CAMPUXz-Xp1==fc67QWeZQB8eDkObaF6_Wh0NLb-c3kp4YR39YQ@mail.gmail.com>
 <BLU151-W24FE71D8CCE20CFDA1652493A00@phx.gbl>
 <CAMPUXz9pZDp_xgjFnPqTMFQo=qYnitYiqmBFXVf8wBt5GYi=aA@mail.gmail.com>
 <4EEB90DA.1070902@cox.net>
 <DE10B00CCE0DC54883734F3060AC9ED454A3086E25@AUSP01VMBX06.collaborationhost.net>
 <DE10B00CCE0DC54883734F3060AC9ED454A325A628@AUSP01VMBX06.collaborationhost.net>
 <CAEDB+kXJLdEnP7+Sw7oVo3M9aBC6HLXdHBzOaBNj-ZogvgvMsg@mail.gmail.com>
 <FD11B289-7F0F-405E-8D86-D56B094A685C@roundroom.net>
 <DE10B00CCE0DC54883734F3060AC9ED454A33B92AD@AUSP01VMBX06.collaborationhost.net>
 <BB0CE359-8BF6-4619-844A-AE4C71EB89D5@roundroom.net>
 <DE10B00CCE0DC54883734F3060AC9ED454A33B92F4@AUSP01VMBX06.collaborationhost.n
 et>
To: dev@river.apache.org
X-Mailer: Apple Mail (2.1084)
X-Virus-Checked: Checked by ClamAV on apache.org

Bryan,

Thanks, that is indeed helpful to see, to be clear about which kind of =
threads are causing the problem.

-- Peter


On Jan 13, 2012, at 9:52 AM, Bryan Thompson wrote:

> Peter,
>=20
> There is very little information in there.  Basically a whole lot of =
"DGC Lease Checker" threads all sleeping in Thread.run(). =20
>=20
> The stacks below are from a capture in yourkit that I had on hand from =
when I was investigating this problem.  The workload had been removed =
from the service but the leases had not yet expired.
>=20
> Thanks,
> Bryan
>=20
> Stacks at 09:16:02 AM (uptime 1:48:56)
>=20
>=20
> (JSK) ConnectionManager.Reaper [SLEEPING] CPU time: 0:00
> java.lang.Thread.sleep(long)
> net.jini.jeri.connection.ConnectionManager$Reaper.run()
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Threa
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> (JSK) DGC Lease Checker [SLEEPING] CPU time: 0:hread.run()
>=20
>=20
>=20
> com.bigdata.journal.ConcurrencyManager.writeService48 [WAITING] CPU =
time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> com.bigdata.journal.ConcurrencyManager.writeService49 [WAITING] CPU =
time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
> com.bigdata.journal.ConcurrencyManager.writeService5 [WAITING] CPU =
time: 0:00
> java.lang.Thread.run()
>=20
>=20
>=20
>> -----Original Message-----
>> From: Peter Jones [mailto:pcj@roundroom.net]=20
>> Sent: Friday, January 13, 2012 9:31 AM
>> To: user@river.apache.org
>> Cc: dev@river.apache.org
>> Subject: Re: DGC threads issue
>>=20
>> Bryan,
>>=20
>> I meant that it might help for the list to "see" the specific=20
>> threads in question, as they appear in a JVM thread dump=20
>> (name, stack frames, etc.), just to be sure that we're=20
>> talking about the same thing.  There is more than one kind of=20
>> thread related to DGC, and it seems that the implementation=20
>> has changed recently.  But I gather that Peter F. may have=20
>> identified the root cause.
>>=20
>> Cheers,
>>=20
>> -- Peter
>>=20
>>=20
> <DGCThreadDump.txt>