From user-return-11319-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Sun Mar 25 11:34:35 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D363118063B for ; Sun, 25 Mar 2018 11:34:30 +0200 (CEST) Received: (qmail 91812 invoked by uid 500); 25 Mar 2018 09:34:29 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 91796 invoked by uid 99); 25 Mar 2018 09:34:28 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 25 Mar 2018 09:34:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id B91ECC1835 for ; Sun, 25 Mar 2018 09:34:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.363 X-Spam-Level: *** X-Spam-Status: No, score=3.363 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, MPART_ALT_DIFF_COUNT=1.483, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gigaspaces.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 53Mm9CobaSBq for ; Sun, 25 Mar 2018 09:34:18 +0000 (UTC) Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id C52765F3FF for ; Sun, 25 Mar 2018 09:34:17 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id r82so10391115wme.0 for ; Sun, 25 Mar 2018 02:34:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gigaspaces.com; s=google; h=from:subject:date:message-id:cc:to:mime-version; bh=CGZhOJfB3Wt3TGumZx6YZfzc2CskNOYfxNXtVIZRWaU=; b=P/bnbj1+mHo1xWfp5VYzipxYq5K9cvoaegGRli0AUoJihvCgJHM7ucQ4p7Y4hPyWLF yO8//Dnrr2uM2xvpcZ0/IdiTJTv7tpHkdWV8BmhOrGAkQpponzti+BMfDuQy+pNNXMPN AcVG1V8xfceh67f5ho71VmbNM0W4cM8sCeKWiPdyRTK6ejR7FKLtoLfkvnCJqm90Z66P s6NQoHJNv7+L5rnHo9QSsmrnw2WgKXOTl7sNExcTbrPVV72Dx6GTFV5VFiY2VA4eLHNb 6RJLy+tWki12xpA6YbiAzajizG4MUh+1kT8HS/WYbSN17ZBZ4WGK1FIXwfpxZzudxZIk 1Ylg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:date:message-id:cc:to:mime-version; bh=CGZhOJfB3Wt3TGumZx6YZfzc2CskNOYfxNXtVIZRWaU=; b=jnx1xZKmNZh/PIJ0hZ3I6R48OM800VH796vFA2HyY9GloRUJsntXv1uJbtkqQQENE/ xMWBazSfFHDfMhvNnaf5X4MK7OX4yhnlXi7mlBOkSnIseRnha/6dej7CXdE6m9PB1zzD 5xu0raupFZhxs0CN73GaVEBx1BOyeIOgVQE9l5tyieWdLIEoq+GlHy6mKXbGfrAAolh8 OfoB5BfrVHhAp+woETNPZtaugEXVWQhJTwbUuMkDKkrtUul+6S26m8YHoomROYt+Ay+J 8ip+3MaBO2hdapxqDaNHCbFPQkD7dXJY3PvJx6yooxwplxnOhk34GSOfKDU2ojH5tUy1 joZQ== X-Gm-Message-State: AElRT7GJORGvp4XUNADqnyc42r+zky1eVq1B/7rgGbKNUXThhdekzP79 EzPHWLT9KxeFwxfW2rsGBkPDB5zuLYXEE2qJJDEoUG7YZC7AMn3VvOTDuUXoCf5MGlm3atU2G23 cVsbV7pAYLNYxrchkSANlB5bRATwgLmQNjwF2KNs0Hg27dZiun2QE5+4BszScSQFTyCd1EAfPpX iQcL0= X-Google-Smtp-Source: AIpwx49RxHQfFSvPMayOEhElE3lWbDMg7klD4/+7xdaN+YTQqwuuIMUoEmsY4xtLESaPP+WGx7Dq6A== X-Received: by 10.28.144.196 with SMTP id s187mr1174994wmd.82.1521970456777; Sun, 25 Mar 2018 02:34:16 -0700 (PDT) Received: from [192.168.9.185] ([94.188.164.162]) by smtp.gmail.com with ESMTPSA id j21sm16937232wrb.49.2018.03.25.02.34.14 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 25 Mar 2018 02:34:15 -0700 (PDT) From: Yuval Dori X-Google-Original-From: Yuval Dori Content-Type: multipart/alternative; boundary="Apple-Mail=_2DBC7B5E-CEA2-4664-88C6-52912FBBFBCC" Subject: IllegalStateException: instance must be started before calling this method Date: Sun, 25 Mar 2018 12:34:13 +0300 Message-Id: <18C41437-95F6-42DB-8D5A-1119A6677B0C@gigaspaces.com> Cc: Yuval Dori To: user@zookeeper.apache.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) --Apple-Mail=_2DBC7B5E-CEA2-4664-88C6-52912FBBFBCC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, Our customer is using ZK 3.4.3 and got IllegalStateException: instance = must be started before calling this method with the following stack trace: 2018-02-15T09:58:12.094+0100; ERROR; WSOSTSLXWIT01/MANAGER; P3424/T194; = [SPACE/LearnerHandler-/10.17.46.142:49336/LearnerHandler = ]; Unexpected exception = causing shutdown while sock still open=20 java.net.SocketTimeoutException: Read timed out=20 at java.net.SocketInputStream.socketRead0(Native Method)=20 at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)=20 at java.net.SocketInputStream.read(SocketInputStream.java:171)=20 at java.net.SocketInputStream.read(SocketInputStream.java:141)=20 at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)=20 at java.io.BufferedInputStream.read(BufferedInputStream.java:265)=20 at java.io.DataInputStream.readInt(DataInputStream.java:387)=20 at = org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)=20= at = org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.j= ava:83)=20 at = org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)=20= at = org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:= 542)=20 This is the description: =20 At first ps-main.1 is started as primary on WSOSTSLXBMS01 GSC 3.=20 24.01.2018 11:31:04,947 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/1 Space instance = is Primary=20 After some relocation of ps-main instances a new backup instance is = started on the same GSC.=20 24.01.2018 11:32:59,886 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T146 SPACE/GS-LRMI Connection-pool-1-thread-6/1 Space = instance [ frqMain_container1:frqMain ] has been selected as Backup=20 Later a failover test is performed by shutting down WSOSTSLXBMS02 and = WSOSTSLXMSS02.=20 We would expect that ps-main.1 backup will be elected as primary because = the former primary was located at WSOSTSLXBMS02 but no election happend. = In the end in the whole cluster there was no primary of ps-main.1 = running.=20 We found suspicious exception in the logs:=20 24.01.2018 11:32:44,987 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T120 = SPACE/Curator-LeaderSelector-0/frqMain_container1:frqMain Lost = leadership [sessionId=3D0x36128253b820001]=20 24.01.2018 11:32:44,987 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T98 SPACE/Curator-Framework-0/CuratorFrameworkImpl = backgroundOperationsLoop exiting=20 24.01.2018 11:32:44,990 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/1 Quiesce state = set to SUSPENDED=20 24.01.2018 11:32:44,991 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/ZooKeeper = Session: 0x36128253b820001 closed=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector The = leader threw an exception=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector = java.lang.IllegalStateException: instance must be started before calling = this method=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(= Preconditions.java:176)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrame= workImpl.java:359)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(Loc= kInternals.java:339)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockI= nternals.java:123)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.locks.InterProcessMutex.release(Inter= ProcessMutex.java:154)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSe= lector.java:449)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(Lead= erSelector.java:466)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(Lead= erSelector.java:65)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSe= lector.java:246)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSe= lector.java:240)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.util.concurrent.FutureTask.run(FutureTask.java:266)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)=20= 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.util.concurrent.FutureTask.run(FutureTask.java:266)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at = java.lang.Thread.run(Thread.java:748)=20 24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 = P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector=09 24.01.2018 11:32:44,991 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T97 SPACE/GS-LRMI = Connection-pool-1-thread-1-EventThread/ClientCnxn EventThread shut = down for session: 0x36128253b820001=20 24.01.2018 11:32:44,997 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/frqMain1_1 = Channel is closed [target=3DfrqMain1_1, target = url=3Djini://*/frqMain_container1_1/frqMain?groups=3DPOLBAY&ignoreValidati= on=3Dtrue&schema=3Dpersistent&id=3D1&total_members=3D4,1&versioned=3Dtrue&= mirror=3Dtrue&cluster_schema=3Dpartitioned&locators=3DWSOSTSLXMSS01,WSOSTS= LXMSS02&state=3Dstarted&timeout=3D5000 <>, target machine connection = url=3DNIO://WSOSTSLXBMS02:11002/pid[3756]/24795468496_3_253803293193956822= 6_details[class <> = com.gigaspaces.internal.cluster.node.impl.router.AbstractConnectionProxyBa= sedReplicationRouter$ConnectionEndpoint(frqMain_container1_1:frqMain)]]=20= 24.01.2018 11:32:44,998 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI = Connection-pool-1-thread-1/ps-main-mirror-service Channel is = closed [target=3Dps-main-mirror-service, target = url=3Djini://*/ps-main-mirror-service_container/ps-main-mirror-service?sch= ema=3Dpersistent&id=3D1&total_members=3D4,1&versioned=3Dtrue&mirror=3Dtrue= &groups=3DPOLBAY&cluster_schema=3Dpartitioned&locators=3DWSOSTSLXMSS01,WSO= STSLXMSS02&state=3Dstarted&timeout=3D5000 <>, target machine connection = url=3DNIO://WSOSTSLXBMS01:11002/pid[3220]/25906827215_3_719639353708517805= 9_details[class <> = com.gigaspaces.internal.cluster.node.impl.router.AbstractConnectionProxyBa= sedReplicationRouter$ConnectionEndpoint(ps-main-mirror-service_container:p= s-main-mirror-service)]]=20 24.01.2018 11:32:45,767 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/1 Shutdown = complete=20 24.01.2018 11:32:45,768 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/container = Container shutdown completed=20 24.01.2018 11:32:46,037 +01:00 WARN WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI = Connection-pool-1-thread-1/PUServiceBeanImpl Failed to delete = deployed processing unit from [C:\Program Files\Frequentis\Frequentis = LifeX Platform\gigaspaces\work\processing-units\ps-main_1_2008048111]=20 24.01.2018 11:32:46,037 +01:00 INFO WSOSTSLXBMS01/GSC/3 = P1280/T24 SPACE/GS-LRMI = Connection-pool-1-thread-1/PUServiceBeanImpl Stopped=20 Attached thread dump. Failover happend around 24.01.2018 11:40:20,893 = +01:00=09 WARN WSOSTSLXBMS01/GSC/3 P1280/T59 = SPACE/GS-Watchdog/watchdog The ServerEndPoint = [WSOSTSLXMSS02/10.17.46.142:11001 ] is not = reachable or is reachable but with no matching invocation in progress at = the server peer, closing invalid connection with local address = [/10.17.46.136:49864 ] [ remote invocation = of: net.jini.lookup.DiscoveryAdmin.getMemberGroups] = [java.net.SocketTimeoutException] Could it be ZK bug? Thanks, Yuval --Apple-Mail=_2DBC7B5E-CEA2-4664-88C6-52912FBBFBCC Content-Type: multipart/mixed; boundary="Apple-Mail=_F234B00D-9523-440D-AC4E-721E3FE9F9C4" --Apple-Mail=_F234B00D-9523-440D-AC4E-721E3FE9F9C4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Hi,

Our customer is using = ZK 3.4.3 and got IllegalStateException: instance must be started before = calling this method


with the following = stack trace:
2018-02-15T09:58:12.094+0100; ERROR; = WSOSTSLXWIT01/MANAGER; P3424/T194; [SPACE/LearnerHandler-/10.17.46.142:49336/LearnerHandler]; Unexpected exception = causing shutdown while sock still open 
java.net.SocketTimeoutException: Read timed = out 
at java.net.SocketInputStream.socketRead0(Native Method) 
at = java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
at java.net.SocketInputStream.read(SocketInputStream.java:171) 
at java.net.SocketInputStream.read(SocketInputStream.java:141) 
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 
at java.io.BufferedInputStream.read(BufferedInputStream.java:265) 
at java.io.DataInputStream.readInt(DataInputStream.java:387) 
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) 
at = org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) 
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99) 
at = org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:542) 


This is the = description:
 
At first ps-main.1 is = started as primary on WSOSTSLXBMS01 GSC 3. 

24.01.2018 11:31:04,947 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/1 Space instance is = Primary 

After some relocation = of ps-main instances a new backup instance is started on the same = GSC. 

24.01.2018 = 11:32:59,886 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T146 SPACE/GS-LRMI Connection-pool-1-thread-6/1 Space instance [ frqMain_container1:frqMain ] has = been selected as Backup 

Later a failover test = is performed by shutting down WSOSTSLXBMS02 and = WSOSTSLXMSS02. 
We would expect that ps-main.1 backup = will be elected as primary because the former primary was located at = WSOSTSLXBMS02 but no election happend. In the end in the whole cluster = there was no primary of ps-main.1 running. 

We found suspicious exception in the = logs: 
24.01.2018 11:32:44,987 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/frqMain_container1:frqMain Lost leadership = [sessionId=3D0x36128253b820001] 
24.01.2018 = 11:32:44,987 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T98 SPACE/Curator-Framework-0/CuratorFrameworkImpl backgroundOperationsLoop = exiting 
24.01.2018 11:32:44,990 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/1 Quiesce state set to = SUSPENDED 
24.01.2018 11:32:44,991 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/ZooKeeper Session: = 0x36128253b820001 closed 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector The leader threw an = exception 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector java.lang.IllegalStateException: = instance must be started before calling this = method 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:176) 
24.01.2018 = 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrameworkImpl.java:359) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:339) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:123) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) 
24.01.2018 = 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:449) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:466) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:65) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:246) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:240) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
24.01.2018 = 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
24.01.2018 = 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
24.01.2018 = 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector at java.lang.Thread.run(Thread.java:748) 
24.01.2018 11:32:44,991 +01:00 ERROR WSOSTSLXBMS01/GSC/3 P1280/T120 SPACE/Curator-LeaderSelector-0/LeaderSelector
24.01.2018 11:32:44,991 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T97 SPACE/GS-LRMI Connection-pool-1-thread-1-EventThread/ClientCnxn EventThread shut down for session: = 0x36128253b820001 
24.01.2018 11:32:44,997 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/frqMain1_1 Channel is closed = [target=3DfrqMain1_1, target url=3Djini://*/frqMain_container1_1/frqMain?groups=3DPOLBAY&ignoreValidation=3Dtrue&schema=3Dpersistent&id=3D1&total_members=3D4,1&versioned=3Dtrue&mirror=3Dtrue&cluster_schema=3Dpartitioned&locators=3DWSOSTSLXMSS01,WSOSTSLXMSS02&state=3Dstarted&timeout=3D5000, target machine = connection url=3DNIO://WSOSTSLXBMS02:11002/pid[3756]/24795468496_3_2538032931939568226_details[class com.gigaspaces.internal.cluster.node.impl.router.AbstractConnectionProxyBasedReplicationRouter$ConnectionEndpoint(frqMain_container1_1:frqMain)]] 
24.01.2018 = 11:32:44,998 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/ps-main-mirror-service Channel is closed [target=3Dps-main-mirror-service, target url=3Djini://*/ps-main-mirror-service_container/ps-main-mirror-service?schema=3Dpersistent&id=3D1&total_members=3D4,1&versioned=3Dtrue&mirror=3Dtrue&groups=3DPOLBAY&cluster_schema=3Dpartitioned&locators=3DWSOSTSLXMSS01,WSOSTSLXMSS02&state=3Dstarted&timeout=3D5000, target machine = connection url=3DNIO://WSOSTSLXBMS01:11002/pid[3220]/25906827215_3_7196393537085178059_details[class com.gigaspaces.internal.cluster.node.impl.router.AbstractConnectionProxyBasedReplicationRouter$ConnectionEndpoint(ps-main-mirror-service_container:ps-main-mirror-service)]] 
24.01.2018 = 11:32:45,767 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/1 Shutdown complete 
24.01.2018 = 11:32:45,768 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/container Container = <frqMain_container1> shutdown = completed 

24.01.2018 = 11:32:46,037 +01:00 WARN WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/PUServiceBeanImpl Failed to delete deployed processing unit from = [C:\Program Files\Frequentis\Frequentis LifeX = Platform\gigaspaces\work\processing-units\ps-main_1_2008048111] 
24.01.2018 11:32:46,037 +01:00 INFO WSOSTSLXBMS01/GSC/3 P1280/T24 SPACE/GS-LRMI Connection-pool-1-thread-1/PUServiceBeanImpl Stopped 

Attached thread dump. Failover happend = around 24.01.2018 11:40:20,893 +01:00
WARN WSOSTSLXBMS01/GSC/3 P1280/T59 SPACE/GS-Watchdog/watchdog The ServerEndPoint [WSOSTSLXMSS02/10.17.46.142:11001] is not reachable or = is reachable but with no matching invocation in progress at the server = peer, closing invalid connection with local address [/10.17.46.136:49864] [ remote invocation of: = net.jini.lookup.DiscoveryAdmin.getMemberGroups] [java.net.SocketTimeoutException]


Could it be ZK = bug?

Thanks,

Yuval

= --Apple-Mail=_F234B00D-9523-440D-AC4E-721E3FE9F9C4 Content-Disposition: attachment; filename=threads.txt Content-Type: text/plain; name="threads.txt" Content-Transfer-Encoding: quoted-printable =3D=3D=3D=3D=3D Deadlocked Threads =3D=3D=3D=3D=3D=0D =0D =0D =0D =3D=3D=3D=3D=3D All Threads =3D=3D=3D=3D=3D=0D =0D Threads count:207=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-42" Id=3D275 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-41" Id=3D274 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-40" Id=3D273 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-39" Id=3D272 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-38" Id=3D271 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-37" Id=3D270 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-36" Id=3D269 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-35" Id=3D268 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-34" Id=3D267 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-33" Id=3D266 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-32" Id=3D265 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-31" Id=3D264 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-30" Id=3D263 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-29" Id=3D262 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-28" Id=3D261 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-27" Id=3D260 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-26" Id=3D259 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-25" Id=3D258 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-24" Id=3D257 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-23" Id=3D256 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D254 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D253 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D252 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-JiniWakeupManager" Id=3D251 TIMED_WAITING on = java.util.TreeSet@167b9195=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.WakeupManager$Kicker.run(WakeupManager.java:603)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-JiniWakeupManager" Id=3D242 TIMED_WAITING on = java.util.TreeSet@19188b62=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.WakeupManager$Kicker.run(WakeupManager.java:603)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D241 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D239 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D238 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D237 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D236 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D235 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D234 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "Event Lease Manager" Id=3D223 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62a8= 7812=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Event Lease Manager" Id=3D222 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62a8= 7812=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Event Lease Manager" Id=3D221 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62a8= 7812=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1081)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Event Lease Manager" Id=3D220 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62a8= 7812=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D219 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D218 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D217 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D216 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D215 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D214 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-state-change-thread" Id=3D213 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f2d= 68fe=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-ProvisionMonitorEventPool-pool-6-thread-5" Id=3D209 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2815= aaeb=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Monitoring Pool-pool-3-thread-2" Id=3D207 RUNNABLE=0D =0D at sun.management.ThreadImpl.dumpThreads0(Native Method)=0D =0D at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)=0D =0D at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)=0D =0D at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 62)=0D =0D at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:43)=0D =0D at java.lang.reflect.Method.invoke(Method.java:498)=0D =0D at = com.gigaspaces.internal.dump.thread.ThreadDumpProcessor.dumpAllThreads(Thr= eadDumpProcessor.java:110)=0D =0D at = com.gigaspaces.internal.dump.thread.ThreadDumpProcessor.processAllThreads(= ThreadDumpProcessor.java:76)=0D =0D at = com.gigaspaces.internal.dump.thread.ThreadDumpProcessor.process(ThreadDump= Processor.java:42)=0D =0D at = com.gigaspaces.internal.dump.InternalSimpleDumpGenerator.generateDump(Inte= rnalSimpleDumpGenerator.java:74)=0D =0D at = com.gigaspaces.internal.dump.InternalDumpHelper.generateDump(InternalDumpH= elper.java:55)=0D =0D at com.gigaspaces.grid.gsm.GSMImpl.generateDump(GSMImpl.java:558)=0D =0D at = com.gigaspaces.internal.dump.InternalDumpProviderGigaspacesMethodinternalI= nvoke1.internalInvoke(Unknown Source)=0D =0D at = com.gigaspaces.internal.reflection.fast.AbstractMethod.invoke(AbstractMeth= od.java:45)=0D =0D at com.gigaspaces.lrmi.LRMIRuntime.invoked(LRMIRuntime.java:441)=0D =0D at = com.gigaspaces.lrmi.nio.Pivot.consumeAndHandleRequest(Pivot.java:494)=0D =0D at com.gigaspaces.lrmi.nio.Pivot.handleRequest(Pivot.java:572)=0D =0D at = com.gigaspaces.lrmi.nio.Pivot$ChannelEntryTask.run(Pivot.java:173)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D206 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D205 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1093)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D203 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D202 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D200 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-ProvisionMonitorEventPool-pool-6-thread-3" Id=3D199 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2815= aaeb=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D198 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D197 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-ProvisionMonitorEventPool-pool-6-thread-2" Id=3D196 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2815= aaeb=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D193 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D191 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D190 TIMED_WAITING on = com.sun.jini.thread.TaskManager@466682b8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "Fault Detection Handler" Id=3D189 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Fault Detection Handler" Id=3D185 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758b= 6710=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Event Lease Manager" Id=3D184 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62a8= 7812=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "RecvWorker:2" Id=3D181 RUNNABLE (in native)=0D =0D at java.net.SocketInputStream.socketRead0(Native Method)=0D =0D at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)=0D= =0D at java.net.SocketInputStream.read(SocketInputStream.java:171)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:141)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:224)=0D =0D at java.io.DataInputStream.readInt(DataInputStream.java:387)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumC= nxManager.java:795)=0D =0D =0D =0D "SendWorker:2" Id=3D180 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78a5= 85b=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)=0D= =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCn= xManager.java:879)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxMa= nager.java:65)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumC= nxManager.java:715)=0D =0D =0D =0D "SyncThread:1" Id=3D178 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2643= d850=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.= java:127)=0D =0D =0D =0D "FollowerRequestProcessor:1" Id=3D177 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2304= 7c81=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRe= questProcessor.java:60)=0D =0D =0D =0D "CommitProcessor:1" Id=3D176 WAITING on = org.apache.zookeeper.server.quorum.CommitProcessor@d993c16=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.jav= a:83)=0D =0D =0D =0D "RecvWorker:3" Id=3D175 RUNNABLE (in native)=0D =0D at java.net.SocketInputStream.socketRead0(Native Method)=0D =0D at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)=0D= =0D at java.net.SocketInputStream.read(SocketInputStream.java:171)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:141)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:224)=0D =0D at java.io.DataInputStream.readInt(DataInputStream.java:387)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumC= nxManager.java:795)=0D =0D =0D =0D "SendWorker:3" Id=3D174 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5884= e57e=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)=0D= =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCn= xManager.java:879)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxMa= nager.java:65)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumC= nxManager.java:715)=0D =0D =0D =0D "GS-LeaseRenewalManager Task" Id=3D173 TIMED_WAITING on = com.sun.jini.thread.TaskManager@60bd2766=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "Curator-LeaderSelector-0-EventThread" Id=3D172 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@60dd= bf84=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)=0D =0D =0D =0D "Curator-LeaderSelector-0-SendThread(wsostslxwit01.polbay.frq:2181)" = Id=3D171 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@618681861=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.j= ava:349)=0D =0D at = org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)=0D =0D =0D =0D "pool-13-thread-2" Id=3D170 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5f93= c6b9=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1093)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "AMQP Connection 127.0.0.1:5672" Id=3D169 RUNNABLE (in native)=0D =0D at java.net.SocketInputStream.socketRead0(Native Method)=0D =0D at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)=0D= =0D at java.net.SocketInputStream.read(SocketInputStream.java:171)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:141)=0D =0D at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)=0D =0D at java.io.BufferedInputStream.read(BufferedInputStream.java:265)=0D =0D - locked java.io.BufferedInputStream@2036724528=0D =0D at = java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)=0D =0D at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)=0D =0D at = com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.j= ava:139)=0D =0D - locked java.io.DataInputStream@634315654=0D =0D at = com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:538= )=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Scheduled-System-Boot-Thread" Id=3D166 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at com.gigaspaces.start.SystemBoot$3.run(SystemBoot.java:440)=0D =0D =0D =0D "qtp97901029-165" Id=3D165 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c4f9= 500=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342= )=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPoo= l.java:526)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool= .java:44)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :572)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-164" Id=3D164 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c4f9= 500=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342= )=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPoo= l.java:526)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool= .java:44)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :572)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-163" Id=3D163 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c4f9= 500=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342= )=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPoo= l.java:526)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool= .java:44)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :572)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-162" Id=3D162 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c4f9= 500=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342= )=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPoo= l.java:526)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool= .java:44)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :572)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-161 Acceptor1 SelectChannelConnector@WSOSTSLXMSS01:8091" = Id=3D161 BLOCKED on java.lang.Object@4da82c3e owned by "qtp97901029-160 = Acceptor0 SelectChannelConnector@WSOSTSLXMSS01:8091" Id=3D160=0D =0D at = sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234= )=0D =0D at = org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelCo= nnector.java:109)=0D =0D at = org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.= java:938)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.jav= a:608)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :543)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-160 Acceptor0 SelectChannelConnector@WSOSTSLXMSS01:8091" = Id=3D160 RUNNABLE (in native)=0D =0D at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)=0D =0D at = sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422= )=0D =0D at = sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250= )=0D =0D - locked java.lang.Object@1302867006=0D =0D at = org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelCo= nnector.java:109)=0D =0D at = org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.= java:938)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.jav= a:608)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :543)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-159 Selector1" Id=3D159 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@453163720=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManage= r.java:569)=0D =0D at = org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)=0D= =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.jav= a:608)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :543)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "qtp97901029-158 Selector0" Id=3D158 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@894488968=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManage= r.java:569)=0D =0D at = org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)=0D= =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.jav= a:608)=0D =0D at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :543)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-scheduled-executor-thread" Id=3D157 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708b= ebf5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-scheduled-executor-thread" Id=3D156 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708b= ebf5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1093)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D155 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D154 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D153 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D152 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D151 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D150 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D149 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D148 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D147 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-admin-scheduled-executor-thread" Id=3D146 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708b= ebf5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-admin-scheduled-executor-thread" Id=3D145 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708b= ebf5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-SDM Cache Task" Id=3D144 TIMED_WAITING on = com.sun.jini.thread.TaskManager@43e4b8fd=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-admin-scheduled-executor-thread" Id=3D143 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@708b= ebf5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1088)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "HashSessionScavenger-0" Id=3D142 TIMED_WAITING on = java.util.TaskQueue@2fca0be8=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.util.TimerThread.mainLoop(Timer.java:552)=0D =0D at java.util.TimerThread.run(Timer.java:505)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-22" Id=3D140 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-21" Id=3D139 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-20" Id=3D138 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-19" Id=3D137 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-18" Id=3D136 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-17" Id=3D135 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-16" Id=3D134 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-15" Id=3D133 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GSM-PendingManagerTask" Id=3D132 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = org.jini.rio.monitor.ServiceProvisioner$PendingManager$ProcessTask.run(Ser= viceProvisioner.java:1235)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-14" Id=3D131 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-13" Id=3D130 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-12" Id=3D129 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-11" Id=3D128 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-10" Id=3D127 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-9" Id=3D126 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-8" Id=3D125 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-7" Id=3D124 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-6" Id=3D123 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-5" Id=3D122 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-4" Id=3D120 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-3" Id=3D119 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LeaseRenewalManager Task" Id=3D116 TIMED_WAITING on = net.jini.lease.LeaseRenewalManager@57ffd856=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = net.jini.lease.LeaseRenewalManager$QueuerTask.run(LeaseRenewalManager.java= :1438)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:423)=0D =0D =0D =0D "Thread-5" Id=3D102 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = net.jini.discovery.LookupLocatorDiscovery$LookupAliveTask.run(LookupLocato= rDiscovery.java:1674)=0D =0D =0D =0D "GS-Watchdog" Id=3D114 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at com.gigaspaces.lrmi.nio.watchdog.Watchdog.run(Watchdog.java:189)=0D= =0D =0D =0D "Timer-1" Id=3D115 WAITING on java.util.TaskQueue@7884551f=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at java.util.TimerThread.mainLoop(Timer.java:526)=0D =0D at java.util.TimerThread.run(Timer.java:505)=0D =0D =0D =0D "SG LeaseReaper" Id=3D113 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = org.jini.rio.resources.servicecore.ResourceLessor$LeaseReaper.run(Resource= Lessor.java:246)=0D =0D =0D =0D "SG LeaseReaper" Id=3D112 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = org.jini.rio.resources.servicecore.ResourceLessor$LeaseReaper.run(Resource= Lessor.java:246)=0D =0D =0D =0D "GS-TimeoutObserverExecutorService-ObserverThread" Id=3D111 = TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = com.j_spaces.kernel.threadpool.monitor.TimeoutObserverExecutorService$Obse= rverThread.run(TimeoutObserverExecutorService.java:61)=0D =0D =0D =0D "Provision Clock:writer" Id=3D110 TIMED_WAITING on = java.util.Collections$SynchronizedList@1ec129a2=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = org.jini.rio.watch.WatchDataSourceImpl$Writer.run(WatchDataSourceImpl.java= :358)=0D =0D =0D =0D "GS-LRMI-async-client-connection-Thread-" Id=3D109 RUNNABLE (in native)=0D= =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1638659289=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.client.ClientConversationRunner.d= oSelect(ClientConversationRunner.java:66)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.client.ClientConversationRunner.r= un(ClientConversationRunner.java:58)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-async-Selector-Thread-3" Id=3D107 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@89293493=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-async-Selector-Thread-2" Id=3D106 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1438426691=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-async-Selector-Thread-1" Id=3D105 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1380191314=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-async-Selector-Thread-0" Id=3D103 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@283664792=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "SG LeaseReaper" Id=3D100 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = org.jini.rio.resources.servicecore.ResourceLessor$LeaseReaper.run(Resource= Lessor.java:246)=0D =0D =0D =0D "ProvisionFailurePool:1517484897959-15" Id=3D99 WAITING on = java.lang.Object@6f678595=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-14" Id=3D98 WAITING on = java.lang.Object@3c50cd40=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-13" Id=3D97 WAITING on = java.lang.Object@38a23d10=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-12" Id=3D96 WAITING on = java.lang.Object@7996f14f=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-11" Id=3D95 WAITING on = java.lang.Object@2a8e98c5=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-10" Id=3D94 WAITING on = java.lang.Object@108e6a19=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-9" Id=3D93 WAITING on = java.lang.Object@39ed9fa5=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionFailurePool:1517484897959-8" Id=3D92 WAITING on = java.lang.Object@773f0d5e=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-7" Id=3D91 WAITING on = java.lang.Object@2929ebcc=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-6" Id=3D90 WAITING on = java.lang.Object@4a948252=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-5" Id=3D89 WAITING on = java.lang.Object@1fd13c12=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-4" Id=3D88 WAITING on = java.lang.Object@108b874b=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-3" Id=3D87 WAITING on = java.lang.Object@761a691d=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-2" Id=3D86 WAITING on = java.lang.Object@18f2f39e=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-1" Id=3D85 WAITING on = java.lang.Object@ed56d3e=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "ProvisionPool:1517484897939-0" Id=3D84 WAITING on = java.lang.Object@42a80270=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at = org.jini.rio.resources.resource.PoolableThread.run(PoolableThread.java:90)= =0D =0D =0D =0D "GS-LookupDiscovery Task" Id=3D80 TIMED_WAITING on = com.sun.jini.thread.TaskManager@675fc9b9=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-multicast-announcement-timer" Id=3D77 TIMED_WAITING on = net.jini.discovery.LookupDiscovery$AnnouncementTimerThread@68bea8cf=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = net.jini.discovery.LookupDiscovery$AnnouncementTimerThread.run(LookupDisco= very.java:1185)=0D =0D =0D =0D "GS-multicast-discovery-announcement-listener" Id=3D76 RUNNABLE (in = native)=0D =0D at java.net.TwoStacksPlainDatagramSocketImpl.peekData(Native Method)=0D= =0D - locked java.net.TwoStacksPlainDatagramSocketImpl@1302828155=0D =0D at java.net.DatagramSocket.receive(DatagramSocket.java:743)=0D =0D - locked java.net.DatagramPacket@1335796797=0D =0D at = net.jini.discovery.LookupDiscovery$AnnouncementListener.run(LookupDiscover= y.java:940)=0D =0D =0D =0D "Curator-LeaderSelector-0" Id=3D75 WAITING on = java.util.concurrent.Semaphore$NonfairSync@4bf08cab=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrup= t(AbstractQueuedSynchronizer.java:836)=0D =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInter= ruptibly(AbstractQueuedSynchronizer.java:997)=0D =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterru= ptibly(AbstractQueuedSynchronizer.java:1304)=0D =0D at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)=0D =0D at = org.openspaces.zookeeper.curator.XapLeaderSelector$InternalListener.takeLe= adership(XapLeaderSelector.java:124)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener= .takeLeadership(LeaderSelector.java:559)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSe= lector.java:421)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(Lead= erSelector.java:466)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(Lead= erSelector.java:65)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSe= lector.java:246)=0D =0D at = org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSe= lector.java:240)=0D =0D at java.util.concurrent.FutureTask.run(FutureTask.java:266)=0D =0D at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)=0D= =0D at java.util.concurrent.FutureTask.run(FutureTask.java:266)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Curator-Framework-0" Id=3D74 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a85= cde2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)=0D =0D at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)=0D =0D at = org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperation= sLoop(CuratorFrameworkImpl.java:804)=0D =0D at = org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorF= rameworkImpl.java:64)=0D =0D at = org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrame= workImpl.java:267)=0D =0D at java.util.concurrent.FutureTask.run(FutureTask.java:266)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces= s$201(ScheduledThreadPoolExecutor.java:180)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S= cheduledThreadPoolExecutor.java:293)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Curator-ConnectionStateManager-0" Id=3D71 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39c3= =0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)=0D= =0D at = org.apache.curator.framework.state.ConnectionStateManager.processEvents(Co= nnectionStateManager.java:245)=0D =0D at = org.apache.curator.framework.state.ConnectionStateManager.access$000(Conne= ctionStateManager.java:43)=0D =0D at = org.apache.curator.framework.state.ConnectionStateManager$1.call(Connectio= nStateManager.java:111)=0D =0D at java.util.concurrent.FutureTask.run(FutureTask.java:266)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "SG LeaseReaper" Id=3D70 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = org.jini.rio.resources.servicecore.ResourceLessor$LeaseReaper.run(Resource= Lessor.java:246)=0D =0D =0D =0D "QuorumPeer[myid=3D1]/0.0.0.0:2181" Id=3D1 RUNNABLE (in native)=0D =0D at java.net.SocketInputStream.socketRead0(Native Method)=0D =0D at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)=0D= =0D at java.net.SocketInputStream.read(SocketInputStream.java:171)=0D =0D at java.net.SocketInputStream.read(SocketInputStream.java:141)=0D =0D at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)=0D =0D at java.io.BufferedInputStream.read(BufferedInputStream.java:265)=0D =0D - locked java.io.BufferedInputStream@2073903531=0D =0D at java.io.DataInputStream.readInt(DataInputStream.java:387)=0D =0D at = org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)=0D= =0D at = org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.j= ava:83)=0D =0D at = org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:99)=0D= =0D at = org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)=0D= =0D - locked org.apache.jute.BinaryInputArchive@885464999=0D =0D at = org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)= =0D =0D at = org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:846)=0D= =0D =0D =0D "WorkerReceiver[myid=3D1]" Id=3D68 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2039= 2e00=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)=0D= =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager.pollRecvQueue(QuorumCn= xManager.java:931)=0D =0D at = org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerRece= iver.run(FastLeaderElection.java:247)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "WorkerSender[myid=3D1]" Id=3D66 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@21f6= c5e5=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSend= er.run(FastLeaderElection.java:430)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "WSOSTSLXMSS01/10.17.46.141:3888" Id=3D64 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)=0D =0D at = java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)=0D= =0D at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)=0D =0D - locked java.net.SocksSocketImpl@813123689=0D =0D at java.net.ServerSocket.implAccept(ServerSocket.java:545)=0D =0D at java.net.ServerSocket.accept(ServerSocket.java:513)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnx= Manager.java:539)=0D =0D =0D =0D "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181" Id=3D62 RUNNABLE (in native)=0D= =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1010914279=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.= java:173)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "PurgeTask" Id=3D61 TIMED_WAITING on java.util.TaskQueue@481121c9=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.util.TimerThread.mainLoop(Timer.java:552)=0D =0D at java.util.TimerThread.run(Timer.java:505)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-2" Id=3D60 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Timer-0" Id=3D58 WAITING on java.util.TaskQueue@7ef98d47=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at java.util.TimerThread.mainLoop(Timer.java:526)=0D =0D at java.util.TimerThread.run(Timer.java:505)=0D =0D =0D =0D "GS-Webster Runner" Id=3D57 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)=0D =0D at = java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)=0D= =0D at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)=0D =0D - locked java.net.SocksSocketImpl@576639714=0D =0D at java.net.ServerSocket.implAccept(ServerSocket.java:545)=0D =0D at java.net.ServerSocket.accept(ServerSocket.java:513)=0D =0D at org.jini.rio.tools.webster.Webster.run(Webster.java:548)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "pool-8-thread-1" Id=3D56 WAITING on = org.apache.zookeeper.server.quorum.QuorumPeer@182423e9=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Thread.join(Thread.java:1252)=0D =0D at java.lang.Thread.join(Thread.java:1326)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeer= Main.java:154)=0D =0D at = org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumP= eerMain.java:111)=0D =0D at = org.openspaces.zookeeper.grid.XapZookeeperContainer$ZookeeperQuorumWrapper= .start(XapZookeeperContainer.java:118)=0D =0D at = org.openspaces.zookeeper.grid.XapZookeeperContainer$1.run(XapZookeeperCont= ainer.java:56)=0D =0D at = java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)=0D= =0D at java.util.concurrent.FutureTask.run(FutureTask.java:266)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LookupDiscovery Task" Id=3D54 TIMED_WAITING on = com.sun.jini.thread.TaskManager@597a564d=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.thread.TaskManager$TaskThread.run(TaskManager.java:408)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-11" Id=3D53 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-discovery announcement" Id=3D41 TIMED_WAITING on = com.sun.jini.reggie.GigaRegistrar$AnnounceThread@3fc67dcb=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = com.sun.jini.reggie.GigaRegistrar$AnnounceThread.run(GigaRegistrar.java:22= 45)=0D =0D =0D =0D "GS-multicast request" Id=3D40 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainDatagramSocketImpl.peekData(Native Method)=0D= =0D - locked java.net.TwoStacksPlainDatagramSocketImpl@113523377=0D =0D at java.net.DatagramSocket.receive(DatagramSocket.java:743)=0D =0D - locked java.net.DatagramPacket@227007514=0D =0D at = com.sun.jini.reggie.GigaRegistrar$MulticastThread.run(GigaRegistrar.java:2= 021)=0D =0D =0D =0D "GS-unicast request" Id=3D39 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)=0D =0D at = java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)=0D= =0D at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)=0D =0D - locked java.net.SocksSocketImpl@831798153=0D =0D at java.net.ServerSocket.implAccept(ServerSocket.java:545)=0D =0D at java.net.ServerSocket.accept(ServerSocket.java:513)=0D =0D at = com.sun.jini.reggie.GigaRegistrar$UnicastThread.run(GigaRegistrar.java:214= 2)=0D =0D =0D =0D "GS-event expire" Id=3D38 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = com.sun.jini.reggie.GigaRegistrar$EventExpireThread.run(GigaRegistrar.java= :1850)=0D =0D =0D =0D "GS-service expire" Id=3D37 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = com.sun.jini.reggie.GigaRegistrar$ServiceExpireThread.run(GigaRegistrar.ja= va:1755)=0D =0D =0D =0D "GS-LRMI-Selector-Accept-Thread-11000-13000" Id=3D50 RUNNABLE (in = native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@371367608=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Write-Thread-3" Id=3D49 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@162319608=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Read-Thread-3" Id=3D48 RUNNABLE=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@667774229=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Write-Thread-2" Id=3D47 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@629559154=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Read-Thread-2" Id=3D46 RUNNABLE=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1712342662=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Write-Thread-1" Id=3D45 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@324441748=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Read-Thread-1" Id=3D44 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1594708841=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Write-Thread-0" Id=3D43 RUNNABLE (in native)=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@638638454=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI-Selector-Read-Thread-0" Id=3D42 RUNNABLE=0D =0D at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)=0D= =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:2= 96)=0D =0D at = sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.= java:278)=0D =0D at = sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)=0D =0D at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)=0D =0D - locked sun.nio.ch.Util$3@1783646818=0D =0D at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(A= bstractSelectorThread.java:70)=0D =0D at = com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(Abstra= ctSelectorThread.java:61)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-10" Id=3D36 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-9" Id=3D35 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-8" Id=3D34 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-6" Id=3D32 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-5" Id=3D31 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-4" Id=3D30 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-3" Id=3D29 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-2" Id=3D28 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-Reggie Comm Task-pool-5-thread-1" Id=3D27 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bdb= 7b90=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-multicast-announcement-timer" Id=3D26 TIMED_WAITING on = net.jini.discovery.LookupDiscovery$AnnouncementTimerThread@5911aa67=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at = net.jini.discovery.LookupDiscovery$AnnouncementTimerThread.run(LookupDisco= very.java:1185)=0D =0D =0D =0D "GS-multicast-discovery-announcement-listener" Id=3D25 RUNNABLE (in = native)=0D =0D at java.net.TwoStacksPlainDatagramSocketImpl.peekData(Native Method)=0D= =0D - locked java.net.TwoStacksPlainDatagramSocketImpl@830138493=0D =0D at java.net.DatagramSocket.receive(DatagramSocket.java:743)=0D =0D - locked java.net.DatagramPacket@611131704=0D =0D at = net.jini.discovery.LookupDiscovery$AnnouncementListener.run(LookupDiscover= y.java:940)=0D =0D =0D =0D "GS-LRMI Custom Pool-pool-4-thread-1" Id=3D24 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@ad8b= ed4=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Monitoring Pool-pool-3-thread-1" Id=3D23 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6c87= 86f2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Liveness Pool-pool-2-thread-1" Id=3D22 WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@21ae= 0ebb=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= t(AbstractQueuedSynchronizer.java:2039)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-LRMI Connection-pool-1-thread-1" Id=3D21 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22ff= bbd2=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467= )=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 73)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "GS-ClassLoaderCache-SelfCleaningTable$Cleaner" Id=3D20 WAITING on = java.lang.ref.ReferenceQueue$Lock@c624ed3=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)=0D =0D at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)=0D =0D at = com.gigaspaces.internal.utils.collections.SelfCleaningTable$Cleaner.run(Se= lfCleaningTable.java:288)=0D =0D =0D =0D "RMI TCP Accept-0" Id=3D19 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)=0D =0D at = java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)=0D= =0D at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)=0D =0D - locked java.net.SocksSocketImpl@1132691349=0D =0D at java.net.ServerSocket.implAccept(ServerSocket.java:545)=0D =0D at java.net.ServerSocket.accept(ServerSocket.java:513)=0D =0D at = sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServ= erSocketFactory.java:52)=0D =0D at = sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspo= rt.java:400)=0D =0D at = sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)=0D= =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "RMI TCP Accept-10104" Id=3D18 RUNNABLE (in native)=0D =0D at java.net.TwoStacksPlainSocketImpl.socketAccept(Native Method)=0D =0D at = java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)=0D= =0D at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)=0D =0D - locked java.net.SocksSocketImpl@325304577=0D =0D at java.net.ServerSocket.implAccept(ServerSocket.java:545)=0D =0D at java.net.ServerSocket.accept(ServerSocket.java:513)=0D =0D at = sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTranspo= rt.java:400)=0D =0D at = sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)=0D= =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Log4j2-TF-5-Scheduled-4" Id=3D17 TIMED_WAITING on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42e9= de00=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:1093)=0D =0D at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:10= 74)=0D =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1134)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Log4j2-TF-2-AsyncLogger[AsyncContext@764c12b6]-2" Id=3D15 TIMED_WAITING = on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ce3= 0458=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWait= Strategy.java:38)=0D =0D at = com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBar= rier.java:56)=0D =0D at = com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:124)=0D= =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Log4j2-TF-1-AsyncLogger[AsyncContext@764c12b6]-1" Id=3D14 TIMED_WAITING = on = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@444b= 786=0D =0D at sun.misc.Unsafe.park(Native Method)=0D =0D at = java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)=0D= =0D at = java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078)=0D =0D at = com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWait= Strategy.java:38)=0D =0D at = com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBar= rier.java:56)=0D =0D at = com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:124)=0D= =0D at = java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:= 1149)=0D =0D at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :624)=0D =0D at java.lang.Thread.run(Thread.java:748)=0D =0D =0D =0D "Attach Listener" Id=3D6 RUNNABLE=0D =0D =0D =0D "Signal Dispatcher" Id=3D5 RUNNABLE=0D =0D =0D =0D "Finalizer" Id=3D3 WAITING on java.lang.ref.ReferenceQueue$Lock@9a215c6=0D= =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)=0D =0D at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)=0D =0D at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)=0D= =0D =0D =0D "Reference Handler" Id=3D2 WAITING on = java.lang.ref.Reference$Lock@537c32c=0D =0D at java.lang.Object.wait(Native Method)=0D =0D at java.lang.Object.wait(Object.java:502)=0D =0D at java.lang.ref.Reference.tryHandlePending(Reference.java:191)=0D =0D at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)=0D= =0D =0D =0D "main" Id=3D1 TIMED_WAITING=0D =0D at java.lang.Thread.sleep(Native Method)=0D =0D at = com.gigaspaces.start.SystemBoot.waitForStopCommand(SystemBoot.java:421)=0D= =0D at com.gigaspaces.start.SystemBoot.mainAgent(SystemBoot.java:356)=0D =0D at com.gigaspaces.start.SystemBoot.main(SystemBoot.java:312)=0D =0D --Apple-Mail=_F234B00D-9523-440D-AC4E-721E3FE9F9C4 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii

--Apple-Mail=_F234B00D-9523-440D-AC4E-721E3FE9F9C4-- --Apple-Mail=_2DBC7B5E-CEA2-4664-88C6-52912FBBFBCC--