flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink checkpointing gets stuck
Date Tue, 07 Mar 2017 18:10:17 GMT
Great to hear it!

What do you think about adding a section to the Flink docs about deployment
on Azure (there is already AWS and GCE, so Azure would make the
cloud-trinity complete) that explains how to set this up and avoid such
pitfalls.

https://ci.apache.org/projects/flink/flink-docs-release-1.2/

If there is anything you think can be improved in the dependencies to help
there, please let us know!


On Tue, Mar 7, 2017 at 5:31 PM, Avihai Berkovitz <
Avihai.Berkovitz@microsoft.com> wrote:

> For anyone seeing this thread in the future, we managed to solve the
> issue. The problem was in the Azure storage SDK.
>
> Flink is using Hadoop 2.7, so we used version 2.7.3 of the Hadoop-azure
> package. This package uses version 2.0.0 of the azure-storage package,
> dated from 2014. It has several bugs that were since fixed, specifically
> one where the socket timeout was infinite. We updated this package to
> version 5.0.0 and everything is working smoothly now.
>
>
>
> *From:* Stephan Ewen [mailto:sewen@apache.org]
> *Sent:* Sunday, February 26, 2017 4:47 PM
> *To:* user@flink.apache.org
> *Subject:* Re: Flink checkpointing gets stuck
>
>
>
> Thanks!
>
>
>
> This looks like a bigger example, involving MongoDB, etc.
>
> Are you able to reproduce this issue with a smaller example?
>
>
>
> It would also help to understand the problem better if we knew the
> topology a bit better.
>
> The stack traces look like "phase 1&2" want to send data (but are back
> pressured) and "phase 3&4&5" wait for input data.
>
>
>
> Stephan
>
>
>
>
>
> On Sun, Feb 26, 2017 at 12:30 PM, Shai Kaplan <Shai.Kaplan@microsoft.com>
> wrote:
>
> Running jstack on one of the Task Managers:
>
>
>
> 2017-02-26 10:06:27
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode):
>
>
>
> "Attach Listener" #6414 daemon prio=9 os_prio=0 tid=0x00007f3c8c089000
> nid=0xe692 waiting on condition [0x0000000000000000]
>
>    java.lang.Thread.State: RUNNABLE
>
>
>
> "Async calls on Sink: phase 5 (32/48)" #2337 daemon prio=5 os_prio=0
> tid=0x00007f3b942fc000 nid=0xb0d5 waiting on condition [0x00007f3adf0af000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000620> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (31/48)" #2336 daemon prio=5 os_prio=0
> tid=0x00007f3b942fb000 nid=0xb0d4 waiting on condition [0x00007f3adf1b0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd7e70> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (30/48)" #2335 daemon prio=5 os_prio=0
> tid=0x00007f3b942f9800 nid=0xb0d3 waiting on condition [0x00007f3adf2b1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdde8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (29/48)" #2334 daemon prio=5 os_prio=0
> tid=0x00007f3b942f8800 nid=0xb0d2 waiting on condition [0x00007f3adf3b2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e0003e8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (28/48)" #2333 daemon prio=5 os_prio=0
> tid=0x00007f3b942f7000 nid=0xb0d1 waiting on condition [0x00007f3adf4b3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8008> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (27/48)" #2332 daemon prio=5 os_prio=0
> tid=0x00007f3b942f6000 nid=0xb0d0 waiting on condition [0x00007f3adf5b4000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000730> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (26/48)" #2331 daemon prio=5 os_prio=0
> tid=0x00007f3b942f4800 nid=0xb0cf waiting on condition [0x00007f3adf6b5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07cdf80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (25/48)" #2330 daemon prio=5 os_prio=0
> tid=0x00007f3b942f3000 nid=0xb0ce waiting on condition [0x00007f3adf7b6000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000a78> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (24/48)" #2329 daemon prio=5 os_prio=0
> tid=0x00007f3b942f1800 nid=0xb0cd waiting on condition [0x00007f3adf8b7000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd81a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (23/48)" #2328 daemon prio=5 os_prio=0
> tid=0x00007f3b942f0800 nid=0xb0cc waiting on condition [0x00007f3adf9b8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000968> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (22/48)" #2327 daemon prio=5 os_prio=0
> tid=0x00007f3b942ef000 nid=0xb0cb waiting on condition [0x00007f3adfab9000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce118> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (21/48)" #2326 daemon prio=5 os_prio=0
> tid=0x00007f3b942ee000 nid=0xb0ca waiting on condition [0x00007f3adfbba000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8338> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (20/48)" #2325 daemon prio=5 os_prio=0
> tid=0x00007f3b942ec800 nid=0xb0c9 waiting on condition [0x00007f3adfcbb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000cb0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (19/48)" #2324 daemon prio=5 os_prio=0
> tid=0x00007f3b942eb000 nid=0xb0c8 waiting on condition [0x00007f3adfdbc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce2b0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (18/48)" #2323 daemon prio=5 os_prio=0
> tid=0x00007f3b942e9800 nid=0xb0c7 waiting on condition [0x00007f3adfebd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd84d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on Sink: phase 5 (17/48)" #2322 daemon prio=5 os_prio=0
> tid=0x00007f3b942e8800 nid=0xb0c6 waiting on condition [0x00007f3adffbe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e000dc0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (32/48)" #2321 daemon prio=5 os_prio=0
> tid=0x00007f3b94257800 nid=0xb0c5 waiting on condition [0x00007f3ae00bf000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d000ff8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (31/48)" #2320 daemon prio=5 os_prio=0
> tid=0x00007f3b94256000 nid=0xb0c4 waiting on condition [0x00007f3ae01c0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce448> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (30/48)" #2319 daemon prio=5 os_prio=0
> tid=0x00007f3b94254800 nid=0xb0c3 waiting on condition [0x00007f3ae02c1000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8668> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (29/48)" #2318 daemon prio=5 os_prio=0
> tid=0x00007f3b94253000 nid=0xb0c2 waiting on condition [0x00007f3ae03c2000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001340> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (28/48)" #2317 daemon prio=5 os_prio=0
> tid=0x00007f3b94251800 nid=0xb0c1 waiting on condition [0x00007f3ae04c3000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001108> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (27/48)" #2316 daemon prio=5 os_prio=0
> tid=0x00007f3b94250000 nid=0xb0c0 waiting on condition [0x00007f3ae06c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce5e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (26/48)" #2315 daemon prio=5 os_prio=0
> tid=0x00007f3b9424e800 nid=0xb0bf waiting on condition [0x00007f3ae09c8000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8800> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (25/48)" #2314 daemon prio=5 os_prio=0
> tid=0x00007f3b9424d800 nid=0xb0be waiting on condition [0x00007f3ae0ccb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001450> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (24/48)" #2313 daemon prio=5 os_prio=0
> tid=0x00007f3b9424c000 nid=0xb0bd waiting on condition [0x00007f3ae0dcc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001688> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (23/48)" #2312 daemon prio=5 os_prio=0
> tid=0x00007f3b9424a800 nid=0xb0bc waiting on condition [0x00007f3ae0ecd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (22/48)" #2311 daemon prio=5 os_prio=0
> tid=0x00007f3b94249000 nid=0xb0bb waiting on condition [0x00007f3ae0fce000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d0019d0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (21/48)" #2310 daemon prio=5 os_prio=0
> tid=0x00007f3b94247800 nid=0xb0ba waiting on condition [0x00007f3ae31f0000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8998> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (20/48)" #2309 daemon prio=5 os_prio=0
> tid=0x00007f3b94246800 nid=0xb0b9 waiting on condition [0x00007f3ae3cfb000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3da07ce778> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (19/48)" #2308 daemon prio=5 os_prio=0
> tid=0x00007f3b94245800 nid=0xb0b8 waiting on condition [0x00007f3ae3efd000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9e001ae0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (18/48)" #2307 daemon prio=5 os_prio=0
> tid=0x00007f3b94244000 nid=0xb0b7 waiting on condition [0x00007f3ae3ffe000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9d001d18> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 4 (17/48)" #2306 daemon prio=5 os_prio=0
> tid=0x00007f3b94242800 nid=0xb0b6 waiting on condition [0x00007f3af02c5000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.park(Native Method)
>
>               - parking to wait for  <0x00007f3d9fbd8b30> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>               at java.util.concurrent.locks.LockSupport.park(LockSupport.
> java:175)
>
>               at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>
>               at java.util.concurrent.LinkedBlockingQueue.take(
> LinkedBlockingQueue.java:442)
>
>               at java.util.concurrent.ThreadPoolExecutor.getTask(
> ThreadPoolExecutor.java:1067)
>
>               at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1127)
>
>               at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>
>               at java.lang.Thread.run(Thread.java:745)
>
>
>
> "Async calls on phase 3 (32/48)" #2305 daemon prio=5 os_prio=0
> tid=0x00007f3b9423e800 nid=0xb0b5 waiting on condition [0x00007f3af09cc000]
>
>    java.lang.Thread.State: WAITING (parking)
>
>               at sun.misc.Unsafe.pa
> <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsun.misc.Unsafe.pa&data=02%7C01%7CAvihai.Berkovitz%40microsoft.com%7Ca43b760ecbc0486ebaf008d45e565fdb%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636237172464130817&sdata=lSZZtCUQk8vQyHPap889W1DGCpTHi7wPFxi3VhhNFsw%3D&reserved=0>
>
> ...
>
> [Message clipped]
>
>
>

Mime
View raw message