Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2A84510B4E for ; Wed, 14 Aug 2013 02:55:30 +0000 (UTC) Received: (qmail 19840 invoked by uid 500); 14 Aug 2013 02:55:29 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 19801 invoked by uid 500); 14 Aug 2013 02:55:29 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 19786 invoked by uid 99); 14 Aug 2013 02:55:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Aug 2013 02:55:29 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates 209.85.217.176 as permitted sender) Received: from [209.85.217.176] (HELO mail-lb0-f176.google.com) (209.85.217.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Aug 2013 02:55:13 +0000 Received: by mail-lb0-f176.google.com with SMTP id w10so6292211lbi.7 for ; Tue, 13 Aug 2013 19:54:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=h4Ez6r4K67nqKaulAFlNsazh3SbTTsmC1C2P4VVDP3A=; b=I5b8gtng2yh4YVVJzgGmYwnTfi0OfGSK68t2nyIhNWTmrqFTxvAxVRNH0WzQNTQhKe 2IJ7PEXTb5ZBFH3kmk9XwDEyZpnX2joo5Ld89ZTjaktGm8kNV2QYqtbDP3lHdOCInb+z gos5XJdzylowSiO5Gw3p5XSZffWVJSt//S4srIkIvzO7DxL9QMWB3YMrmvqo9YISEalr epy8KlR2IK3DPASsRa/1ziSdhUfWxUra7zCBvBzzSXVavr/lZIQ1YUO5gf/OiPmH0PJ5 +RSLbgvkkjvX3Ry1NlcXJ+MC7WdJJS23Bfjt2E8HRRboar13TS7nCaWYzMPUgM1fhqp1 tnHg== X-Gm-Message-State: ALoCoQkFN+CA09Yf2+p+hblYAz2FSXMdQPkWts96vWFVZYdn9uEf6OkkvCvmmp4u4qH3doIBOfTg MIME-Version: 1.0 X-Received: by 10.152.44.225 with SMTP id h1mr6520943lam.15.1376448892309; Tue, 13 Aug 2013 19:54:52 -0700 (PDT) Received: by 10.112.69.38 with HTTP; Tue, 13 Aug 2013 19:54:51 -0700 (PDT) Received: by 10.112.69.38 with HTTP; Tue, 13 Aug 2013 19:54:51 -0700 (PDT) In-Reply-To: References: <56F5E94591A54564BAAF4195537DC8EA@cloudera.com> <0DCD20A2CAAC4A9BB3A1D1ECB9D33E0A@cloudera.com> Date: Tue, 13 Aug 2013 21:54:51 -0500 Message-ID: Subject: Re: Lock contention in FileChannel From: Brock Noland To: user@flume.apache.org Content-Type: multipart/alternative; boundary=089e0160b7be47233a04e3df7b34 X-Virus-Checked: Checked by ClamAV on apache.org --089e0160b7be47233a04e3df7b34 Content-Type: text/plain; charset=ISO-8859-1 Gotcha. When you run tge test what is tye disk utilization percentage? Iostat can be used for this. On Aug 13, 2013 9:47 PM, "Pankaj Gupta" wrote: > Those are the boxes we want to collect data from. They run flume and send > data through their avro sinks to the avro source on this box. We are > getting data at a pretty good rate and the problem is in fact that the > events don't drain from the FileChannel fast enough and the channel fill > percentage keeps getting higher. > > > On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland wrote: > >> What is sending the events to the avro source? >> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" wrote: >> >>> Here's the config: >>> # define channels, one for each disk >>> >>> >>> >>> >>> agent1.channels.ch1.type = FILE >>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint >>> agent1.channels.ch1.dataDirs = /flume1/data >>> agent1.channels.ch1.maxFileSize = 375809638400 >>> agent1.channels.ch1.capacity = 75000000 >>> agent1.channels.ch1.transactionCapacity = 4000 >>> >>> agent1.channels.ch2.type = FILE >>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint >>> agent1.channels.ch2.dataDirs = /flume2/data >>> agent1.channels.ch2.maxFileSize = 375809638400 >>> agent1.channels.ch2.capacity = 75000000 >>> agent1.channels.ch2.transactionCapacity = 4000 >>> >>> >>> >>> # Define an Avro source named avroSource1 >>> # Each sink can connect to only one channel. >>> # Connect it to channel ch1. Load balance it to 2 avroSinks >>> >>> >>> agent1.sources.avroSource1.channels = ch1 >>> agent1.sources.avroSource1.type = avro >>> agent1.sources.avroSource1.bind = 0.0.0.0 >>> agent1.sources.avroSource1.port = >>> >>> >>> >>> >>> agent1.sinks.avroSink1-1-1.type = avro >>> agent1.sinks.avroSink1-1-1.channel = ch1 >>> agent1.sinks.avroSink1-1-1.hostname = >>> agent1.sinks.avroSink1-1-1.port = >>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000 >>> agent1.sinks.avroSink1-1-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink1-2-1.type = avro >>> agent1.sinks.avroSink1-2-1.channel = ch1 >>> agent1.sinks.avroSink1-2-1.hostname = >>> agent1.sinks.avroSink1-2-1.port = >>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000 >>> agent1.sinks.avroSink1-2-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink1-3-1.type = avro >>> agent1.sinks.avroSink1-3-1.channel = ch1 >>> agent1.sinks.avroSink1-3-1.hostname = >>> agent1.sinks.avroSink1-3-1.port = >>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000 >>> agent1.sinks.avroSink1-3-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink1-4-1.type = avro >>> agent1.sinks.avroSink1-4-1.channel = ch1 >>> agent1.sinks.avroSink1-4-1.hostname = >>> agent1.sinks.avroSink1-4-1.port = >>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000 >>> agent1.sinks.avroSink1-4-1.batch-size = 4000 >>> >>> >>> >>> #Add the sink groups; load-balance between each group of sinks which >>> round robin between different hops >>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1 >>> avroSink1-3-1 avroSink1-4-1 >>> agent1.sinkgroups.group1.processor.type = load_balance >>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN >>> agent1.sinkgroups.group1.processor.backoff = true >>> >>> >>> #End of set >>> >>> # Define an Avro source named avroSource2 >>> # Each sink can connect to only one channel. >>> # Connect it to channel ch2. Load balance it to 2 avroSinks >>> >>> >>> agent1.sources.avroSource2.channels = ch2 >>> agent1.sources.avroSource2.type = avro >>> agent1.sources.avroSource2.bind = 0.0.0.0 >>> agent1.sources.avroSource2.port = >>> >>> >>> >>> >>> agent1.sinks.avroSink2-1-1.type = avro >>> agent1.sinks.avroSink2-1-1.channel = ch2 >>> agent1.sinks.avroSink2-1-1.hostname = >>> agent1.sinks.avroSink2-1-1.port = >>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000 >>> agent1.sinks.avroSink2-1-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink2-2-1.type = avro >>> agent1.sinks.avroSink2-2-1.channel = ch2 >>> agent1.sinks.avroSink2-2-1.hostname = >>> agent1.sinks.avroSink2-2-1.port = >>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000 >>> agent1.sinks.avroSink2-2-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink2-3-1.type = avro >>> agent1.sinks.avroSink2-3-1.channel = ch2 >>> agent1.sinks.avroSink2-3-1.hostname = >>> agent1.sinks.avroSink2-3-1.port = >>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000 >>> agent1.sinks.avroSink2-3-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink2-4-1.type = avro >>> agent1.sinks.avroSink2-4-1.channel = ch2 >>> agent1.sinks.avroSink2-4-1.hostname = >>> agent1.sinks.avroSink2-4-1.port = >>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000 >>> agent1.sinks.avroSink2-4-1.batch-size = 4000 >>> >>> >>> >>> #Add the sink groups; load-balance between each group of sinks which >>> round robin between different hops >>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1 >>> avroSink2-3-1 avroSink2-4-1 >>> agent1.sinkgroups.group2.processor.type = load_balance >>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN >>> agent1.sinkgroups.group2.processor.backoff = true >>> >>> >>> #End of set >>> >>> # Define an Avro source named avroSource3 >>> # Each sink can connect to only one channel. >>> # Connect it to channel ch1. Load balance it to 2 avroSinks >>> >>> >>> agent1.sources.avroSource3.channels = ch1 >>> agent1.sources.avroSource3.type = avro >>> agent1.sources.avroSource3.bind = 0.0.0.0 >>> agent1.sources.avroSource3.port = >>> >>> >>> >>> >>> agent1.sinks.avroSink3-1-1.type = avro >>> agent1.sinks.avroSink3-1-1.channel = ch1 >>> agent1.sinks.avroSink3-1-1.hostname = >>> agent1.sinks.avroSink3-1-1.port = >>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000 >>> agent1.sinks.avroSink3-1-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink3-2-1.type = avro >>> agent1.sinks.avroSink3-2-1.channel = ch1 >>> agent1.sinks.avroSink3-2-1.hostname = >>> agent1.sinks.avroSink3-2-1.port = >>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000 >>> agent1.sinks.avroSink3-2-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink3-3-1.type = avro >>> agent1.sinks.avroSink3-3-1.channel = ch1 >>> agent1.sinks.avroSink3-3-1.hostname = >>> agent1.sinks.avroSink3-3-1.port = >>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000 >>> agent1.sinks.avroSink3-3-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink3-4-1.type = avro >>> agent1.sinks.avroSink3-4-1.channel = ch1 >>> agent1.sinks.avroSink3-4-1.hostname = >>> agent1.sinks.avroSink3-4-1.port = >>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000 >>> agent1.sinks.avroSink3-4-1.batch-size = 4000 >>> >>> >>> >>> #Add the sink groups; load-balance between each group of sinks which >>> round robin between different hops >>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1 >>> avroSink3-3-1 avroSink3-4-1 >>> agent1.sinkgroups.group3.processor.type = load_balance >>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN >>> agent1.sinkgroups.group3.processor.backoff = true >>> >>> >>> #End of set >>> >>> # Define an Avro source named avroSource4 >>> # Each sink can connect to only one channel. >>> # Connect it to channel ch2. Load balance it to 2 avroSinks >>> >>> >>> agent1.sources.avroSource4.channels = ch2 >>> agent1.sources.avroSource4.type = avro >>> agent1.sources.avroSource4.bind = 0.0.0.0 >>> agent1.sources.avroSource4.port = >>> >>> >>> >>> >>> agent1.sinks.avroSink4-1-1.type = avro >>> agent1.sinks.avroSink4-1-1.channel = ch2 >>> agent1.sinks.avroSink4-1-1.hostname = >>> agent1.sinks.avroSink4-1-1.port = >>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000 >>> agent1.sinks.avroSink4-1-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink4-2-1.type = avro >>> agent1.sinks.avroSink4-2-1.channel = ch2 >>> agent1.sinks.avroSink4-2-1.hostname = >>> agent1.sinks.avroSink4-2-1.port = >>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000 >>> agent1.sinks.avroSink4-2-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink4-3-1.type = avro >>> agent1.sinks.avroSink4-3-1.channel = ch2 >>> agent1.sinks.avroSink4-3-1.hostname = >>> agent1.sinks.avroSink4-3-1.port = >>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000 >>> agent1.sinks.avroSink4-3-1.batch-size = 4000 >>> >>> >>> >>> >>> agent1.sinks.avroSink4-4-1.type = avro >>> agent1.sinks.avroSink4-4-1.channel = ch2 >>> agent1.sinks.avroSink4-4-1.hostname = >>> agent1.sinks.avroSink4-4-1.port = >>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000 >>> agent1.sinks.avroSink4-4-1.batch-size = 4000 >>> >>> >>> >>> #Add the sink groups; load-balance between each group of sinks which >>> round robin between different hops >>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1 >>> avroSink4-3-1 avroSink4-4-1 >>> agent1.sinkgroups.group4.processor.type = load_balance >>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN >>> agent1.sinkgroups.group4.processor.backoff = true >>> >>> >>> #End of set >>> >>> >>> >>> # Finally, now that we've defined all of our components, tell >>> # agent1 which ones we want to activate. >>> agent1.channels = ch1 ch2 >>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4 >>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4-1 >>> avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1 >>> avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1 >>> avroSink4-3-1 avroSink4-4-1 >>> agent1.sinkgroups = group1 group2 group3 group4 >>> >>> There are two disks on the box /flume1 and /flume2 >>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet. >>> >>> >>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland wrote: >>> >>>> Can you share your conf file? >>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" >>>> wrote: >>>> >>>>> Even though the writes are done per batch, they don't go to disk >>>>> rightaway - commits are the only ones which actually cause an fsync - which >>>>> is when writes actually go to disk. >>>>> >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote: >>>>> >>>>> Looking at the code it seems like the lock and the i/o is done per >>>>> event and not for a batch. Is that correct? If that is the case then it >>>>> seems like there is a lot of overhead per event. The throughput I'm seeing >>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write >>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't >>>>> help, they just block waiting for the queue to become free. CPU usage is >>>>> 20%, there is enough RAM for page cache so that no read is going to disk. >>>>> The queue seems to be the bottleneck. What is the throughput I should >>>>> expect per disk? >>>>> >>>>> >>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland wrote: >>>>> >>>>> The lock is per file. Adding more directories to the channel will >>>>> cause more files to be created. Of course you'll need additional disks >>>>> behind those directories to see any performance increase. >>>>> >>>>> >>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan < >>>>> hshreedharan@cloudera.com> wrote: >>>>> >>>>> Yes, IO is done inside locks to avoid multiple takes and puts getting >>>>> written out at the same time. Even though Java makes sure the writes are >>>>> serialized, Flume still needs to keep track of some counters etc, so the >>>>> lock is required. Note that the lock you are talking about is in the >>>>> LogFile class, which represents a single file - so even if the write is >>>>> inside that lock (which is also inside that class itself) that does not >>>>> cause any contention - because the lock is just preventing two IO ops to >>>>> happen at the same time. >>>>> >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote: >>>>> >>>>> It seems like some i/o is done inside the lock, which means that time >>>>> for taking a lock is proportional to the time for i/o and thus it becomes a >>>>> problem. I apologize in advance if I am wrong but the call stack and >>>>> behavior I'm seeing seems to suggest that. Specifically, it seems that we >>>>> do a write while inside take: >>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 >>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.NativeThread.current(Native Method) >>>>> at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27) >>>>> at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194) >>>>> - locked <0x00000005190ec998> (a java.lang.Object) >>>>> at >>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247) >>>>> at >>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212) >>>>> - locked <0x0000000519111590> (a >>>>> org.apache.flume.channel.file.LogFileV3$Writer) >>>>> at org.apache.flume.channel.file.Log.take(Log.java:550) >>>>> at >>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499) >>>>> at >>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) >>>>> at >>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) >>>>> at >>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330) >>>>> at >>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154) >>>>> at >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> >>>>> >>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan < >>>>> hshreedharan@cloudera.com> wrote: >>>>> >>>>> Since the channel is designed to make sure that events are not >>>>> duplicated to multiple sinks, and to protect against corruption due to >>>>> concurrency issues, we do not need the locking in the channel's flume event >>>>> queue. It is unlikely that locking is what is causing performance issues >>>>> because the channel is heavily I/O bound. If you take a series of thread >>>>> dumps, you will probably see that those threads are moving forward and the >>>>> ones reading/writing from/to disk are the ones which are slower. These >>>>> locks are unlikely to hit performance much. >>>>> >>>>> Thanks, >>>>> Hari >>>>> >>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote: >>>>> >>>>> Hi, >>>>> >>>>> Spent some more time debugging issues with FileChannel. The problem >>>>> seems to lock contention reading from FlumeEventQueue: >>>>> >>>>> I see a lot of threads like this: >>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10 >>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry >>>>> [0x00007f821afee000] >>>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>>> at >>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117) >>>>> - waiting to lock <0x0000000518ee4c90> (a >>>>> org.apache.flume.channel.file.FlumeEventQueue) >>>>> at >>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492) >>>>> at >>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113) >>>>> at >>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95) >>>>> at >>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330) >>>>> at >>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154) >>>>> at >>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> >>>>> I have two file channels and 8 Avro Sinks per file channel. I added >>>>> more sinks because they weren't draining fast enough. It seems like they >>>>> send the batch then wait for an ack before sending again, thus sends are >>>>> not pipelined and having more sinks seemed like a good way of getting some >>>>> parallelism. >>>>> >>>>> Here's the full stack trace: >>>>> 2013-08-13 15:30:32 >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02 mixed >>>>> mode): >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000] >>>>> java.lang.Thread.State: WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x000000056d932120> (a >>>>> java.util.concurrent.CountDownLatch$Sync) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) >>>>> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) >>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141) >>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:150) >>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129) >>>>> at >>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68) >>>>> at $Proxy7.appendBatch(Unknown Source) >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314) >>>>> at >>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310) >>>>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000 nid=0x41a1 >>>>> waiting on condition [0x0000000000000000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005194c3340> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005191a16d8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519174e08> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519174e98> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005195f9670> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519174ce8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005191a1918> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005191e7f60> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153 >>>>> runnable [0x00007f8210e4d000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000054a291760> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000054a2917a8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519174f58> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146 >>>>> runnable [0x00007f8210f4e000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000546e5dd28> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118 >>>>> runnable [0x00007f821104f000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000544f763c0> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111 >>>>> runnable [0x00007f8211150000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000054473d5a8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110 >>>>> runnable [0x00007f8211251000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2) >>>>> - locked <0x00000005437f4f70> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f >>>>> runnable [0x00007f8211352000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x00000005422a9030> (a sun.nio.ch.Util$2) >>>>> - locked <0x00000005422a9018> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e >>>>> runnable [0x00007f8211453000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000541063d68> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000541063d50> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d >>>>> runnable [0x00007f8211554000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000540dc1110> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a >>>>> runnable [0x00007f8211655000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000053fbbc3b0> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109 >>>>> runnable [0x00007f8211756000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000053f812878> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000053f812860> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd >>>>> runnable [0x00007f8211857000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000053e6e5570> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519175138> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000519174c28> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005191751c8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0 >>>>> runnable [0x00007f8211c5b000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000536579db0> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x00000005191750a8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1 >>>>> runnable [0x00007f8211e5d000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x00000005313943c0> (a sun.nio.ch.Util$2) >>>>> - locked <0x00000005313943a8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x000000051936f460> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x000000051936dfa8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de >>>>> runnable [0x00007f8212160000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000052dad4480> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000052dad4468> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd >>>>> runnable [0x00007f8212261000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000052c73c608> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000052c73c5f0> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc >>>>> runnable [0x00007f8212362000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2) >>>>> - locked <0x000000052b8da998> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db waiting >>>>> for monitor entry [0x00007f8212462000] >>>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>>> at >>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184) >>>>> - waiting to lock <0x0000000518ee4c90> (a >>>>> org.apache.flume.channel.file.FlumeEventQueue) >>>>> at >>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460) >>>>> at >>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93) >>>>> at >>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80) >>>>> at >>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189) >>>>> at >>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330) >>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88) >>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149) >>>>> at >>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188) >>>>> at >>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) >>>>> at >>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) >>>>> at >>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321) >>>>> at >>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303) >>>>> at >>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220) >>>>> at >>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) >>>>> at >>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) >>>>> at >>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238) >>>>> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x000000051936df18> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10 >>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000] >>>>> java.lang.Thread.State: TIMED_WAITING (parking) >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> - parking to wait for <0x0000000518fe7ae8> (a >>>>> java.util.concurrent.SynchronousQueue$TransferStack) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424) >>>>> at >>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) >>>>> at >>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8 >>>>> runnable [0x00007f8212766000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2) >>>>> - locked <0x00000005263d6ba8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7 >>>>> runnable [0x00007f8212867000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000525d1d0e0> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6 >>>>> runnable [0x00007f8212968000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000525a689d8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5 >>>>> runnable [0x00007f8212a69000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000524e1e7d8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4 >>>>> runnable [0x00007f8212b6a000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000524d437a8> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> "pool-142-thread-36" prio=10 tid=0x00007f85843ad800 nid=0x40d3 >>>>> runnable [0x00007f8212c6b000] >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked <0x0000000524cf4140> (a sun.nio.ch.Util$2) >>>>> - locked <0x0000000524cf4128> (a java.util.Collections$UnmodifiableSet) >>>>> - locked <0x0000000524ab32e8> (a sun.nio.ch.EPollSelectorImpl) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200) >>>>> at >>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) >>>>> at >>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>>>> >>>>> ... --089e0160b7be47233a04e3df7b34 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Gotcha. When you run tge test what is tye disk utilization p= ercentage? Iostat can be used for this.

On Aug 13, 2013 9:47 PM, "Pankaj Gupta"= ; <pankaj@brightroll.com>= ; wrote:
Those are the boxes we want to collect data from. They run= flume and send data through their avro sinks to the avro source on this bo= x. We are getting data at a pretty good rate and the problem is in fact tha= t the events don't drain from the FileChannel fast enough and the chann= el fill percentage keeps getting higher.


On Tue, Aug 1= 3, 2013 at 7:41 PM, Brock Noland <brock@cloudera.com> wrote= :

What is sending the events to the avro source?

On Aug 13, 2013 9:34 PM, "Pankaj Gupta"= ; <pankaj@bri= ghtroll.com> wrote:
Here's the config:
# define channels, one for= each disk




agent1.channels.ch1.type =3D FILE
agent1.channels.ch1.check= pointDir =3D /flume1/checkpoint
agent1.channels.ch1.dataDirs =3D /flume1/data
agent1.channel= s.ch1.maxFileSize =3D 375809638400
agent1.channels.ch1.capacity = =3D 75000000
agent1.channels.ch1.transactionCapacity =3D 4000

agent1.channels.ch2.type =3D FILE
agent1.channels.= ch2.checkpointDir =3D /flume2/checkpoint
agent1.channels.ch2.data= Dirs =3D /flume2/data
agent1.channels.ch2.maxFileSize =3D 3758096= 38400
agent1.channels.ch2.capacity =3D 75000000
agent1.channels.ch= 2.transactionCapacity =3D 4000



=
# Define an Avro source named avroSource1
# Each sink = can connect to only one channel.
# Connect it to channel ch1. Load balance it to 2 avroSinks
=

agent1.sources.avroSource1.channels =3D ch1
agent1.sources.avroSource1.type =3D avro
agent1.sources.= avroSource1.bind =3D 0.0.0.0
agent1.sources.avroSource1.port =3D <port>

<= div>


agent1.sinks.avroSink1-1-1= .type =3D avro
agent1.sinks.avroSink1-1-1.channel =3D ch1
agent1.sinks.avroSink1-1-1.hostname =3D <hostname>
agent1.s= inks.avroSink1-1-1.port =3D <port>
agent1.sinks.avroSink1-1= -1.connect-timeout =3D 300000
agent1.sinks.avroSink1-1-1.batch-si= ze =3D 4000




agent1.sin= ks.avroSink1-2-1.type =3D avro
agent1.sinks.avroSink1-2-1.channel= =3D ch1
agent1.sinks.avroSink1-2-1.hostname =3D <hostname>=
agent1.sinks.avroSink1-2-1.port =3D <port>
agent1.sink= s.avroSink1-2-1.connect-timeout =3D 300000
agent1.sinks.avroSink1= -2-1.batch-size =3D 4000



=

agent1.sinks.avroSink1-3-1.type =3D avro
agent1.si= nks.avroSink1-3-1.channel =3D ch1
agent1.sinks.avroSink1-3-1.host= name =3D <hostname>
agent1.sinks.avroSink1-3-1.port =3D <= ;port>
agent1.sinks.avroSink1-3-1.connect-timeout =3D 300000
agent1= .sinks.avroSink1-3-1.batch-size =3D 4000




agent1.sinks.avroSink1-4-1.type =3D avr= o
agent1.sinks.avroSink1-4-1.channel =3D ch1
agent1.sinks.avro= Sink1-4-1.hostname =3D <hostname>
agent1.sinks.avroSink1-4-= 1.port =3D <port>
agent1.sinks.avroSink1-4-1.connect-timeou= t =3D 300000
agent1.sinks.avroSink1-4-1.batch-size =3D 4000



#Add the sink groups; load-balance between = each group of sinks which round robin between different hops
agen= t1.sinkgroups.group1.sinks =3D avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 av= roSink1-4-1
agent1.sinkgroups.group1.processor.type =3D load_balance
age= nt1.sinkgroups.group1.processor.selector =3D ROUND_ROBIN
agent1.s= inkgroups.group1.processor.backoff =3D true


#End of set

# Define an Avro source named avr= oSource2
# Each sink can connect to only one channel.
#= Connect it to channel ch2. Load balance it to 2 avroSinks


agent1.sources.avroSource2.channels =3D ch2
=
agent1.sources.avroSource2.type =3D avro
agent1.sources.avro= Source2.bind =3D 0.0.0.0
agent1.sources.avroSource2.port =3D <= port>




agent1.sin= ks.avroSink2-1-1.type =3D avro
agent1.sinks.avroSink2-1-1.channel= =3D ch2
agent1.sinks.avroSink2-1-1.hostname =3D <hostname>=
agent1.sinks.avroSink2-1-1.port =3D <port>
agent1.sink= s.avroSink2-1-1.connect-timeout =3D 300000
agent1.sinks.avroSink2= -1-1.batch-size =3D 4000



=

agent1.sinks.avroSink2-2-1.type =3D avro
agent1.si= nks.avroSink2-2-1.channel =3D ch2
agent1.sinks.avroSink2-2-1.host= name =3D <hostname>
agent1.sinks.avroSink2-2-1.port =3D <= ;port>
agent1.sinks.avroSink2-2-1.connect-timeout =3D 300000
agent1= .sinks.avroSink2-2-1.batch-size =3D 4000




agent1.sinks.avroSink2-3-1.type =3D avr= o
agent1.sinks.avroSink2-3-1.channel =3D ch2
agent1.sinks.avro= Sink2-3-1.hostname =3D <hostname>
agent1.sinks.avroSink2-3-= 1.port =3D <port>
agent1.sinks.avroSink2-3-1.connect-timeou= t =3D 300000
agent1.sinks.avroSink2-3-1.batch-size =3D 4000




agent1.sinks.avroSink2-4-1.t= ype =3D avro
agent1.sinks.avroSink2-4-1.channel =3D ch2
agent1.sinks.avroSink2-4-1.hostname =3D <hostname>
agent1.sinks.avroSink2-4-1.port =3D <port>
agent1.sink= s.avroSink2-4-1.connect-timeout =3D 300000
agent1.sinks.avroSink2= -4-1.batch-size =3D 4000



=
#Add the sink groups; load-balance between each group of sinks which round = robin between different hops
agent1.sinkgroups.group2.sinks =3D a= vroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
agent1.sin= kgroups.group2.processor.type =3D load_balance
agent1.sinkgroups.group2.processor.selector =3D ROUND_ROBIN
= agent1.sinkgroups.group2.processor.backoff =3D true


#End of set

# Define an Avro sour= ce named avroSource3
# Each sink can connect to only one channel.
# Connect it to= channel ch1. Load balance it to 2 avroSinks


<= /div>
agent1.sources.avroSource3.channels =3D ch1
agent1.sour= ces.avroSource3.type =3D avro
agent1.sources.avroSource3.bind =3D 0.0.0.0
agent1.sources.a= vroSource3.port =3D <port>



agent1.sinks.avroSink3-1-1.type =3D avro
<= div> agent1.sinks.avroSink3-1-1.channel =3D ch1
agent1.sinks.avroSink3= -1-1.hostname =3D <hostname>
agent1.sinks.avroSink3-1-1.por= t =3D <port>
agent1.sinks.avroSink3-1-1.connect-timeout =3D= 300000
agent1.sinks.avroSink3-1-1.batch-size =3D 4000




agent1.sinks.avroSink3-2-1.t= ype =3D avro
agent1.sinks.avroSink3-2-1.channel =3D ch1
agent1.sinks.avroSink3-2-1.hostname =3D <hostname>
agent1.sinks.avroSink3-2-1.port =3D <port>
agent1.sink= s.avroSink3-2-1.connect-timeout =3D 300000
agent1.sinks.avroSink3= -2-1.batch-size =3D 4000



=

agent1.sinks.avroSink3-3-1.type =3D avro
agent1.si= nks.avroSink3-3-1.channel =3D ch1
agent1.sinks.avroSink3-3-1.host= name =3D <hostname>
agent1.sinks.avroSink3-3-1.port =3D <= ;port>
agent1.sinks.avroSink3-3-1.connect-timeout =3D 300000
agent1= .sinks.avroSink3-3-1.batch-size =3D 4000




agent1.sinks.avroSink3-4-1.type =3D avr= o
agent1.sinks.avroSink3-4-1.channel =3D ch1
agent1.sinks.avro= Sink3-4-1.hostname =3D <hostname>
agent1.sinks.avroSink3-4-= 1.port =3D <port>
agent1.sinks.avroSink3-4-1.connect-timeou= t =3D 300000
agent1.sinks.avroSink3-4-1.batch-size =3D 4000



#Add the sink groups; load-balance between = each group of sinks which round robin between different hops
agen= t1.sinkgroups.group3.sinks =3D avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 av= roSink3-4-1
agent1.sinkgroups.group3.processor.type =3D load_balance
age= nt1.sinkgroups.group3.processor.selector =3D ROUND_ROBIN
agent1.s= inkgroups.group3.processor.backoff =3D true


#End of set

# Define an Avro source named avr= oSource4
# Each sink can connect to only one channel.
#= Connect it to channel ch2. Load balance it to 2 avroSinks


agent1.sources.avroSource4.channels =3D ch2
=
agent1.sources.avroSource4.type =3D avro
agent1.sources.avro= Source4.bind =3D 0.0.0.0
agent1.sources.avroSource4.port =3D <= port>




agent1.sin= ks.avroSink4-1-1.type =3D avro
agent1.sinks.avroSink4-1-1.channel= =3D ch2
agent1.sinks.avroSink4-1-1.hostname =3D <hostname>=
agent1.sinks.avroSink4-1-1.port =3D <port>
agent1.sink= s.avroSink4-1-1.connect-timeout =3D 300000
agent1.sinks.avroSink4= -1-1.batch-size =3D 4000



=

agent1.sinks.avroSink4-2-1.type =3D avro
agent1.si= nks.avroSink4-2-1.channel =3D ch2
agent1.sinks.avroSink4-2-1.host= name =3D <hostname>
agent1.sinks.avroSink4-2-1.port =3D <= ;port>
agent1.sinks.avroSink4-2-1.connect-timeout =3D 300000
agent1= .sinks.avroSink4-2-1.batch-size =3D 4000




agent1.sinks.avroSink4-3-1.type =3D avr= o
agent1.sinks.avroSink4-3-1.channel =3D ch2
agent1.sinks.avro= Sink4-3-1.hostname =3D <hostname>
agent1.sinks.avroSink4-3-= 1.port =3D <port>
agent1.sinks.avroSink4-3-1.connect-timeou= t =3D 300000
agent1.sinks.avroSink4-3-1.batch-size =3D 4000




agent1.sinks.avroSink4-4-1.t= ype =3D avro
agent1.sinks.avroSink4-4-1.channel =3D ch2
agent1.sinks.avroSink4-4-1.hostname =3D <hostname>
agent1.sinks.avroSink4-4-1.port =3D <port>
agent1.sink= s.avroSink4-4-1.connect-timeout =3D 300000
agent1.sinks.avroSink4= -4-1.batch-size =3D 4000



=
#Add the sink groups; load-balance between each group of sinks which round = robin between different hops
agent1.sinkgroups.group4.sinks =3D a= vroSink4-1-1 avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
agent1.sin= kgroups.group4.processor.type =3D load_balance
agent1.sinkgroups.group4.processor.selector =3D ROUND_ROBIN
= agent1.sinkgroups.group4.processor.backoff =3D true


#End of set



# Finally, now that we've defined all of our components, tell
# agent1 which ones we want to activate.
agent1.channels = =3D ch1 ch2
agent1.sources =3D avroSource1 avroSource2 avroSource= 3 avroSource4
agent1.sinks =3D avroSink1-1-1 avroSink1-2-1 avroSink1-3-1 avroSink1-4= -1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1 avroSink3-1-1 av= roSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1 avroSink4-2-1 avroSin= k4-3-1 avroSink4-4-1
agent1.sinkgroups =3D group1 group2 group3 group4

=
There are two disks on the box /flume1 and /flume2
32 = cores of CPU, 32 GB RAM, 1GbPS ethernet.


On Tue, Aug 13, 2013 at 7:22 PM, Brock N= oland <brock@cloudera.com> wrote:

Can you share your conf file?

On Aug 13, 2013 9:19 PM, "Hari Sh= reedharan" <hshreedharan@cloudera.com> wrote:
Even though the writes are done per batch, they don'= ;t go to disk rightaway - commits are the only ones which actually cause an= fsync - which is when writes actually go to disk.=A0


Thanks,
Hari

=20

On Tuesday, August 13, 2013 at 7= :06 PM, Pankaj Gupta wrote:

Looking at the code it= seems like the lock and the i/o is done per event and not for a batch. Is = that correct? If that is the case then it seems like there is a lot of over= head per event. The throughput I'm seeing is 1 - 1.5 MBps per disk whic= h is way below the sequential read/write capacity of the disk which is easi= ly over 50MBps. Adding more sinks doesn't help, they just block waiting= for the queue to become free. CPU usage is 20%, there is enough RAM for pa= ge cache so that no read is going to disk. The queue seems to be the bottle= neck. What is the throughput I should expect per disk?=A0


On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <brock@= cloudera.com> wrote:
The lock is per file. Adding more directories to the chann= el will cause more files to be created. Of course you'll need additiona= l disks behind those directories to see any performance increase.=A0


On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <hs= hreedharan@cloudera.com> wrote:
=
Yes, IO is done inside locks to avoid multiple takes an= d puts getting written out at the same time. Even though Java makes sure th= e writes are serialized, Flume still needs to keep track of some counters e= tc, so the lock is required. Note that the lock you are talking about is = =A0in the LogFile class, which represents a single file - so even if the wr= ite is inside that lock (which is also inside that class itself) that =A0do= es not cause any contention - because the lock is just preventing two IO op= s to happen at the same time.


Thanks,
Hari

=20

On Tuesday, August 13, 2013 at 5= :01 PM, Pankaj Gupta wrote:

It seems like some i/o= is done inside the lock, which means that time for taking a lock is propor= tional to the time for i/o and thus it becomes a problem. I apologize in ad= vance if I am wrong but the call stack and behavior I'm seeing seems to= suggest that. Specifically, it seems that we do a write while inside take:=
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio= =3D10 tid=3D0x00007f857338c800 nid=3D0x404a runnable [0x00007f821b2f1000]
=A0 =A0java.lang.Thread.State: RUNNABLE
=A0 =A0 =A0 =A0 = at sun.nio.ch.NativeThread.current(Native Method)
=A0 =A0 =A0 =A0 at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java= :27)
=A0 =A0 =A0 =A0 at sun.nio.ch.FileChannelImpl.write(FileChan= nelImpl.java:194)
=A0 =A0 =A0 =A0 - locked <0x00000005190ec998= > (a java.lang.Object)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.file.LogFile$Writer.write(= LogFile.java:247)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.fil= e.LogFile$Writer.take(LogFile.java:212)
=A0 =A0 =A0 =A0 - locked = <0x0000000519111590> (a org.apache.flume.channel.file.LogFileV3$Write= r)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.file.Log.take(Log.java:550= )
=A0 =A0 =A0 =A0 at org.apache.flume.channel.file.FileChannel$Fi= leBackedTransaction.doTake(FileChannel.java:499)
=A0 =A0 =A0 =A0 = at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransaction= Semantics.java:113)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.BasicChannelSemantics.take= (BasicChannelSemantics.java:95)
=A0 =A0 =A0 =A0 at org.apache.flu= me.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
=A0 =A0= =A0 =A0 at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBa= lancingSinkProcessor.java:154)
=A0 =A0 =A0 =A0 at org.apache.flume.SinkRunner$PollingRunner.run(SinkR= unner.java:147)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(Thread.ja= va:662)



On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <hshreedharan@cl= oudera.com> wrote:
Since the channel is designed to make sure that events= are not duplicated to multiple sinks, and to protect against corruption du= e to concurrency issues, we do not need the locking in the channel's fl= ume event queue. It is unlikely that locking is what is causing performance= issues because the channel is heavily I/O bound. If you take a series of t= hread dumps, you will probably see that those threads are moving forward an= d the ones reading/writing from/to disk are the ones which are slower. Thes= e locks are unlikely to hit performance much.

Thanks,
Hari

=20

On Tuesday, August 13, 2013 at 4= :13 PM, Pankaj Gupta wrote:

Hi,

Spent some more time debugging issues with FileChannel. The problem seems = to lock contention reading from FlumeEventQueue:

I see a lot of threads like this:
"SinkRunner-PollingRunner-LoadBalancingSinkProcessor" p= rio=3D10 tid=3D0x00007f857b378800 nid=3D0x404d waiting for monitor entry [0= x00007f821afee000]
=A0 =A0java.lang.Thread.State: BLOCKED (on obj= ect monitor)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.file.FlumeEventQueue.remov= eHead(FlumeEventQueue.java:117)
=A0 =A0 =A0 =A0 - waiting to lock= <0x0000000518ee4c90> (a org.apache.flume.channel.file.FlumeEventQueu= e)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.file.FileChannel$FileBackedTran= saction.doTake(FileChannel.java:492)
=A0 =A0 =A0 =A0 at org.apach= e.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.ja= va:113)
=A0 =A0 =A0 =A0 at org.apache.flume.channel.BasicChannelSemantics.take= (BasicChannelSemantics.java:95)
=A0 =A0 =A0 =A0 at org.apache.flu= me.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
=A0 =A0= =A0 =A0 at org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBa= lancingSinkProcessor.java:154)
=A0 =A0 =A0 =A0 at org.apache.flume.SinkRunner$PollingRunner.run(SinkR= unner.java:147)
=A0 =A0 =A0 =A0 at java.lang.Thread.run(Thread.ja= va:662)


I have two file channels an= d 8 Avro Sinks per file channel. I added more sinks because they weren'= t draining fast enough. It seems like they send the batch then wait for an = ack before sending again, thus sends are not pipelined and having more sink= s seemed like a good way of getting some parallelism.

Here's the full stack trace:
2013-08= -13 15:30:32
Full thread dump Java HotSpot(TM) 64-Bit Server VM (= 20.13-b02 mixed mode):

"Flume Avro RPC Client= Call Invoker 1" daemon prio=3D10 tid=3D0x00007f834c02c000 nid=3D0x41a= 2 waiting on condition [0x00007f8210341000]
=A0 =A0java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for = =A0<0x000000056d932120> (a java.util.concurrent.CountDownLatch$Sync)<= /div>
at java.util.concurrent.l= ocks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchro= nizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.l= ocks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueued= Synchronizer.java:969)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedIn= terruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.C= ountDownLatch.await(CountDownLatch.java:207)
at org.apache.avro.ipc.CallFuture.await(CallFuture= .java:141)
at org.apache.avro.ipc.Re= questor.request(Requestor.java:150)
at org.apache.avro.ipc.Requestor.request(Requestor.java:129= )
at org.apache.avro.ipc.sp= ecific.SpecificRequestor.invoke(SpecificRequestor.java:68)
at $Proxy7.appendBatch(Unknown Sourc= e)
at org.apache.flume.api.N= ettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
at org.apache.flume.api.NettyAvroRpcClien= t$2.call(NettyAvroRpcClient.java:310)
at java.util.concurrent.F= utureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.= java:138)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"Attach Listener" daemon= prio=3D10 tid=3D0x00007f85fc016000 nid=3D0x41a1 waiting on condition [0x00= 00000000000000]
=A0 =A0java.lang.Thread.State: RUNNABLE

"= ;Flume Avro RPC Client Call Invoker 1" daemon prio=3D10 tid=3D0x00007f= 834c02a800 nid=3D0x4173 waiting on condition [0x00007f8210544000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(N= ative Method)
- park= ing to wait for =A0<0x00000005194c3340> (a java.util.concurrent.Synch= ronousQueue$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f836000f800 nid=3D0x4172 waiti= ng on condition [0x00007f8210645000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005191a16d8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f82ec01f000 nid=3D0x4164 waiti= ng on condition [0x00007f8210746000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519174e08> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f8374023000 nid=3D0x4163 waiti= ng on condition [0x00007f8210847000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519174e98> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f82ec030000 nid=3D0x415d waiti= ng on condition [0x00007f8210948000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005195f9670> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f8374021800 nid=3D0x415b waiti= ng on condition [0x00007f8210a49000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519174ce8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f836000f000 nid=3D0x415a waiti= ng on condition [0x00007f8210b4a000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005191a1918> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f834c024000 nid=3D0x4154 waiti= ng on condition [0x00007f8210c4b000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005191e7f60> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-41" pri= o=3D10 tid=3D0x00007f85d8466800 nid=3D0x4153 runnable [0x00007f8210e4d000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000054a= 291760> (a sun.nio.ch.Util$2)
- locked <0x000000054a2917a8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000054a= 2c5348> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f82ec029800 nid=3D0x4147 waiti= ng on condition [0x00007f8210d4c000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519174f58> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-41" pri= o=3D10 tid=3D0x00007f8584469800 nid=3D0x4146 runnable [0x00007f8210f4e000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000546= e5dd40> (a sun.nio.ch.Util$2)
- locked <0x0000000546e5dd28> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000546= ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-41" pri= o=3D10 tid=3D0x00007f8588411000 nid=3D0x4118 runnable [0x00007f821104f000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000544= f763d8> (a sun.nio.ch.Util$2)
- locked <0x0000000544f763c0> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000544= ee55f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-139-thread-41" pri= o=3D10 tid=3D0x00007f85803ff000 nid=3D0x4111 runnable [0x00007f8211150000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000544= 73d5c0> (a sun.nio.ch.Util$2)
- locked <0x000000054473d5a8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000544= 74d088> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-40" pri= o=3D10 tid=3D0x00007f85843c7800 nid=3D0x4110 runnable [0x00007f8211251000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000543= 7f4f88> (a sun.nio.ch.Util$2)
- locked <0x00000005437f4f70> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000543= 803478> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-40" pri= o=3D10 tid=3D0x00007f858840f800 nid=3D0x410f runnable [0x00007f8211352000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000542= 2a9030> (a sun.nio.ch.Util$2)
- locked <0x00000005422a9018> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000542= 2ca310> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-39" pri= o=3D10 tid=3D0x00007f85843c6000 nid=3D0x410e runnable [0x00007f8211453000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000541= 063d68> (a sun.nio.ch.Util$2)
- locked <0x0000000541063d50> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000541= 0c8240> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-40" pri= o=3D10 tid=3D0x00007f85d83c6000 nid=3D0x410d runnable [0x00007f8211554000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000540= dc1128> (a sun.nio.ch.Util$2)
- locked <0x0000000540dc1110> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000540= c94928> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-39" pri= o=3D10 tid=3D0x00007f85d83c4800 nid=3D0x410a runnable [0x00007f8211655000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000053f= bbc3c8> (a sun.nio.ch.Util$2)
- locked <0x000000053fbbc3b0> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000053f= 897d00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-38" pri= o=3D10 tid=3D0x00007f85843c4800 nid=3D0x4109 runnable [0x00007f8211756000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000053f= 812878> (a sun.nio.ch.Util$2)
- locked <0x000000053f812860> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000053f= 897cb8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-38" pri= o=3D10 tid=3D0x00007f85d83c3000 nid=3D0x40fd runnable [0x00007f8211857000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000053e= 6e5588> (a sun.nio.ch.Util$2)
- locked <0x000000053e6e5570> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000053e= 714e70> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f8374020800 nid=3D0x40f3 waiti= ng on condition [0x00007f8211958000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519175138> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f834c022000 nid=3D0x40f2 waiti= ng on condition [0x00007f8211a59000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000519174c28> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f82ec032000 nid=3D0x40f1 waiti= ng on condition [0x00007f8211b5a000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005191751c8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-39" pri= o=3D10 tid=3D0x00007f85883ad800 nid=3D0x40f0 runnable [0x00007f8211c5b000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000536= 579dc8> (a sun.nio.ch.Util$2)
- locked <0x0000000536579db0> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000536= 585830> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f836000d800 nid=3D0x40ef waiti= ng on condition [0x00007f8211d5c000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x00000005191750a8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-38" pri= o=3D10 tid=3D0x00007f85883ac000 nid=3D0x40e1 runnable [0x00007f8211e5d000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000531= 3943c0> (a sun.nio.ch.Util$2)
- locked <0x00000005313943a8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000531= 4e0fe0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f837401f000 nid=3D0x40e0 waiti= ng on condition [0x00007f8211f5e000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x000000051936f460> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f82ec028000 nid=3D0x40df waiti= ng on condition [0x00007f821205f000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x000000051936dfa8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-37" pri= o=3D10 tid=3D0x00007f85883aa800 nid=3D0x40de runnable [0x00007f8212160000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000052d= ad4480> (a sun.nio.ch.Util$2)
- locked <0x000000052dad4468> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000052d= a31528> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-139-thread-40" pri= o=3D10 tid=3D0x00007f85803fd800 nid=3D0x40dd runnable [0x00007f8212261000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000052c= 73c608> (a sun.nio.ch.Util$2)
- locked <0x000000052c73c5f0> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000052c= 652608> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-36" pri= o=3D10 tid=3D0x00007f85883a9000 nid=3D0x40dc runnable [0x00007f8212362000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x000000052b= 8da9b0> (a sun.nio.ch.Util$2)
- locked <0x000000052b8da998> (a java.util.Collections$U= nmodifiableSet)
- locked <0x000000052b= 8fc590> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-139-thread-39" pri= o=3D10 tid=3D0x00007f85803fc000 nid=3D0x40db waiting for monitor entry [0x0= 0007f8212462000]
=A0 =A0java.lang.Thread.State: BLOCKED (on object monitor)
<= span style=3D"white-space:pre-wrap"> at org.apache.flume.channel.fil= e.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
- waiting to lock <0x0000000518= ee4c90> (a org.apache.flume.channel.file.FlumeEventQueue)
at org.apache.flume.chann= el.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
=
at org.apache.flume.chann= el.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at org.apache.flume.chann= el.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at org.apache.flume.channel.Channel= Processor.processEventBatch(ChannelProcessor.java:189)
at org.apache.flume.sourc= e.AvroSource.appendBatch(AvroSource.java:330)
at sun.reflect.GeneratedMethodAccessor6.invoke(Un= known Source)
at sun.reflect.Delegating= MethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
<= span style=3D"white-space:pre-wrap"> at java.lang.reflect.Method.inv= oke(Method.java:597)
at org.apache.avro.ipc.sp= ecific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.re= spond(Responder.java:149)
at org.apache.avro.ipc.Ne= ttyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at org.jboss.netty.chann= el.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler= .java:75)
at org.apache.avro.ipc.Ne= ttyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
=
at org.jboss.netty.channe= l.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channe= l.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultC= hannelPipeline.java:792)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:= 296)
at org.jboss.netty.handle= r.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:3= 21)
at org.jboss.net= ty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at org.jboss.netty.handle= r.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at org.jboss.netty.channel.Si= mpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java= :75)
at org.jboss.netty.channe= l.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.chann= el.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channe= l.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessa= geReceived(Channels.java:255)
at org.jboss.netty.channe= l.socket.nio.NioWorker.read(NioWorker.java:94)
at org.jboss.netty.channel.socket.nio.AbstractNi= oWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.jboss.netty.channe= l.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socke= t.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.i= nternal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPo= olExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.T= hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:6= 62)

"Flume Avro RPC Client Call Invoker 1" daemon= prio=3D10 tid=3D0x00007f836000b000 nid=3D0x40da waiting on condition [0x00= 007f8212564000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (par= king)
at sun.misc.Unsafe.park(N= ative Method)
- park= ing to wait for =A0<0x000000051936df18> (a java.util.concurrent.Synch= ronousQueue$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"Flume Avro RPC Client Call I= nvoker 1" daemon prio=3D10 tid=3D0x00007f834c020000 nid=3D0x40d9 waiti= ng on condition [0x00007f8212665000]
=A0 =A0java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Meth= od)
- parking to wai= t for =A0<0x0000000518fe7ae8> (a java.util.concurrent.SynchronousQueu= e$TransferStack)
at java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.SynchronousQueue$Trans= ferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.S= ynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.Synch= ronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.T= hreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecut= or$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-37" pri= o=3D10 tid=3D0x00007f85843c3000 nid=3D0x40d8 runnable [0x00007f8212766000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000526= 3d6bc0> (a sun.nio.ch.Util$2)
- locked <0x00000005263d6ba8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000526= 4b1a48> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-37" pri= o=3D10 tid=3D0x00007f85d83c1800 nid=3D0x40d7 runnable [0x00007f8212867000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000525= d1d0f8> (a sun.nio.ch.Util$2)
- locked <0x0000000525d1d0e0> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000525= d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-144-thread-36" pri= o=3D10 tid=3D0x00007f85d83c0000 nid=3D0x40d6 runnable [0x00007f8212968000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000525= a689f0> (a sun.nio.ch.Util$2)
- locked <0x0000000525a689d8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000525= a24570> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-140-thread-35" pri= o=3D10 tid=3D0x00007f85883a8000 nid=3D0x40d5 runnable [0x00007f8212a69000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000524= e1e7f0> (a sun.nio.ch.Util$2)
- locked <0x0000000524e1e7d8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000524= cfd670> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-139-thread-38" pri= o=3D10 tid=3D0x00007f85803fa800 nid=3D0x40d4 runnable [0x00007f8212b6a000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000524= d437c0> (a sun.nio.ch.Util$2)
- locked <0x0000000524d437a8> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000524= ab3330> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.T= hreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(T= hread.java:662)

"pool-142-thread-36" pri= o=3D10 tid=3D0x00007f85843ad800 nid=3D0x40d3 runnable [0x00007f8212c6b000]<= /div>
=A0 =A0java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native M= ethod)
at sun.nio.ch= .EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelect= orImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorI= mpl.java:69)
- locked <0x0000000524= cf4140> (a sun.nio.ch.Util$2)
- locked <0x0000000524cf4128> (a java.util.Collections$U= nmodifiableSet)
- locked <0x0000000524= ab32e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java= :80)
at org.jboss.netty.channe= l.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
at org.jboss.netty.channel.socket.nio.Ab= stractNioWorker.run(AbstractNioWorker.java:200)
at org.jboss.netty.channe= l.socket.nio.NioWorker.run(NioWorker.java:38)
at org.jboss.netty.util.internal.DeadLockProofWor= ker$1.run(DeadLockProofWorker.java:42)
= ... --089e0160b7be47233a04e3df7b34--