flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pankaj Gupta <pan...@brightroll.com>
Subject Re: Lock contention in FileChannel
Date Wed, 14 Aug 2013 18:57:57 GMT
I tried increasing the dataDirs to 2 and 4 per disk but doesn't seem to
help much. I then replaced the avro sink with null sinks and events are
still filling up in the channel. I tried with both 2 and 4 dataDirs per
disk and null sink, still don't get throughput higher than 1.5 MBps.


On Tue, Aug 13, 2013 at 8:30 PM, Brock Noland <brock@cloudera.com> wrote:

> Increasing the number of file channels will result in more checkpoints.
> Therefore there will be more io than simply increasing the number of
> dataDirs.  However, this might be a case where it'd be nice to relax the
> file channel data consistency constraints a little to get increased
> throughput. That feature does not exist at present.
>
>
> On Tue, Aug 13, 2013 at 10:16 PM, Pankaj Gupta <pankaj@brightroll.com>wrote:
>
>> I did try increasing number of FileChannels. At 2 FileChannels per disk
>> performance seemed to be 25% better. At 4 FileChannels per disk performance
>> dropped to even below 1 FileChannel per disk. I will try increasing the
>> dataDirs tomorrow.
>>
>>
>> On Tue, Aug 13, 2013 at 8:06 PM, Brock Noland <brock@cloudera.com> wrote:
>>
>>> dataDirs is a comma separated list. Try 3-4 directories and then the
>>> same test.
>>> On Aug 13, 2013 9:58 PM, "Pankaj Gupta" <pankaj@brightroll.com> wrote:
>>>
>>>> Both disks were at around 15-25%.
>>>>
>>>>
>>>> On Tue, Aug 13, 2013 at 7:54 PM, Brock Noland <brock@cloudera.com>wrote:
>>>>
>>>>> Gotcha. When you run tge test what is tye disk utilization percentage?
>>>>> Iostat can be used for this.
>>>>> On Aug 13, 2013 9:47 PM, "Pankaj Gupta" <pankaj@brightroll.com> wrote:
>>>>>
>>>>>> Those are the boxes we want to collect data from. They run flume and
>>>>>> send data through their avro sinks to the avro source on this box. We are
>>>>>> getting data at a pretty good rate and the problem is in fact that the
>>>>>> events don't drain from the FileChannel fast enough and the channel fill
>>>>>> percentage keeps getting higher.
>>>>>>
>>>>>>
>>>>>> On Tue, Aug 13, 2013 at 7:41 PM, Brock Noland <brock@cloudera.com>wrote:
>>>>>>
>>>>>>> What is sending the events to the avro source?
>>>>>>> On Aug 13, 2013 9:34 PM, "Pankaj Gupta" <pankaj@brightroll.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Here's the config:
>>>>>>>> # define channels, one for each disk
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.channels.ch1.type = FILE
>>>>>>>> agent1.channels.ch1.checkpointDir = /flume1/checkpoint
>>>>>>>> agent1.channels.ch1.dataDirs = /flume1/data
>>>>>>>> agent1.channels.ch1.maxFileSize = 375809638400
>>>>>>>> agent1.channels.ch1.capacity = 75000000
>>>>>>>> agent1.channels.ch1.transactionCapacity = 4000
>>>>>>>>
>>>>>>>> agent1.channels.ch2.type = FILE
>>>>>>>> agent1.channels.ch2.checkpointDir = /flume2/checkpoint
>>>>>>>> agent1.channels.ch2.dataDirs = /flume2/data
>>>>>>>> agent1.channels.ch2.maxFileSize = 375809638400
>>>>>>>> agent1.channels.ch2.capacity = 75000000
>>>>>>>> agent1.channels.ch2.transactionCapacity = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource1
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource1.channels = ch1
>>>>>>>> agent1.sources.avroSource1.type = avro
>>>>>>>> agent1.sources.avroSource1.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource1.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-1-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-2-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-3-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink1-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink1-4-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink1-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink1-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink1-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink1-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group1.sinks = avroSink1-1-1 avroSink1-2-1
>>>>>>>> avroSink1-3-1 avroSink1-4-1
>>>>>>>> agent1.sinkgroups.group1.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group1.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group1.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource2
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource2.channels = ch2
>>>>>>>> agent1.sources.avroSource2.type = avro
>>>>>>>> agent1.sources.avroSource2.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource2.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-1-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-2-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-3-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink2-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink2-4-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink2-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink2-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink2-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink2-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group2.sinks = avroSink2-1-1 avroSink2-2-1
>>>>>>>> avroSink2-3-1 avroSink2-4-1
>>>>>>>> agent1.sinkgroups.group2.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group2.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group2.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource3
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch1. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource3.channels = ch1
>>>>>>>> agent1.sources.avroSource3.type = avro
>>>>>>>> agent1.sources.avroSource3.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource3.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-1-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-2-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-3-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink3-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink3-4-1.channel = ch1
>>>>>>>> agent1.sinks.avroSink3-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink3-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink3-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink3-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group3.sinks = avroSink3-1-1 avroSink3-2-1
>>>>>>>> avroSink3-3-1 avroSink3-4-1
>>>>>>>> agent1.sinkgroups.group3.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group3.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group3.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>> # Define an Avro source named avroSource4
>>>>>>>> # Each sink can connect to only one channel.
>>>>>>>> # Connect it to channel ch2. Load balance it to 2 avroSinks
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sources.avroSource4.channels = ch2
>>>>>>>> agent1.sources.avroSource4.type = avro
>>>>>>>> agent1.sources.avroSource4.bind = 0.0.0.0
>>>>>>>> agent1.sources.avroSource4.port = <port>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-1-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-1-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-1-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-1-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-1-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-1-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-2-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-2-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-2-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-2-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-2-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-2-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-3-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-3-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-3-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-3-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-3-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-3-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> agent1.sinks.avroSink4-4-1.type = avro
>>>>>>>> agent1.sinks.avroSink4-4-1.channel = ch2
>>>>>>>> agent1.sinks.avroSink4-4-1.hostname = <hostname>
>>>>>>>> agent1.sinks.avroSink4-4-1.port = <port>
>>>>>>>> agent1.sinks.avroSink4-4-1.connect-timeout = 300000
>>>>>>>> agent1.sinks.avroSink4-4-1.batch-size = 4000
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> #Add the sink groups; load-balance between each group of sinks
>>>>>>>> which round robin between different hops
>>>>>>>> agent1.sinkgroups.group4.sinks = avroSink4-1-1 avroSink4-2-1
>>>>>>>> avroSink4-3-1 avroSink4-4-1
>>>>>>>> agent1.sinkgroups.group4.processor.type = load_balance
>>>>>>>> agent1.sinkgroups.group4.processor.selector = ROUND_ROBIN
>>>>>>>> agent1.sinkgroups.group4.processor.backoff = true
>>>>>>>>
>>>>>>>>
>>>>>>>> #End of set
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> # Finally, now that we've defined all of our components, tell
>>>>>>>> # agent1 which ones we want to activate.
>>>>>>>> agent1.channels = ch1 ch2
>>>>>>>> agent1.sources = avroSource1 avroSource2 avroSource3 avroSource4
>>>>>>>> agent1.sinks = avroSink1-1-1 avroSink1-2-1 avroSink1-3-1
>>>>>>>> avroSink1-4-1 avroSink2-1-1 avroSink2-2-1 avroSink2-3-1 avroSink2-4-1
>>>>>>>> avroSink3-1-1 avroSink3-2-1 avroSink3-3-1 avroSink3-4-1 avroSink4-1-1
>>>>>>>> avroSink4-2-1 avroSink4-3-1 avroSink4-4-1
>>>>>>>> agent1.sinkgroups = group1 group2 group3 group4
>>>>>>>>
>>>>>>>> There are two disks on the box /flume1 and /flume2
>>>>>>>> 32 cores of CPU, 32 GB RAM, 1GbPS ethernet.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Aug 13, 2013 at 7:22 PM, Brock Noland <brock@cloudera.com>wrote:
>>>>>>>>
>>>>>>>>> Can you share your conf file?
>>>>>>>>> On Aug 13, 2013 9:19 PM, "Hari Shreedharan" <
>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>
>>>>>>>>>>  Even though the writes are done per batch, they don't go to disk
>>>>>>>>>> rightaway - commits are the only ones which actually cause an fsync - which
>>>>>>>>>> is when writes actually go to disk.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 7:06 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> Looking at the code it seems like the lock and the i/o is done
>>>>>>>>>> per event and not for a batch. Is that correct? If that is the case then it
>>>>>>>>>> seems like there is a lot of overhead per event. The throughput I'm seeing
>>>>>>>>>> is 1 - 1.5 MBps per disk which is way below the sequential read/write
>>>>>>>>>> capacity of the disk which is easily over 50MBps. Adding more sinks doesn't
>>>>>>>>>> help, they just block waiting for the queue to become free. CPU usage is
>>>>>>>>>> 20%, there is enough RAM for page cache so that no read is going to disk.
>>>>>>>>>> The queue seems to be the bottleneck. What is the throughput I should
>>>>>>>>>> expect per disk?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 5:51 PM, Brock Noland <brock@cloudera.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>> The lock is per file. Adding more directories to the channel will
>>>>>>>>>> cause more files to be created. Of course you'll need additional disks
>>>>>>>>>> behind those directories to see any performance increase.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 7:14 PM, Hari Shreedharan <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>>  Yes, IO is done inside locks to avoid multiple takes and puts
>>>>>>>>>> getting written out at the same time. Even though Java makes sure the
>>>>>>>>>> writes are serialized, Flume still needs to keep track of some counters
>>>>>>>>>> etc, so the lock is required. Note that the lock you are talking about is
>>>>>>>>>>  in the LogFile class, which represents a single file - so even if the
>>>>>>>>>> write is inside that lock (which is also inside that class itself) that
>>>>>>>>>>  does not cause any contention - because the lock is just preventing two IO
>>>>>>>>>> ops to happen at the same time.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 5:01 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> It seems like some i/o is done inside the lock, which means that
>>>>>>>>>> time for taking a lock is proportional to the time for i/o and thus it
>>>>>>>>>> becomes a problem. I apologize in advance if I am wrong but the call stack
>>>>>>>>>> and behavior I'm seeing seems to suggest that. Specifically, it seems that
>>>>>>>>>> we do a write while inside take:
>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>> tid=0x00007f857338c800 nid=0x404a runnable [0x00007f821b2f1000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>         at sun.nio.ch.NativeThread.current(Native Method)
>>>>>>>>>>         at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)
>>>>>>>>>>         at
>>>>>>>>>> sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:194)
>>>>>>>>>>         - locked <0x00000005190ec998> (a java.lang.Object)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:247)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.LogFile$Writer.take(LogFile.java:212)
>>>>>>>>>>         - locked <0x0000000519111590> (a
>>>>>>>>>> org.apache.flume.channel.file.LogFileV3$Writer)
>>>>>>>>>>         at org.apache.flume.channel.file.Log.take(Log.java:550)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:499)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 13, 2013 at 4:39 PM, Hari Shreedharan <
>>>>>>>>>> hshreedharan@cloudera.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Since the channel is designed to make sure that events are not
>>>>>>>>>> duplicated to multiple sinks, and to protect against corruption due to
>>>>>>>>>> concurrency issues, we do not need the locking in the channel's flume event
>>>>>>>>>> queue. It is unlikely that locking is what is causing performance issues
>>>>>>>>>> because the channel is heavily I/O bound. If you take a series of thread
>>>>>>>>>> dumps, you will probably see that those threads are moving forward and the
>>>>>>>>>> ones reading/writing from/to disk are the ones which are slower. These
>>>>>>>>>> locks are unlikely to hit performance much.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Hari
>>>>>>>>>>
>>>>>>>>>> On Tuesday, August 13, 2013 at 4:13 PM, Pankaj Gupta wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Spent some more time debugging issues with FileChannel. The
>>>>>>>>>> problem seems to lock contention reading from FlumeEventQueue:
>>>>>>>>>>
>>>>>>>>>> I see a lot of threads like this:
>>>>>>>>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor" prio=10
>>>>>>>>>> tid=0x00007f857b378800 nid=0x404d waiting for monitor entry
>>>>>>>>>> [0x00007f821afee000]
>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.removeHead(FlumeEventQueue.java:117)
>>>>>>>>>>         - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:492)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:330)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:154)
>>>>>>>>>>         at
>>>>>>>>>> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>>>>>>>>         at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have two file channels and 8 Avro Sinks per file channel. I
>>>>>>>>>> added more sinks because they weren't draining fast enough. It seems like
>>>>>>>>>> they send the batch then wait for an ack before sending again, thus sends
>>>>>>>>>> are not pipelined and having more sinks seemed like a good way of getting
>>>>>>>>>> some parallelism.
>>>>>>>>>>
>>>>>>>>>> Here's the full stack trace:
>>>>>>>>>> 2013-08-13 15:30:32
>>>>>>>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.13-b02
>>>>>>>>>> mixed mode):
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c02c000 nid=0x41a2 waiting on condition [0x00007f8210341000]
>>>>>>>>>>    java.lang.Thread.State: WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000056d932120> (a
>>>>>>>>>> java.util.concurrent.CountDownLatch$Sync)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>>>>>>>>>> at org.apache.avro.ipc.CallFuture.await(CallFuture.java:141)
>>>>>>>>>>  at org.apache.avro.ipc.Requestor.request(Requestor.java:150)
>>>>>>>>>> at org.apache.avro.ipc.Requestor.request(Requestor.java:129)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:68)
>>>>>>>>>> at $Proxy7.appendBatch(Unknown Source)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:314)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.api.NettyAvroRpcClient$2.call(NettyAvroRpcClient.java:310)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Attach Listener" daemon prio=10 tid=0x00007f85fc016000
>>>>>>>>>> nid=0x41a1 waiting on condition [0x0000000000000000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c02a800 nid=0x4173 waiting on condition [0x00007f8210544000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005194c3340> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000f800 nid=0x4172 waiting on condition [0x00007f8210645000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191a16d8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec01f000 nid=0x4164 waiting on condition [0x00007f8210746000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174e08> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374023000 nid=0x4163 waiting on condition [0x00007f8210847000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174e98> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec030000 nid=0x415d waiting on condition [0x00007f8210948000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005195f9670> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374021800 nid=0x415b waiting on condition [0x00007f8210a49000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174ce8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000f000 nid=0x415a waiting on condition [0x00007f8210b4a000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191a1918> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c024000 nid=0x4154 waiting on condition [0x00007f8210c4b000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191e7f60> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-41" prio=10 tid=0x00007f85d8466800 nid=0x4153
>>>>>>>>>> runnable [0x00007f8210e4d000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000054a291760> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000054a2917a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000054a2c5348> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec029800 nid=0x4147 waiting on condition [0x00007f8210d4c000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174f58> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-41" prio=10 tid=0x00007f8584469800 nid=0x4146
>>>>>>>>>> runnable [0x00007f8210f4e000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000546e5dd40> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000546e5dd28> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000546ec1ea0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-41" prio=10 tid=0x00007f8588411000 nid=0x4118
>>>>>>>>>> runnable [0x00007f821104f000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000544f763d8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000544f763c0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000544ee55f8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-41" prio=10 tid=0x00007f85803ff000 nid=0x4111
>>>>>>>>>> runnable [0x00007f8211150000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000054473d5c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000054473d5a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000054474d088> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-40" prio=10 tid=0x00007f85843c7800 nid=0x4110
>>>>>>>>>> runnable [0x00007f8211251000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005437f4f88> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005437f4f70> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000543803478> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-40" prio=10 tid=0x00007f858840f800 nid=0x410f
>>>>>>>>>> runnable [0x00007f8211352000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005422a9030> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005422a9018> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005422ca310> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-39" prio=10 tid=0x00007f85843c6000 nid=0x410e
>>>>>>>>>> runnable [0x00007f8211453000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000541063d68> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000541063d50> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005410c8240> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-40" prio=10 tid=0x00007f85d83c6000 nid=0x410d
>>>>>>>>>> runnable [0x00007f8211554000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000540dc1128> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000540dc1110> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000540c94928> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-39" prio=10 tid=0x00007f85d83c4800 nid=0x410a
>>>>>>>>>> runnable [0x00007f8211655000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053fbbc3c8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053fbbc3b0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053f897d00> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-38" prio=10 tid=0x00007f85843c4800 nid=0x4109
>>>>>>>>>> runnable [0x00007f8211756000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053f812878> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053f812860> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053f897cb8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-38" prio=10 tid=0x00007f85d83c3000 nid=0x40fd
>>>>>>>>>> runnable [0x00007f8211857000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000053e6e5588> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000053e6e5570> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000053e714e70> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f8374020800 nid=0x40f3 waiting on condition [0x00007f8211958000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519175138> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c022000 nid=0x40f2 waiting on condition [0x00007f8211a59000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000519174c28> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec032000 nid=0x40f1 waiting on condition [0x00007f8211b5a000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191751c8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-39" prio=10 tid=0x00007f85883ad800 nid=0x40f0
>>>>>>>>>> runnable [0x00007f8211c5b000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000536579dc8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000536579db0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000536585830> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000d800 nid=0x40ef waiting on condition [0x00007f8211d5c000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x00000005191750a8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-38" prio=10 tid=0x00007f85883ac000 nid=0x40e1
>>>>>>>>>> runnable [0x00007f8211e5d000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005313943c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005313943a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005314e0fe0> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f837401f000 nid=0x40e0 waiting on condition [0x00007f8211f5e000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936f460> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f82ec028000 nid=0x40df waiting on condition [0x00007f821205f000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936dfa8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-37" prio=10 tid=0x00007f85883aa800 nid=0x40de
>>>>>>>>>> runnable [0x00007f8212160000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052dad4480> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052dad4468> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052da31528> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-40" prio=10 tid=0x00007f85803fd800 nid=0x40dd
>>>>>>>>>> runnable [0x00007f8212261000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052c73c608> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052c73c5f0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052c652608> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-36" prio=10 tid=0x00007f85883a9000 nid=0x40dc
>>>>>>>>>> runnable [0x00007f8212362000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x000000052b8da9b0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x000000052b8da998> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x000000052b8fc590> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-39" prio=10 tid=0x00007f85803fc000 nid=0x40db
>>>>>>>>>> waiting for monitor entry [0x00007f8212462000]
>>>>>>>>>>    java.lang.Thread.State: BLOCKED (on object monitor)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue.addWithoutCommit(FlumeEventQueue.java:184)
>>>>>>>>>> - waiting to lock <0x0000000518ee4c90> (a
>>>>>>>>>> org.apache.flume.channel.file.FlumeEventQueue)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:460)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
>>>>>>>>>> at
>>>>>>>>>> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:330)
>>>>>>>>>> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>>>>>>>>>>  at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
>>>>>>>>>> at org.apache.avro.ipc.Responder.respond(Responder.java:149)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>  at
>>>>>>>>>> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f836000b000 nid=0x40da waiting on condition [0x00007f8212564000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>>  at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x000000051936df18> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "Flume Avro RPC Client Call Invoker 1" daemon prio=10
>>>>>>>>>> tid=0x00007f834c020000 nid=0x40d9 waiting on condition [0x00007f8212665000]
>>>>>>>>>>    java.lang.Thread.State: TIMED_WAITING (parking)
>>>>>>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>>>>>>> - parking to wait for  <0x0000000518fe7ae8> (a
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-142-thread-37" prio=10 tid=0x00007f85843c3000 nid=0x40d8
>>>>>>>>>> runnable [0x00007f8212766000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x00000005263d6bc0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x00000005263d6ba8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x00000005264b1a48> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-37" prio=10 tid=0x00007f85d83c1800 nid=0x40d7
>>>>>>>>>> runnable [0x00007f8212867000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000525d1d0f8> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000525d1d0e0> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000525d2cdf8> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-144-thread-36" prio=10 tid=0x00007f85d83c0000 nid=0x40d6
>>>>>>>>>> runnable [0x00007f8212968000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000525a689f0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000525a689d8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000525a24570> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-140-thread-35" prio=10 tid=0x00007f85883a8000 nid=0x40d5
>>>>>>>>>> runnable [0x00007f8212a69000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000524e1e7f0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000524e1e7d8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000524cfd670> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>  at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>
>>>>>>>>>> "pool-139-thread-38" prio=10 tid=0x00007f85803fa800 nid=0x40d4
>>>>>>>>>> runnable [0x00007f8212b6a000]
>>>>>>>>>>    java.lang.Thread.State: RUNNABLE
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>>>>>>>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>>>>>>>>>>  at
>>>>>>>>>> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>>>>>>>>>>  - locked <0x0000000524d437c0> (a sun.nio.ch.Util$2)
>>>>>>>>>> - locked <0x0000000524d437a8> (a
>>>>>>>>>> java.util.Collections$UnmodifiableSet)
>>>>>>>>>>  - locked <0x0000000524ab3330> (a sun.nio.ch.EPollSelectorImpl)
>>>>>>>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:200)
>>>>>>>>>>  at
>>>>>>>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
>>>>>>>>>> at
>>>>>>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>>>>>>>
>>>>>>>>>> ...
>>>
>>>
>>
>>
>> --
>>
>>
>> *P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 |
>> pankaj@brightroll.com
>>
>> Pankaj Gupta | Software Engineer
>>
>> *BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com
>>
>>
>> United States | Canada | United Kingdom | Germany
>>
>>
>> We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
>> !
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
>



-- 


*P* | (415) 677-9222 ext. 205 *F *| (415) 677-0895 | pankaj@brightroll.com

Pankaj Gupta | Software Engineer

*BrightRoll, Inc. *| Smart Video Advertising | www.brightroll.com


United States | Canada | United Kingdom | Germany


We're hiring<http://newton.newtonsoftware.com/career/CareerHome.action?clientId=8a42a12b3580e2060135837631485aa7>
!

Mime
View raw message