flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Gandevia <cgande...@gmail.com>
Subject Re: Weird interrupted exception in DirectDriver during append
Date Mon, 17 Oct 2011 21:11:58 GMT
I have been experiencing a similar error and notice it only happens when I
have a large number of files open to hdfs. I am running some tests removing
the bucketing to see if I can send the same files. I will let you know if I
come across anything. I have around 50 nodes writing to a single collector.
This error consistently happens within 10 mins of starting my collector.

My exception.

2011-10-17 17:30:07,173 [Roll-TriggerThread-0] INFO
 com.cloudera.flume.handlers.hdfs.CustomDfsSink - done writing raw file to
hdfs
2011-10-17 17:30:07,189 [logicalNode collector0_log_dir-19] ERROR
com.cloudera.flume.core.connector.DirectDriver - Closing down due to
exception during append calls
2011-10-17 17:30:07,190 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.core.connector.DirectDriver - Connector logicalNode
collector0_log_dir-19 exited with error: Blocked append interrupted by
rotation event
java.lang.InterruptedException: Blocked append interrupted by rotation event
        at
com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:169)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:71)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
        at
com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129)
        at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
        at
com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88)
        at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:133)
2011-10-17 17:30:07,191 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.collector.CollectorSource - closed
2011-10-17 17:30:08,191 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.handlers.thrift.ThriftEventSource - Closed server on
port 36892...
2011-10-17 17:30:08,191 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.handlers.thrift.ThriftEventSource - Queue still has 1000
elements ...
2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] WARN
 com.cloudera.flume.handlers.thrift.ThriftEventSource - Close timed out due
to no progress.  Closing despite having 1000 values still enqueued
2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.handlers.rolling.RollSink - closing RollSink
'escapedCustomDfs("hdfs://
van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}"
)'
2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] INFO
 com.cloudera.flume.handlers.rolling.RollSink - double close
'escapedCustomDfs("hdfs://
van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}"
)'
2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] ERROR
com.cloudera.flume.core.connector.DirectDriver - Exiting driver logicalNode
collector0_log_dir-19 in error state CollectorSource | RegexExtractor
because Blocked append interrupted by rotation event


On Mon, Oct 17, 2011 at 12:47 PM, Stephen Layland <stephen.layland@gmail.com
> wrote:

> Hi, we're not actually using flume nodes, but just the collector at the
> moment.  We're listening on a syslog port and dumping straight to HDFS for
> now.  After some digging, I'm pretty sure it's related to this:
>
> https://issues.apache.org/jira/browse/FLUME-757
>
> -Steve
>
>
> On Mon, Oct 17, 2011 at 11:53 AM, AD <straightflush@gmail.com> wrote:
>
>> weird i have been seeing the same thing.  Do you have the node and
>> collector on different hosts?  Are you using hbase by chance?
>>
>>
>> On Mon, Oct 17, 2011 at 1:48 PM, Stephen Layland <
>> stephen.layland@gmail.com> wrote:
>>
>>> Hi, after letting flume idle for the weekend listening in on a small
>>> stream of live data, we noticed several of our flume collector nodes failing
>>> with InterruptedException's being thrown.  Logs have errors that look
>>> something like below.  Any idea of what's going on here and how to fix it?
>>>
>>> 2011-10-17 08:02:02,107 INFO
>>>> com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted
>>>> event 'flume-node.lindenlab.com' [INFO Mon Oct 17 08:01:59 UTC 2011] {
>>>> syslogfacility : 16 } { syslogseverity : 6 } Oct 17 01:01:59 SOME MESSAGE'
>>>> with error: Blocked append interrupted by rotation event
>>>> 2011-10-17 08:02:02,107 INFO
>>>> com.cloudera.flume.handlers.rolling.RollSink: closing RollSink
>>>> 'escapedCustomDfs("hdfs://master-node:54310/logs/raw/%Y/%m/%d/%H00","test%{rolltag}"
>>>> )'
>>>> 2011-10-17 08:02:02,109 ERROR
>>>> com.cloudera.flume.core.connector.DirectDriver: Closing down due to
>>>> exception during append calls
>>>> java.lang.InterruptedException
>>>>         at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>>>>         at
>>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>>>>         at
>>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>>>>         at
>>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>>>>         at
>>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
>>>> 2011-10-17 08:02:02,109 INFO
>>>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode
>>>> node6-22 exited with error: nulljava.lang.InterruptedException
>>>>         at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>>>>
>>>>         at
>>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>>>>         at
>>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>>>>         at
>>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>>>>         at
>>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
>>>> 2011-10-17 08:02:02,109 INFO
>>>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode
>>>> node6-22 exited with error: null
>>>> java.lang.InterruptedException
>>>>         at
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>>>>         at
>>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>>>>         at
>>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>>>>         at
>>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>>>>         at
>>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>>>>         at
>>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>>>>         at
>>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
>>>
>>>
>>> Many thanks,
>>>
>>> -Steve
>>>
>>
>>
>


-- 
Thanks

Cameron Gandevia

Mime
View raw message