flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "huyanping (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLUME-3011) metrics messages does not change when spooling directory source exit abnormaly
Date Fri, 21 Oct 2016 04:28:58 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593941#comment-15593941
] 

huyanping edited comment on FLUME-3011 at 10/21/16 4:28 AM:
------------------------------------------------------------

I found that all the StopTime fields of every type of sources doest not change when the source
exit abnormally. Maybe it it better to users that add a new field or just change the value
of the StopTime field when the sources exit abnormally.


was (Author: jenner):
I found that all the StopTime fields of every type of sources doest not change when the source
exit abnormally. Maybe it it better for users that add a new field or just change the value
of the StopTime field when the sources exit abnormally.

> metrics messages does not change when spooling directory source exit abnormaly
> ------------------------------------------------------------------------------
>
>                 Key: FLUME-3011
>                 URL: https://issues.apache.org/jira/browse/FLUME-3011
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.6.0, v1.5.2
>            Reporter: huyanping
>
> I found this error in my flume logs. The source_xxx has stopped but the StopTime field
of metrics message is not changed. 
> 21 Oct 2016 11:26:02,460 ERROR [pool-88871-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:256)
 - FATAL: Spool Directory source source_xxx: { spoolDir: /data/flume/logs/l }: Uncaught exception
in SpoolDirectorySource thread. Restart or reconfigure Flume to continue processing.
> java.lang.OutOfMemoryError: Direct buffer memory
>         at java.nio.Bits.reserveMemory(Bits.java:658)
>         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
>         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
>         at org.apache.flume.serialization.ResettableFileInputStream.<init>(ResettableFileInputStream.java:108)
>         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.openFile(ReliableSpoolingFileEventReader.java:491)
>         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile(ReliableSpoolingFileEventReader.java:459)
>         at org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:229)
>         at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:227)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)      
> The metrics message:
> "SOURCE.source_xxx":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"20660","AppendBatchReceivedCount":"20660","EventAcceptedCount":"165337","AppendReceivedCount":"0","StopTime":"0","StartTime":"1475502164604","EventReceivedCount":"165337","AppendAcceptedCount":"0"}
          



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message