flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shady Xu <shad...@gmail.com>
Subject Re: The channel is full, and cannot write data now (Flume 1.5)
Date Wed, 12 Aug 2015 06:39:21 GMT
Flume source cannot put data into channel unless there is free space. You
can try increasing the capacity of the channel.

2015-08-12 2:05 GMT+08:00 George Blazer <gblazer@gmail.com>:

> Hellok
>
> I'm getting "Channel is full" error because ChannelFillPercentage is
> nearly at 100%.
>
> What can I look at?
>
> 2015-08-11 17:55:51,173 (pool-5-thread-1) [INFO -
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)]
> Last read was never committed - resetting mark position.
> 2015-08-11 17:55:54,195 (pool-5-thread-1) [WARN -
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)]
> *The channel is full, and cannot write data now. The source will try again
> after 4000 milliseconds*
>
> curl localhost:5653/metrics
>
> {"SINK.s3-sink":{"BatchCompleteCount":"0","ConnectionFailedCount":"1","EventDrainAttemptCount":"0","ConnectionCreatedCount":"0","Type":"SINK","BatchEmptyCount":"0","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1439315713336","BatchUnderflowCount":"0"},"SOURCE.spooling-directory":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"10","EventAcceptedCount":"0","AppendReceivedCount":"0","StopTime":"0","StartTime":"1439315713378","EventReceivedCount":"1000","AppendAcceptedCount":"0"},"CHANNEL.fileChannel":{"EventPutSuccessCount":"0","ChannelFillPercentage":"99.9956","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"450","ChannelSize":"999956","StartTime":"1439315713327","EventTakeSuccessCount":"0","ChannelCapacity":"1000000","EventTakeAttemptCount":"1"}}
>
> My current config is:
>
> agent_messaging.sinks = s3-sink
> agent_messaging.sources = spooling-directory
> agent_messaging.channels = fileChannel
>
> agent_messaging.channels.fileChannel.type = file
> agent_messaging.channels.fileChannel.checkpointDir = /mnt/flume/checkpoint
> agent_messaging.channels.fileChannel.dataDirs = /mnt/flume/data
>
> agent_messaging.sources.spooling-directory.channels = fileChannel
> agent_messaging.sources.spooling-directory.type = spooldir
> agent_messaging.sources.spooling-directory.deletePolicy = immediate
> agent_messaging.sources.spooling-directory.spoolDir = /mnt/flume/spool
>
> agent_messaging.sources.spooling-directory.interceptors = sr i1 ts
> agent_messaging.sources.spooling-directory.interceptors.sr.type =
> org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder
> agent_messaging.sources.spooling-directory.interceptors.sr.searchPattern =
> (\\d{4})/(\\d{2})/(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2})
> agent_messaging.sources.spooling-directory.interceptors.sr.replaceString =
> $1-$2-$3 $4:$5:$6
>
> agent_messaging.sources.spooling-directory.interceptors.i1.type =
> regex_extractor
> agent_messaging.sources.spooling-directory.interceptors.i1.regex =
> ^(?:\\n)?(\\d\\d\\d\\d-\\d\\d-\\d\\d\\s\\d\\d:\\d\\d:\\d\\d)
> agent_messaging.sources.spooling-directory.interceptors.i1.serializers = s1
> agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.type
> = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
>
> agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.name
> = timestamp
> agent_messaging.sources.spooling-directory.interceptors.i1.serializers.s1.pattern
> = yyyy-MM-dd HH:mm:ss
>
> agent_messaging.sources.spooling-directory.interceptors.ts.type =
> org.apache.flume.interceptor.TimestampInterceptor$Builder
> agent_messaging.sources.spooling-directory.interceptors.ts.preserveExisting
> = true
>
> agent_messaging.sources.spooling-directory.deserializer.maxLineLength =
> 10000
>
> agent_messaging.sinks.s3-sink.channel = fileChannel
> agent_messaging.sinks.s3-sink.type = hdfs
> agent_messaging.sinks.s3-sink.hdfs.path = s3n://SECRETS@logs
> /events/%Y-%m-%d
> agent_messaging.sinks.s3-sink.hdfs.filePrefix = prod-role-8599226c
> agent_messaging.sinks.s3-sink.hdfs.rollInterval = 0
> agent_messaging.sinks.s3-sink.hdfs.rollSize = 67108864
> agent_messaging.sinks.s3-sink.hdfs.rollCount = 0
> agent_messaging.sinks.s3-sink.hdfs.serializer = TEXT
> agent_messaging.sinks.s3-sink.hdfs.fileType = DataStream
> agent_messaging.sinks.s3-sink.hdfs.closeTries = 3
> agent_messaging.sinks.s3-sink.hdfs.idleTimeout = 0
> agent_messaging.sinks.s3-sink.hdfs.callTimeout = 180000
>

Mime
View raw message