flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ronald Van De Kuil <ronald.van.de.k...@gmail.com>
Subject Re: Flume not marking log files as completed and do not process file further
Date Wed, 20 Apr 2016 06:53:36 GMT
Not sure that this helps, ... have you checked your operating system settings for the maximum
number of open files? 

Met vriendelijke groet,
Ronald van de Kuil

> Op 20 apr. 2016 om 08:48 heeft Saurabh Sharma <saurabh.sharma@nviz.com> het volgende
geschreven:
> 
> Hi,
>  
> I have a scenario where we are ingesting the log files(around 80MB each) in flume and
flume does process these files and marks as completed but after processing few files it does
not process any files further and moreover it does not mark the log file as completed.
>  
> We are using spooling directory source.
>  
> I looked into the flume logs and found that when this scenario happens it shows the following
line continuously in flume logs.
>  
> DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126)
 - Checking file:../conf/commerce-sense.conf for changes
>  
> We have following configuration.
> agent.sources = spoolDir
> agent.channels = memoryChannel
> agent.sinks = sink
> agent.sources.spoolDir.interceptors = i1
>  
>  
> #Channel Configuration
> agent.channels.memoryChannel.type = memory
>  
> #Source configuration
> agent.sources.spoolDir.type = spooldir
> agent.sources.spoolDir.spoolDir = /opt/flume/spoolDir
> agent.sources.spoolDir.fileHeader = true
> agent.sources.spoolDir.basenameHeader = true
> agent.sources.spoolDir.deserializer = LINE
> agent.sources.spoolDir.inputCharset = ISO8859-1
> agent.sources.spoolDir.deserializer.maxLineLength = 10000
> agent.sources.spoolDir.interceptors.i1.type = org.apache.flume.sink.solr.morphline.UUIDInterceptor$Builder
> agent.sources.spoolDir.interceptors.i1.preserveExisting = true
> agent.sources.spoolDir.interceptors.i1.prefix = test
> agent.sources.spoolDir.channels = memoryChannel
>  
>  
> #Sink Configuration
> agent.sinks.sink.type = com. flume.sink.ExtendedKafkaSink
> agent.sinks.sink.topic = cdnLogsTopic
> agent.sinks.sink.brokerList = localhost:9092
> agent.sinks.sink.batchSize = 100
> agent.sinks.sink.sink.serializer = com. flume.serializer.ExtendedSerializer$Builder
> agent.sinks.sink.channel = memoryChannel
>  
> Thanks,
> Saurabh

Mime
View raw message