flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: ExecSource tail -F stops after a while
Date Mon, 05 Nov 2012 13:51:17 GMT
Can you take a thread dump, using kill -3 not jstack, when you hit that point?


On Monday, November 5, 2012 at 7:39 AM, Parag Hukeri wrote:

> While starting my flume-agent I did use these options:-  java -Xms512m -Xmx2048m. But
that did not help. What should I tweak in flume-env.sh? Not sure?
> I noticed that the source for fileChannel was included in flume 1.2. So I tried compiling
that & using that jar; but got the exact same results with fileChannel - the process stopped
after approx 9.5 hrs (after having written approx 9.5 million records).
>  
> Any help would be appreciated. Thanks.
> 
> --- On Sat, 11/3/12, Alexander Lorenz <wget.null@gmail.com> wrote:
> > 
> > From: Alexander Lorenz <wget.null@gmail.com>
> > Subject: Re: ExecSource tail -F stops after a while
> > To: user@flume.apache.org
> > Date: Saturday, November 3, 2012, 1:55 AM
> > 
> > Hi,
> > 
> > http://flume.apache.org/FlumeUserGuide.html#exec-source
> > Please see the WARNING ;)
> > 
> > Anyhow, this error indicates that you've to less memory configured for too many
events. Tune flume-env.sh and tweak JAVA_OPTS into "-Xms100m -Xmx200m" or something else
> > 
> > FileChannel is available in from up to 1.3
> > http://flume.apache.org/FlumeUserGuide.html#recoverable-memory-channel
> > 
> > cheers
> > - Alex
> > 
> > 
> > On Nov 3, 2012, at 1:21 AM, Parag Hukeri <paraghukeri@yahoo.com (http://us.mc1634.mail.yahoo.com/mc/compose?to=paraghukeri@yahoo.com)>
wrote:
> > 
> > > I tried to use the FileChannel; but am getting an exception -
> > >  
> > > org.apache.flume.FlumeException: Unable to load channel type: file, class:
org.apache.flume.channel.file.FileChannel
> > > Caused by: java.lang.ClassNotFoundException: org.apache.flume.channel.file.FileChannel
> > >  
> > > I am using  apache-flume-1.2.0. What version of flume is needed for FileChannel?
> > >  
> > > Thanks
> > > 
> > > --- On Fri, 11/2/12, Josh West <jsw@one.com (http://us.mc1634.mail.yahoo.com/mc/compose?to=jsw@one.com)>
wrote:
> > > 
> > > 
> > > From: Josh West <jsw@one.com (http://us.mc1634.mail.yahoo.com/mc/compose?to=jsw@one.com)>
> > > Subject: Re: ExecSource tail -F stops after a while
> > > To: user@flume.apache.org (http://us.mc1634.mail.yahoo.com/mc/compose?to=user@flume.apache.org)
> > > Date: Friday, November 2, 2012, 1:48 PM
> > > 
> > > 
> > > 
> > > This happened to me consistently when buffering tail -F contents into the Memory
Channel.  Try switching to the File Channel.  This solved the issue for me.
> > > 
> > > 
> > > On 11/02/2012 07:15 PM, Parag Hukeri wrote:
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Hi
> > >  
> > > I am using apache-flume-1.2.0. I have set up an ExecSource that tails a log
file & gets the entries from this log file. The log file size is approx 4 GB (approx 12
million lines written over a period of 12 hrs). As the process starts everything seems to
work fine for approx 9.5 hrs. (approx 9.5 million lines of log data is successfully retrieved
by ExecSource). It then stops reading the output from tail. I don't see any exceptions being
thrown. I don't think the problem is with the actual tail command. I was able to successfully
manually tail the log file & all 12 million records were retrieved by tail. Running on
a Linux system. Not sure what the issue might be with ExecSource? Any help would be appreciated.
Thanks.
> > > -- 
> > > Josh West
> > > Lead Systems Administrator
> > > One.com, jsw@one.com (http://us.mc1634.mail.yahoo.com/mc/compose?to=jsw@one.com)
> > 
> > --
> > Alexander Alten-Lorenz
> > http://mapredit.blogspot.com (http://mapredit.blogspot.com/)
> > German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> > 


Mime
View raw message