Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 30CFC1120A for ; Thu, 17 Jul 2014 06:42:16 +0000 (UTC) Received: (qmail 29564 invoked by uid 500); 17 Jul 2014 06:42:15 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 29513 invoked by uid 500); 17 Jul 2014 06:42:15 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 29503 invoked by uid 99); 17 Jul 2014 06:42:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 06:42:15 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of paliwalashish@gmail.com designates 74.125.82.174 as permitted sender) Received: from [74.125.82.174] (HELO mail-we0-f174.google.com) (74.125.82.174) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 06:42:11 +0000 Received: by mail-we0-f174.google.com with SMTP id x48so1935607wes.5 for ; Wed, 16 Jul 2014 23:41:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=BfYsSwT5cMij460YBjqXlLIex5OQ73kH982GHNA3Ua0=; b=Y87U933FaGRo2yv6UQbwhHQfPt0JHdex7wP9lPLWED5rZQ3PsyLAQwIcKxT2Ai0Po3 IKJu552dFXTQUhJ1hpyduKqoz/WjjRIcRsR/Ue4C1vAW1GRhDpn6dTgzlEeaFJVi+KV/ /A3N/7y4Ne2NVN5+dHU0/SDw8kFZ/Kou55OJxkFI6lv0i1kxRNBz8rTCz3Zuz4PhznyX xR5brSIHE0FSDycPN9QeO7OGdcufvICoB2hQMknavBr5ESftT7UeXVI64/yU+9g9rOkE lMquClmpvHpOYCOJNRRywe1AxujtrEI7PQ1nSJbssCfFMxdOIGAtV3Os2gPdq6hK4Jl1 7XTg== MIME-Version: 1.0 X-Received: by 10.194.60.35 with SMTP id e3mr43213015wjr.12.1405579310525; Wed, 16 Jul 2014 23:41:50 -0700 (PDT) Received: by 10.194.29.167 with HTTP; Wed, 16 Jul 2014 23:41:50 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 Jul 2014 12:11:50 +0530 Message-ID: Subject: Re: Flume stops processing event after a while From: Ashish To: user@flume.apache.org Content-Type: multipart/alternative; boundary=047d7b86e8ba81f35904fe5def22 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86e8ba81f35904fe5def22 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Nope, a heap dump shall be generated. Please see more options at http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.htm= l to specify path use this -XX:HeapDumpPath=3D./java_pid.hprof On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR wrote: > yes , sorry I missed to update as 1 GB. > > But for out of memory error ,do we get notified in flume logs? I haven't > see any exception till now. > > > On 17 July 2014 11:55, SaravanaKumar TR wrote: > >> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with >> >> JAVA_OPTS=3D"-Xms100m -Xmx200m -Dcom.sun.management.jmxremote >> -XX:-HeapDumpOnOutOfMemoryError" >> >> >> On 17 July 2014 11:39, Ashish wrote: >> >>> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your process >>> is OOME, would generate a Heap dump. Allocate Heap based on the number = of >>> events you need to keep in channel. Try with 1 GB, but calculate accord= ing >>> the Channel size as (average event size * number of events), plus objec= t >>> over heads. >>> >>> Please note, this is just a rough calculation, actual memory usage woul= d >>> be higher. >>> >>> >>> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR < >>> saran0081986@gmail.com> wrote: >>> >>>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory fo= r >>>> flume agent. >>>> >>>> But I am surprised why there was no error registered for this memory >>>> issues in log file (flume.log). >>>> >>>> Do i need to check in any other logs? >>>> >>>> >>>> On 16 July 2014 21:55, Jonathan Natkins wrote: >>>> >>>>> That's definitely your problem. 20MB is way too low for this. >>>>> Depending on the other processes you're running with your system, the >>>>> amount of memory you'll need will vary, but I'd recommend at least 1G= B. You >>>>> should define it exactly where it's defined right now, so instead of = the >>>>> current command, you can run: >>>>> >>>>> "/cv/jvendor/bin/java -Xmx1g -Dflume.root.logger=3DDEBUG,LOGFILE.....= ." >>>>> >>>>> >>>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR < >>>>> saran0081986@gmail.com> wrote: >>>>> >>>>>> I guess i am using defaulk values , from running flume i could see >>>>>> these lines "/cv/jvendor/bin/java -Xmx20m >>>>>> -Dflume.root.logger=3DDEBUG,LOGFILE......" >>>>>> >>>>>> so i guess it takes 20 mb as agent flume memory. >>>>>> My RAM is 128 GB.So please suggest how much can i assign as heap >>>>>> memory and where to define it. >>>>>> >>>>>> >>>>>> On 16 July 2014 15:05, Jonathan Natkins wrote= : >>>>>> >>>>>>> Hey Saravana, >>>>>>> >>>>>>> I'm attempting to reproduce this, but do you happen to know what th= e >>>>>>> Java heap size is for your Flume agent? This information leads me t= o >>>>>>> believe that you don't have enough memory allocated to the agent, w= hich you >>>>>>> may need to do with the -Xmx parameter when you start up your agent= . That >>>>>>> aside, you can set the byteCapacity parameter on the memory channel= to >>>>>>> specify how much memory it is allowed to use. It should default to = 80% of >>>>>>> the Java heap size, but if your heap is too small, this might be a = cause of >>>>>>> errors. >>>>>>> >>>>>>> Does anything get written to the log when you try to pass in an >>>>>>> event of this size? >>>>>>> >>>>>>> Thanks, >>>>>>> Natty >>>>>>> >>>>>>> >>>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR < >>>>>>> saran0081986@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Natty, >>>>>>>> >>>>>>>> While looking further , i could see memory channal stops if a line >>>>>>>> comes with greater than 2 MB.Let me know which parameter helps us = to define >>>>>>>> max event size of about 3 MB. >>>>>>>> >>>>>>>> >>>>>>>> On 16 July 2014 12:46, SaravanaKumar TR >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I am asking point 1 , because in some cases I could see a line i= n >>>>>>>>> logfile around 2 MB.So i need to know what mamimum event size.How= to >>>>>>>>> measure it? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 16 July 2014 10:18, SaravanaKumar TR >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Natty, >>>>>>>>>> >>>>>>>>>> Please help me to get the answers for the below queries. >>>>>>>>>> >>>>>>>>>> 1,In case of exec source , (tail -F ) , is that each >>>>>>>>>> line in file is considered to be a single event ? >>>>>>>>>> If suppose a line is considered to be a event , what is that >>>>>>>>>> maximum size of event supported by flume?I mean maximum characte= rs in a >>>>>>>>>> line supported? >>>>>>>>>> 2.When event stop processing , I am not seeing "tail -F" command >>>>>>>>>> running in the background. >>>>>>>>>> I have used option like "a1.sources.r1.restart =3D true >>>>>>>>>> a1.sources.r1.logStdErr =3D true".. >>>>>>>>>> Does these config will not send any errors to flume.log if any >>>>>>>>>> issues in tail? >>>>>>>>>> Will this config doesnt try to restart the "tail -F" if its not >>>>>>>>>> running in the background. >>>>>>>>>> >>>>>>>>>> 3.Does flume supports all formats of data in logfile or it has >>>>>>>>>> any predefined data formats.. >>>>>>>>>> >>>>>>>>>> Please help me with these to understand better.. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 16 July 2014 00:56, Jonathan Natkins >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Saravana, >>>>>>>>>>> >>>>>>>>>>> Everything here looks pretty sane. Do you have a record of the >>>>>>>>>>> events that came in leading up to the agent stopping collection= ? If you can >>>>>>>>>>> provide the last file created by the agent, and ideally whateve= r events had >>>>>>>>>>> come in, but not been written out to your HDFS sink, it might b= e possible >>>>>>>>>>> for me to reproduce this issue. Would it be possible to get som= e sample >>>>>>>>>>> data from you? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Natty >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar TR < >>>>>>>>>>> saran0081986@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Natty , >>>>>>>>>>>> >>>>>>>>>>>> Just to understand , at present my settings is as >>>>>>>>>>>> "flume.root.logger=3DINFO,LOGFILE" >>>>>>>>>>>> in log4j.properties , do you want me to change it to >>>>>>>>>>>> "flume.root.logger=3DDEBUG,LOGFILE" and restart the agent. >>>>>>>>>>>> >>>>>>>>>>>> But when I start agent , I am already starting with below >>>>>>>>>>>> command.I guess i am using DEBUG already but not in config fil= e , while >>>>>>>>>>>> starting agent. >>>>>>>>>>>> >>>>>>>>>>>> ../bin/flume-ng agent -c /d0/flume/conf -f >>>>>>>>>>>> /d0/flume/conf/flume-conf.properties -n a1 -Dflume.root.logger= =3DDEBUG,LOGFILE >>>>>>>>>>>> >>>>>>>>>>>> If I do some changes in config "flume-conf.properties" or >>>>>>>>>>>> restart the agent , it works again and starts collecting the d= ata. >>>>>>>>>>>> >>>>>>>>>>>> currently all my logs move to flume.log , I dont see any >>>>>>>>>>>> exception . >>>>>>>>>>>> >>>>>>>>>>>> cat flume.log | grep "Exception" doesnt show any. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 15 July 2014 22:24, Jonathan Natkins >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>> >>>>>>>>>>>>> Our best bet on figuring out what's going on here may be to >>>>>>>>>>>>> turn on the debug logging. What I would recommend is stopping= your agents, >>>>>>>>>>>>> and modifying the log4j properties to turn on DEBUG logging f= or the root >>>>>>>>>>>>> logger, and then restart the agents. Once the agent stops pro= ducing new >>>>>>>>>>>>> events, send out the logs and I'll be happy to take a look ov= er them. >>>>>>>>>>>>> >>>>>>>>>>>>> Does the system begin working again if you restart the agents= ? >>>>>>>>>>>>> Have you noticed any other events correlated with the agent s= topping >>>>>>>>>>>>> collecting events? Maybe a spike in events or something like = that? And for >>>>>>>>>>>>> my own peace of mind, if you run `cat /var/log/flume-ng/* | g= rep >>>>>>>>>>>>> "Exception"`, does it bring anything back? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> Natty >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jul 15, 2014 at 2:55 AM, SaravanaKumar TR < >>>>>>>>>>>>> saran0081986@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Natty, >>>>>>>>>>>>>> >>>>>>>>>>>>>> This is my entire config file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Name the components on this agent >>>>>>>>>>>>>> a1.sources =3D r1 >>>>>>>>>>>>>> a1.sinks =3D k1 >>>>>>>>>>>>>> a1.channels =3D c1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Describe/configure the source >>>>>>>>>>>>>> a1.sources.r1.type =3D exec >>>>>>>>>>>>>> a1.sources.r1.command =3D tail -F /data/logs/test_log >>>>>>>>>>>>>> a1.sources.r1.restart =3D true >>>>>>>>>>>>>> a1.sources.r1.logStdErr =3D true >>>>>>>>>>>>>> >>>>>>>>>>>>>> #a1.sources.r1.batchSize =3D 2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> a1.sources.r1.interceptors =3D i1 >>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type =3D regex_filter >>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex =3D resuming normal >>>>>>>>>>>>>> operations|Received|Response >>>>>>>>>>>>>> >>>>>>>>>>>>>> #a1.sources.r1.interceptors =3D i2 >>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type =3D timestamp >>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting =3D true >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Describe the sink >>>>>>>>>>>>>> a1.sinks.k1.type =3D hdfs >>>>>>>>>>>>>> a1.sinks.k1.hdfs.path =3D hdfs:// >>>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=3D%Y-%m-%d >>>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat =3D Text >>>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType =3D DataStream >>>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix =3D events- >>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval =3D 600 >>>>>>>>>>>>>> ##need to run hive query randomly to check teh long running >>>>>>>>>>>>>> process , so we need to commit events in hdfs files regular= ly >>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount =3D 0 >>>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize =3D 10 >>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize =3D 0 >>>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp =3D true >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Use a channel which buffers events in memory >>>>>>>>>>>>>> a1.channels.c1.type =3D memory >>>>>>>>>>>>>> a1.channels.c1.capacity =3D 10000 >>>>>>>>>>>>>> a1.channels.c1.transactionCapacity =3D 10000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Bind the source and sink to the channel >>>>>>>>>>>>>> a1.sources.r1.channels =3D c1 >>>>>>>>>>>>>> a1.sinks.k1.channel =3D c1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 14 July 2014 22:54, Jonathan Natkins >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Saravana, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What does your sink configuration look like? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Natty >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar TR < >>>>>>>>>>>>>>> saran0081986@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Assuming each line in the logfile is considered as a event >>>>>>>>>>>>>>>> for flume , >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1.Do we have any maximum size of event defined for >>>>>>>>>>>>>>>> memory/file channel.like any maximum no of characters in a= line. >>>>>>>>>>>>>>>> 2.Does flume supports all formats of data to be processed >>>>>>>>>>>>>>>> as events or do we have any limitation. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am just still trying to understanding why the flume stop= s >>>>>>>>>>>>>>>> processing events after sometime. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can someone please help me out here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> saravana >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 11 July 2014 17:49, SaravanaKumar TR < >>>>>>>>>>>>>>>> saran0081986@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi , >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am new to flume and using Apache Flume 1.5.0. Quick >>>>>>>>>>>>>>>>> setup explanation here. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Source:exec , tail =E2=80=93F command for a logfile. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Channel: tried with both Memory & file channel >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sink: HDFS >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> When flume starts , processing events happens properly an= d >>>>>>>>>>>>>>>>> its moved to hdfs without any issues. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> But after sometime flume suddenly stops sending events to >>>>>>>>>>>>>>>>> HDFS. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am not seeing any errors in logfile flume.log as >>>>>>>>>>>>>>>>> well.Please let me know if I am missing any configuration= here. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Below is the channel configuration defined and I left the >>>>>>>>>>>>>>>>> remaining to be default values. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> a1.channels.c1.type =3D FILE >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity =3D 100000 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> a1.channels.c1.capacity =3D 10000000 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Saravana >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> thanks >>> ashish >>> >>> Blog: http://www.ashishpaliwal.com/blog >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >> >> > --=20 thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal --047d7b86e8ba81f35904fe5def22 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Nope, a heap dump shall be generated. Please see more opti= ons at=C2=A0http://www.oracle.com/technetwork/java/javase/tec= h/vmoptions-jsp-140102.html

to specify path use this=C2=A0-XX:HeapDumpPath= =3D./java_pid<pid>.hprof


On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar = TR <saran0081986@gmail.com> wrote:
yes , sorry I missed to update as 1 GB.

But for out of memory error ,do we get notified in flume logs? I haven'= ;t see any exception till now.


On 17 July 2014 11:55, SaravanaKumar TR <s= aran0081986@gmail.com> wrote:
Thanks Ashish , So I wil go ahead and update the flume-env= ,sh file with

JAVA_OPTS=3D"-Xms100m -Xmx200m -Dcom.= sun.management.jmxremote -XX:-HeapDumpOnOutOfMemoryError"


On 17 July 2014 11:39, Ashish <p= aliwalashish@gmail.com> wrote:
Add=C2=A0-XX:-HeapDumpOnOutOfMemoryError param= eter as well, if your process is OOME, would generate a Heap dump. Allocate= Heap based on the number of events you need to keep in channel. Try with 1= GB, but calculate according the Channel size as (average event size * numb= er of events), plus object over heads.

Please note, this is ju= st a rough calculation, actual memory usage would be higher.<= /div>


On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR <= ;saran0081986@g= mail.com> wrote:
Okay thanks , So for 128 GB= , I will allocate 1 GB as a heap memory for flume agent.

But I am surprised why there was no error registered for this memory issues= in log file (flume.log).

Do i need to check in any other logs?=C2=A0
=


On = 16 July 2014 21:55, Jonathan Natkins <natty@streamsets.com> wrote:
That's definitely your = problem. 20MB is way too low for this. Depending on the other processes you= 're running with your system, the amount of memory you'll need will= vary, but I'd recommend at least 1GB. You should define it exactly whe= re it's defined right now, so instead of the current command, you can r= un:

"/cv/jvendor/bin/java -Xmx1g -Dflume.root.logger=3DDEBU= G,LOGFILE......"

<= br>
On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKuma= r TR <saran0081986@gmail.com> wrote:
I guess i am using defaulk = values , from running flume i could see these lines =C2=A0"/cv/jvendor= /bin/java -Xmx20m -Dflume.root.logger=3DDEBUG,LOGFILE......"

so i guess it takes 20 mb as agent flume memory.
My RAM is 128 GB.So please suggest how much can i assign as heap memor= y and where to define it.
<= br>
On 16 July 2014 15:05, Jonathan Natkins <= span dir=3D"ltr"><natty@streamsets.com> wrote:
Hey Saravana,

I'm attempting to reproduce this, but do you happen to know what = the Java heap size is for your Flume agent? This information leads me to be= lieve that you don't have enough memory allocated to the agent, which y= ou may need to do with the -Xmx parameter when you start up your agent. Tha= t aside, you can set the byteCapacity parameter on the memory channel to sp= ecify how much memory it is allowed to use. It should default to 80% of the= Java heap size, but if your heap is too small, this might be a cause of er= rors.=C2=A0

Does anything get written to the log when you try to pa= ss in an event of this size?

Thanks,
Nat= ty


On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR <saran0081986@gmail= .com> wrote:
Hi Natty,

While looking further , i cou= ld see memory channal stops if a line comes with greater than 2 MB.Let me k= now which parameter helps us to define max event size of about 3 MB.


On 16 July 2014 12:46, SaravanaKumar TR <saran0081986@gmail.com> wrote:
I am asking point 1 , becau= se in some cases =C2=A0I could see a line in logfile around 2 MB.So i need = to know what mamimum event size.How to measure it?




On 16 July 2014 10:18, SaravanaKumar TR = <saran0081986@gmail.com> wrote:
Hi Natty,

Please help me to get the ans= wers for the below queries.

1,In case of exec sour= ce , (tail -F <logfile>) , is that each line in file is considered to= be a single event ?
If suppose a line is considered to be a event , what is that maximum s= ize of event supported by flume?I mean maximum characters in a line support= ed?
2.When event stop processing , I am not seeing "tail -F&= quot; command running in the background.
I have used option like "a1.sources.r1.restart =3D true
a1.sources.r1.logStdErr =3D true"..
Does these config will = not send any errors to flume.log if any issues in tail?
Will this= config doesnt try to restart the "tail -F" if its not running in= the background.

3.Does flume supports all formats of data in logfile or= it has any predefined data formats..

Please help = me with these to understand better..



On 16 July 2014 00:56, Jonathan Natkins = <natty@streamsets.com> wrote:
Saravana,

Everything here looks pretty = sane. Do you have a record of the events that came in leading up to the age= nt stopping collection? If you can provide the last file created by the age= nt, and ideally whatever events had come in, but not been written out to yo= ur HDFS sink, it might be possible for me to reproduce this issue. Would it= be possible to get some sample data from you?

Thanks,
Natty


On Tue, Jul 15, 2014 at= 10:26 AM, SaravanaKumar TR <saran0081986@gmail.com> wr= ote:
Hi Natty ,

Just to understand , at present my settings is as "flume.root.logge= r=3DINFO,LOGFILE"
in=C2=A0log4j.properties , do you want me to change it to "flume.= root.logger=3DDEBUG,LOGFILE" and restart the agent.

But when I start agent , I am already starting with bel= ow command.I guess i am using DEBUG already but not in config file , while = starting agent.

../bin/flume-ng agent -c /d0/flume= /conf -f /d0/flume/conf/flume-conf.properties -n a1 -Dflume.root.logger=3DD= EBUG,LOGFILE

If I do some changes in config "flume-conf.p= roperties" or restart the agent , it works again and starts collecting= the data.

currently all my logs move to flume.log= , I dont see any exception .

cat flume.log | grep "Exception" =C2=A0doesnt= show any.


On 15 July 2014 22:24, Jonathan Natkins <natty@s= treamsets.com> wrote:
Hi Saravana,

=
Our best bet on figuring out what's going on here may be to turn o= n the debug logging. What I would recommend is stopping your agents, and mo= difying the log4j properties to turn on DEBUG logging for the root logger, = and then restart the agents. Once the agent stops producing new events, sen= d out the logs and I'll be happy to take a look over them.

Does the system begin working again if you restart the = agents? Have you noticed any other events correlated with the agent stoppin= g collecting events? Maybe a spike in events or something like that? And fo= r my own peace of mind, if you run `cat /var/log/flume-ng/* | grep "Ex= ception"`, does it bring anything back?

Thanks!
Natty


On Tue, Jul 15, 2014 at= 2:55 AM, SaravanaKumar TR <saran0081986@gmail.com> wro= te:
Hi Natty,

This is my entire config file.

# Name the components on this agent
a1.sources =3D r1
a1.sinks =3D k1
a1.channels =3D c1

# Describe/configure the source
a1.sources.r1.type =3D exec<= /div>
= a1.sources.r1.command =3D tail -F /data/logs/test_log
a1.sources.r1.re= start =3D true
= a1.sources.r1.logStdErr =3D true

#a1.sources.r1.batchSize =3D 2

a1.sources.r1.interceptors = =3D i1
= a1.sources.r1.interceptors.i1.type =3D regex_filter
a1.sources.r1.inte= rceptors.i1.regex =3D resuming normal operations|Received|Response
=
#a1.sources.r1.interceptors =3D i2
#a1.sources.r1.interceptors.i2.type =3D timestamp
#a1.sources.r1.inter= ceptors.i2.preserveExisting =3D true

# Describe the sink
a1.sinks.k1.type =3D hdfs
a1.sinks.k1.hdfs.writeFormat =3D Text
a1.sinks.k1.hdfs.fileType =3D Da= taStream
a1.sinks.k1.hdfs.filePrefix =3D events-
a1.sinks.k1.hdfs.rollInterval = =3D 600
##need to run hive query randomly to check teh long running process , so we= =C2=A0need to commit events in hdfs files regularly
a1.sinks.k1.hdfs.= rollCount =3D 0
= a1.sinks.k1.hdfs.batchSize =3D 10
a1.sinks.k1.hdfs.rollSize =3D 0
a1.sinks.k1.hdfs.useLocalTimeStamp =3D true

# Use a channel = which buffers events in memory
= a1.channels.c1.type =3D memory
a1.channels.c1.capacity =3D 10000
=
a1.channels.c1.transactionCapacity =3D 10000

# Bind the sour= ce and sink to the channel
= a1.sources.r1.channels =3D c1
a1.sinks.k1.channel =3D c1


On 14 July 20= 14 22:54, Jonathan Natkins <natty@streamsets.com> wrote:<= br>
Hi Saravana,

What does your sink config= uration look like?

Thanks,
Natty


On Fri, Jul 11, 2014 at 11:05 PM, SaravanaKumar = TR <saran0081986@gmail.com> wrote:
Assuming each line in the l= ogfile is considered as a event for flume ,=C2=A0

1.Do w= e have any maximum size of event defined for memory/file channel.like any m= aximum no of characters in a line.
2.Does flume supports all formats of data to be processed as events or= do we have any limitation.

I am just still trying= to understanding why the flume stops processing events after sometime.

Can someone please help me out here.

Thanks,
saravana

On 11 July 2014 17:49, SaravanaKumar TR <saran0081986@gmail.com> wrote:
Hi ,

I a= m new to flume and =C2=A0using=C2=A0Apache Flume 1.5.0. Quick setup explana= tion here.

Source:exec , tail =E2=80=93F co= mmand for a logfile.

Channel: tried with both Memory & file channel

Sink: HDFS


When flume= starts , processing events happens properly and its moved to hdfs without = any issues.

But after sometime flume suddenly stops sending even= ts to HDFS.

=C2=A0

I am not seeing any errors in logfile flume.log as w= ell.Please let me know if I am missing any configuration here.


Below is the channel configur= ation defined and I left the remaining to be default values.


a1.channels.c1.type = =3D FILE

a1.channels.c1.transactionCapacity =3D 1= 00000

a1.channels.c1.capacity =3D 10000000


Thanks,
Saravana

=C2=A0


















<= /div>--
thanks
ashish

Blog: http://www.ash= ishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal




--
= thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal --047d7b86e8ba81f35904fe5def22--