flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SaravanaKumar TR <saran0081...@gmail.com>
Subject Re: Flume stops processing event after a while
Date Thu, 17 Jul 2014 06:54:37 GMT
thanks its really helpful.
I guess the default heap dump path is /tmp ?


On 17 July 2014 12:11, Ashish <paliwalashish@gmail.com> wrote:

> Nope, a heap dump shall be generated. Please see more options at
> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
>
> to specify path use this -XX:HeapDumpPath=./java_pid<pid>.hprof
>
>
> On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR <saran0081986@gmail.com
> > wrote:
>
>> yes , sorry I missed to update as 1 GB.
>>
>> But for out of memory error ,do we get notified in flume logs? I haven't
>> see any exception till now.
>>
>>
>> On 17 July 2014 11:55, SaravanaKumar TR <saran0081986@gmail.com> wrote:
>>
>>> Thanks Ashish , So I wil go ahead and update the flume-env,sh file with
>>>
>>> JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote
>>> -XX:-HeapDumpOnOutOfMemoryError"
>>>
>>>
>>> On 17 July 2014 11:39, Ashish <paliwalashish@gmail.com> wrote:
>>>
>>>> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if your process
>>>> is OOME, would generate a Heap dump. Allocate Heap based on the number of
>>>> events you need to keep in channel. Try with 1 GB, but calculate according
>>>> the Channel size as (average event size * number of events), plus object
>>>> over heads.
>>>>
>>>> Please note, this is just a rough calculation, actual memory usage
>>>> would be higher.
>>>>
>>>>
>>>> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR <
>>>> saran0081986@gmail.com> wrote:
>>>>
>>>>> Okay thanks , So for 128 GB , I will allocate 1 GB as a heap memory
>>>>> for flume agent.
>>>>>
>>>>> But I am surprised why there was no error registered for this memory
>>>>> issues in log file (flume.log).
>>>>>
>>>>> Do i need to check in any other logs?
>>>>>
>>>>>
>>>>> On 16 July 2014 21:55, Jonathan Natkins <natty@streamsets.com>
wrote:
>>>>>
>>>>>> That's definitely your problem. 20MB is way too low for this.
>>>>>> Depending on the other processes you're running with your system,
the
>>>>>> amount of memory you'll need will vary, but I'd recommend at least
1GB. You
>>>>>> should define it exactly where it's defined right now, so instead
of the
>>>>>> current command, you can run:
>>>>>>
>>>>>> "/cv/jvendor/bin/java -Xmx1g -Dflume.root.logger=DEBUG,LOGFILE......"
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR <
>>>>>> saran0081986@gmail.com> wrote:
>>>>>>
>>>>>>> I guess i am using defaulk values , from running flume i could
see
>>>>>>> these lines  "/cv/jvendor/bin/java -Xmx20m
>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......"
>>>>>>>
>>>>>>> so i guess it takes 20 mb as agent flume memory.
>>>>>>> My RAM is 128 GB.So please suggest how much can i assign as heap
>>>>>>> memory and where to define it.
>>>>>>>
>>>>>>>
>>>>>>> On 16 July 2014 15:05, Jonathan Natkins <natty@streamsets.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Saravana,
>>>>>>>>
>>>>>>>> I'm attempting to reproduce this, but do you happen to know
what
>>>>>>>> the Java heap size is for your Flume agent? This information
leads me to
>>>>>>>> believe that you don't have enough memory allocated to the
agent, which you
>>>>>>>> may need to do with the -Xmx parameter when you start up
your agent. That
>>>>>>>> aside, you can set the byteCapacity parameter on the memory
channel to
>>>>>>>> specify how much memory it is allowed to use. It should default
to 80% of
>>>>>>>> the Java heap size, but if your heap is too small, this might
be a cause of
>>>>>>>> errors.
>>>>>>>>
>>>>>>>> Does anything get written to the log when you try to pass
in an
>>>>>>>> event of this size?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Natty
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar TR <
>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Natty,
>>>>>>>>>
>>>>>>>>> While looking further , i could see memory channal stops
if a line
>>>>>>>>> comes with greater than 2 MB.Let me know which parameter
helps us to define
>>>>>>>>> max event size of about 3 MB.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 16 July 2014 12:46, SaravanaKumar TR <saran0081986@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I am asking point 1 , because in some cases  I could
see a line
>>>>>>>>>> in logfile around 2 MB.So i need to know what mamimum
event size.How to
>>>>>>>>>> measure it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 16 July 2014 10:18, SaravanaKumar TR <saran0081986@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Natty,
>>>>>>>>>>>
>>>>>>>>>>> Please help me to get the answers for the below
queries.
>>>>>>>>>>>
>>>>>>>>>>> 1,In case of exec source , (tail -F <logfile>)
, is that each
>>>>>>>>>>> line in file is considered to be a single event
?
>>>>>>>>>>> If suppose a line is considered to be a event
, what is that
>>>>>>>>>>> maximum size of event supported by flume?I mean
maximum characters in a
>>>>>>>>>>> line supported?
>>>>>>>>>>> 2.When event stop processing , I am not seeing
"tail -F" command
>>>>>>>>>>> running in the background.
>>>>>>>>>>> I have used option like "a1.sources.r1.restart
= true
>>>>>>>>>>> a1.sources.r1.logStdErr = true"..
>>>>>>>>>>> Does these config will not send any errors to
flume.log if any
>>>>>>>>>>> issues in tail?
>>>>>>>>>>> Will this config doesnt try to restart the "tail
-F" if its not
>>>>>>>>>>> running in the background.
>>>>>>>>>>>
>>>>>>>>>>> 3.Does flume supports all formats of data in
logfile or it has
>>>>>>>>>>> any predefined data formats..
>>>>>>>>>>>
>>>>>>>>>>> Please help me with these to understand better..
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 16 July 2014 00:56, Jonathan Natkins <natty@streamsets.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Saravana,
>>>>>>>>>>>>
>>>>>>>>>>>> Everything here looks pretty sane. Do you
have a record of the
>>>>>>>>>>>> events that came in leading up to the agent
stopping collection? If you can
>>>>>>>>>>>> provide the last file created by the agent,
and ideally whatever events had
>>>>>>>>>>>> come in, but not been written out to your
HDFS sink, it might be possible
>>>>>>>>>>>> for me to reproduce this issue. Would it
be possible to get some sample
>>>>>>>>>>>> data from you?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Natty
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Jul 15, 2014 at 10:26 AM, SaravanaKumar
TR <
>>>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Natty ,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Just to understand , at present my settings
is as
>>>>>>>>>>>>> "flume.root.logger=INFO,LOGFILE"
>>>>>>>>>>>>> in log4j.properties , do you want me
to change it to
>>>>>>>>>>>>> "flume.root.logger=DEBUG,LOGFILE" and
restart the agent.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But when I start agent , I am already
starting with below
>>>>>>>>>>>>> command.I guess i am using DEBUG already
but not in config file , while
>>>>>>>>>>>>> starting agent.
>>>>>>>>>>>>>
>>>>>>>>>>>>> ../bin/flume-ng agent -c /d0/flume/conf
-f
>>>>>>>>>>>>> /d0/flume/conf/flume-conf.properties
-n a1 -Dflume.root.logger=DEBUG,LOGFILE
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I do some changes in config "flume-conf.properties"
or
>>>>>>>>>>>>> restart the agent , it works again and
starts collecting the data.
>>>>>>>>>>>>>
>>>>>>>>>>>>> currently all my logs move to flume.log
, I dont see any
>>>>>>>>>>>>> exception .
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat flume.log | grep "Exception"  doesnt
show any.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 15 July 2014 22:24, Jonathan Natkins
<natty@streamsets.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Saravana,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Our best bet on figuring out what's
going on here may be to
>>>>>>>>>>>>>> turn on the debug logging. What I
would recommend is stopping your agents,
>>>>>>>>>>>>>> and modifying the log4j properties
to turn on DEBUG logging for the root
>>>>>>>>>>>>>> logger, and then restart the agents.
Once the agent stops producing new
>>>>>>>>>>>>>> events, send out the logs and I'll
be happy to take a look over them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does the system begin working again
if you restart the
>>>>>>>>>>>>>> agents? Have you noticed any other
events correlated with the agent
>>>>>>>>>>>>>> stopping collecting events? Maybe
a spike in events or something like that?
>>>>>>>>>>>>>> And for my own peace of mind, if
you run `cat /var/log/flume-ng/* | grep
>>>>>>>>>>>>>> "Exception"`, does it bring anything
back?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>> Natty
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 2:55 AM,
SaravanaKumar TR <
>>>>>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Natty,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is my entire config file.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # Name the components on this
agent
>>>>>>>>>>>>>>> a1.sources = r1
>>>>>>>>>>>>>>> a1.sinks = k1
>>>>>>>>>>>>>>> a1.channels = c1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # Describe/configure the source
>>>>>>>>>>>>>>> a1.sources.r1.type = exec
>>>>>>>>>>>>>>> a1.sources.r1.command = tail
-F /data/logs/test_log
>>>>>>>>>>>>>>> a1.sources.r1.restart = true
>>>>>>>>>>>>>>> a1.sources.r1.logStdErr = true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #a1.sources.r1.batchSize = 2
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a1.sources.r1.interceptors =
i1
>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type
= regex_filter
>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex
= resuming normal
>>>>>>>>>>>>>>> operations|Received|Response
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #a1.sources.r1.interceptors =
i2
>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type
= timestamp
>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting
= true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # Describe the sink
>>>>>>>>>>>>>>> a1.sinks.k1.type = hdfs
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.path = hdfs://
>>>>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=%Y-%m-%d
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat
= Text
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType = DataStream
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix =
events-
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval
= 600
>>>>>>>>>>>>>>> ##need to run hive query randomly
to check teh long running
>>>>>>>>>>>>>>> process , so we  need to commit
events in hdfs files regularly
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount =
0
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize =
10
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize = 0
>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp
= true
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # Use a channel which buffers
events in memory
>>>>>>>>>>>>>>> a1.channels.c1.type = memory
>>>>>>>>>>>>>>> a1.channels.c1.capacity = 10000
>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity
= 10000
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # Bind the source and sink to
the channel
>>>>>>>>>>>>>>> a1.sources.r1.channels = c1
>>>>>>>>>>>>>>> a1.sinks.k1.channel = c1
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 14 July 2014 22:54, Jonathan
Natkins <
>>>>>>>>>>>>>>> natty@streamsets.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Saravana,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> What does your sink configuration
look like?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Natty
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 11, 2014 at 11:05
PM, SaravanaKumar TR <
>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Assuming each line in
the logfile is considered as a event
>>>>>>>>>>>>>>>>> for flume ,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1.Do we have any maximum
size of event defined for
>>>>>>>>>>>>>>>>> memory/file channel.like
any maximum no of characters in a line.
>>>>>>>>>>>>>>>>> 2.Does flume supports
all formats of data to be processed
>>>>>>>>>>>>>>>>> as events or do we have
any limitation.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am just still trying
to understanding why the flume
>>>>>>>>>>>>>>>>> stops processing events
after sometime.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can someone please help
me out here.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> saravana
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 11 July 2014 17:49,
SaravanaKumar TR <
>>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi ,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am new to flume
and  using Apache Flume 1.5.0. Quick
>>>>>>>>>>>>>>>>>> setup explanation
here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Source:exec , tail
–F command for a logfile.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Channel: tried with
both Memory & file channel
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Sink: HDFS
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> When flume starts
, processing events happens properly
>>>>>>>>>>>>>>>>>> and its moved to
hdfs without any issues.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But after sometime
flume suddenly stops sending events to
>>>>>>>>>>>>>>>>>> HDFS.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am not seeing any
errors in logfile flume.log as
>>>>>>>>>>>>>>>>>> well.Please let me
know if I am missing any configuration here.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Below is the channel
configuration defined and I left the
>>>>>>>>>>>>>>>>>> remaining to be default
values.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> a1.channels.c1.type
= FILE
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity
= 100000
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity
= 10000000
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Saravana
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> thanks
>>>> ashish
>>>>
>>>> Blog: http://www.ashishpaliwal.com/blog
>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>
>>>
>>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Mime
View raw message