flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From SaravanaKumar TR <saran0081...@gmail.com>
Subject Re: Flume stops processing event after a while
Date Wed, 30 Jul 2014 16:06:45 GMT
It looks like JvisualVM is a GUI tool.

can you please suggest me some CLI tool to get more info about JVM process.
It looks JVM was hanged because jstack <pid> doesn't show any ..only -F
option could get some output.


On Wed, Jul 30, 2014 at 7:45 PM, Ashish <paliwalashish@gmail.com> wrote:

> Watch for GC activity (either enable GC logging or use visualvm)
> VisualVM would give complete idea of what's happening inside the running
> JVM
>
>
> On Tue, Jul 29, 2014 at 11:53 AM, SaravanaKumar TR <saran0081986@gmail.com
> > wrote:
>
>> Out of curiosity to find the exact issue , i started flume with 20 MB
>> memory (default) with the option as /cv/jvendor/bin/java -Xms20m -Xmx20m
>> -Dcom.sun.management.jmxremote -XX:-HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=./java_pid<pid>.hprof
>>
>> Again memory channel stops processing the event after a while , but heap
>> dump is  not generated here.Is the arguments i provided is  corrrect ?
>>
>> Tried getting the jstack of running flume agent and seems the process got
>> hung."tail -F <logfile> process is no more running.
>>
>> From jstack -F <pid> ,
>>
>> Deadlock Detection:
>>
>> No deadlocks found.
>>
>> Thread 14788: (state = BLOCKED)
>>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame;
>> information may be imprecise)
>>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
>> line=186 (Compiled frame)
>>  -
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
>> @bci=42, line=2043 (Compiled frame)
>>  -
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
>> @bci=98, line=1085 (Compiled frame)
>>  -
>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take()
>> @bci=1, line=807 (Interpreted frame)
>>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068
>> (Interpreted frame)
>>  -
>> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
>> @bci=26, line=1130 (Interpreted frame)
>>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615
>> (Interpreted frame)
>>  - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
>>
>>
>> Thread 14785: (state = IN_NATIVE)
>>  - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0
>> (Interpreted frame)
>>  - java.net.AbstractPlainSocketImpl.accept(java.net.SocketImpl) @bci=7,
>> line=398 (Interpreted frame)
>>  - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=530
>> (Interpreted frame)
>>  - java.net.ServerSocket.accept() @bci=48, line=498 (Interpreted frame)
>>  - sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept()
>> @bci=1, line=52 (Interpreted frame)
>>  - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop()
>> @bci=55, line=388 (Interpreted frame)
>>  - sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run() @bci=1, line=360
>> (Interpreted frame)
>>  - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
>>
>>
>>
>> On Thu, Jul 17, 2014 at 8:21 PM, SaravanaKumar TR <saran0081986@gmail.com
>> > wrote:
>>
>>> Thanks Natty & Ashish.
>>>
>>> I have restarted flume agent with below config.Will monitor it for
>>> couple of days whether it stops randomly.
>>>
>>> JAVA_OPTS="-Xms1g -Xmx1g -Dcom.sun.management.jmxremote
>>> -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./java_pid<pid>.hprof"
>>>
>>> thanks for all again.Hope this will work well.
>>>
>>>
>>> On 17 July 2014 12:24, SaravanaKumar TR <saran0081986@gmail.com> wrote:
>>>
>>>> thanks its really helpful.
>>>> I guess the default heap dump path is /tmp ?
>>>>
>>>>
>>>> On 17 July 2014 12:11, Ashish <paliwalashish@gmail.com> wrote:
>>>>
>>>>> Nope, a heap dump shall be generated. Please see more options at
>>>>> http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
>>>>>
>>>>> to specify path use this -XX:HeapDumpPath=./java_pid<pid>.hprof
>>>>>
>>>>>
>>>>> On Thu, Jul 17, 2014 at 12:09 PM, SaravanaKumar TR <
>>>>> saran0081986@gmail.com> wrote:
>>>>>
>>>>>> yes , sorry I missed to update as 1 GB.
>>>>>>
>>>>>> But for out of memory error ,do we get notified in flume logs? I
>>>>>> haven't see any exception till now.
>>>>>>
>>>>>>
>>>>>> On 17 July 2014 11:55, SaravanaKumar TR <saran0081986@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Ashish , So I wil go ahead and update the flume-env,sh
file
>>>>>>> with
>>>>>>>
>>>>>>> JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote
>>>>>>> -XX:-HeapDumpOnOutOfMemoryError"
>>>>>>>
>>>>>>>
>>>>>>> On 17 July 2014 11:39, Ashish <paliwalashish@gmail.com>
wrote:
>>>>>>>
>>>>>>>> Add -XX:-HeapDumpOnOutOfMemoryError parameter as well, if
your
>>>>>>>> process is OOME, would generate a Heap dump. Allocate Heap
based on the
>>>>>>>> number of events you need to keep in channel. Try with 1
GB, but calculate
>>>>>>>> according the Channel size as (average event size * number
of events), plus
>>>>>>>> object over heads.
>>>>>>>>
>>>>>>>> Please note, this is just a rough calculation, actual memory
usage
>>>>>>>> would be higher.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 17, 2014 at 11:21 AM, SaravanaKumar TR <
>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Okay thanks , So for 128 GB , I will allocate 1 GB as
a heap
>>>>>>>>> memory for flume agent.
>>>>>>>>>
>>>>>>>>> But I am surprised why there was no error registered
for this
>>>>>>>>> memory issues in log file (flume.log).
>>>>>>>>>
>>>>>>>>> Do i need to check in any other logs?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 16 July 2014 21:55, Jonathan Natkins <natty@streamsets.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> That's definitely your problem. 20MB is way too low
for this.
>>>>>>>>>> Depending on the other processes you're running with
your system, the
>>>>>>>>>> amount of memory you'll need will vary, but I'd recommend
at least 1GB. You
>>>>>>>>>> should define it exactly where it's defined right
now, so instead of the
>>>>>>>>>> current command, you can run:
>>>>>>>>>>
>>>>>>>>>> "/cv/jvendor/bin/java -Xmx1g
>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......"
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 16, 2014 at 3:03 AM, SaravanaKumar TR
<
>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I guess i am using defaulk values , from running
flume i could
>>>>>>>>>>> see these lines  "/cv/jvendor/bin/java -Xmx20m
>>>>>>>>>>> -Dflume.root.logger=DEBUG,LOGFILE......"
>>>>>>>>>>>
>>>>>>>>>>> so i guess it takes 20 mb as agent flume memory.
>>>>>>>>>>> My RAM is 128 GB.So please suggest how much can
i assign as heap
>>>>>>>>>>> memory and where to define it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 16 July 2014 15:05, Jonathan Natkins <natty@streamsets.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey Saravana,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm attempting to reproduce this, but do
you happen to know
>>>>>>>>>>>> what the Java heap size is for your Flume
agent? This information leads me
>>>>>>>>>>>> to believe that you don't have enough memory
allocated to the agent, which
>>>>>>>>>>>> you may need to do with the -Xmx parameter
when you start up your agent.
>>>>>>>>>>>> That aside, you can set the byteCapacity
parameter on the memory channel to
>>>>>>>>>>>> specify how much memory it is allowed to
use. It should default to 80% of
>>>>>>>>>>>> the Java heap size, but if your heap is too
small, this might be a cause of
>>>>>>>>>>>> errors.
>>>>>>>>>>>>
>>>>>>>>>>>> Does anything get written to the log when
you try to pass in an
>>>>>>>>>>>> event of this size?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Natty
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 16, 2014 at 1:46 AM, SaravanaKumar
TR <
>>>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Natty,
>>>>>>>>>>>>>
>>>>>>>>>>>>> While looking further , i could see memory
channal stops if a
>>>>>>>>>>>>> line comes with greater than 2 MB.Let
me know which parameter helps us to
>>>>>>>>>>>>> define max event size of about 3 MB.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 16 July 2014 12:46, SaravanaKumar
TR <
>>>>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am asking point 1 , because in
some cases  I could see a
>>>>>>>>>>>>>> line in logfile around 2 MB.So i
need to know what mamimum event size.How
>>>>>>>>>>>>>> to measure it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 16 July 2014 10:18, SaravanaKumar
TR <
>>>>>>>>>>>>>> saran0081986@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Natty,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please help me to get the answers
for the below queries.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1,In case of exec source , (tail
-F <logfile>) , is that
>>>>>>>>>>>>>>> each line in file is considered
to be a single event ?
>>>>>>>>>>>>>>> If suppose a line is considered
to be a event , what is that
>>>>>>>>>>>>>>> maximum size of event supported
by flume?I mean maximum characters in a
>>>>>>>>>>>>>>> line supported?
>>>>>>>>>>>>>>> 2.When event stop processing
, I am not seeing "tail -F"
>>>>>>>>>>>>>>> command running in the background.
>>>>>>>>>>>>>>> I have used option like "a1.sources.r1.restart
= true
>>>>>>>>>>>>>>> a1.sources.r1.logStdErr = true"..
>>>>>>>>>>>>>>> Does these config will not send
any errors to flume.log if
>>>>>>>>>>>>>>> any issues in tail?
>>>>>>>>>>>>>>> Will this config doesnt try to
restart the "tail -F" if its
>>>>>>>>>>>>>>> not running in the background.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 3.Does flume supports all formats
of data in logfile or it
>>>>>>>>>>>>>>> has any predefined data formats..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please help me with these to
understand better..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 16 July 2014 00:56, Jonathan
Natkins <
>>>>>>>>>>>>>>> natty@streamsets.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Saravana,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Everything here looks pretty
sane. Do you have a record of
>>>>>>>>>>>>>>>> the events that came in leading
up to the agent stopping collection? If you
>>>>>>>>>>>>>>>> can provide the last file
created by the agent, and ideally whatever events
>>>>>>>>>>>>>>>> had come in, but not been
written out to your HDFS sink, it might be
>>>>>>>>>>>>>>>> possible for me to reproduce
this issue. Would it be possible to get some
>>>>>>>>>>>>>>>> sample data from you?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Natty
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Jul 15, 2014 at 10:26
AM, SaravanaKumar TR <
>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Natty ,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Just to understand ,
at present my settings is as
>>>>>>>>>>>>>>>>> "flume.root.logger=INFO,LOGFILE"
>>>>>>>>>>>>>>>>> in log4j.properties ,
do you want me to change it to
>>>>>>>>>>>>>>>>> "flume.root.logger=DEBUG,LOGFILE"
and restart the agent.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But when I start agent
, I am already starting with below
>>>>>>>>>>>>>>>>> command.I guess i am
using DEBUG already but not in config file , while
>>>>>>>>>>>>>>>>> starting agent.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ../bin/flume-ng agent
-c /d0/flume/conf -f
>>>>>>>>>>>>>>>>> /d0/flume/conf/flume-conf.properties
-n a1 -Dflume.root.logger=DEBUG,LOGFILE
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If I do some changes
in config "flume-conf.properties" or
>>>>>>>>>>>>>>>>> restart the agent , it
works again and starts collecting the data.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> currently all my logs
move to flume.log , I dont see any
>>>>>>>>>>>>>>>>> exception .
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cat flume.log | grep
"Exception"  doesnt show any.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 15 July 2014 22:24,
Jonathan Natkins <
>>>>>>>>>>>>>>>>> natty@streamsets.com>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Saravana,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Our best bet on figuring
out what's going on here may be
>>>>>>>>>>>>>>>>>> to turn on the debug
logging. What I would recommend is stopping your
>>>>>>>>>>>>>>>>>> agents, and modifying
the log4j properties to turn on DEBUG logging for the
>>>>>>>>>>>>>>>>>> root logger, and
then restart the agents. Once the agent stops producing
>>>>>>>>>>>>>>>>>> new events, send
out the logs and I'll be happy to take a look over them.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Does the system begin
working again if you restart the
>>>>>>>>>>>>>>>>>> agents? Have you
noticed any other events correlated with the agent
>>>>>>>>>>>>>>>>>> stopping collecting
events? Maybe a spike in events or something like that?
>>>>>>>>>>>>>>>>>> And for my own peace
of mind, if you run `cat /var/log/flume-ng/* | grep
>>>>>>>>>>>>>>>>>> "Exception"`, does
it bring anything back?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>> Natty
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Jul 15, 2014
at 2:55 AM, SaravanaKumar TR <
>>>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Natty,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is my entire
config file.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # Name the components
on this agent
>>>>>>>>>>>>>>>>>>> a1.sources =
r1
>>>>>>>>>>>>>>>>>>> a1.sinks = k1
>>>>>>>>>>>>>>>>>>> a1.channels =
c1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # Describe/configure
the source
>>>>>>>>>>>>>>>>>>> a1.sources.r1.type
= exec
>>>>>>>>>>>>>>>>>>> a1.sources.r1.command
= tail -F /data/logs/test_log
>>>>>>>>>>>>>>>>>>> a1.sources.r1.restart
= true
>>>>>>>>>>>>>>>>>>> a1.sources.r1.logStdErr
= true
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> #a1.sources.r1.batchSize
= 2
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors
= i1
>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.type
= regex_filter
>>>>>>>>>>>>>>>>>>> a1.sources.r1.interceptors.i1.regex
= resuming normal
>>>>>>>>>>>>>>>>>>> operations|Received|Response
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors
= i2
>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.type
= timestamp
>>>>>>>>>>>>>>>>>>> #a1.sources.r1.interceptors.i2.preserveExisting
= true
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # Describe the
sink
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.type
= hdfs
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.path
= hdfs://
>>>>>>>>>>>>>>>>>>> testing.sck.com:9000/running/test.sck/date=%Y-%m-%d
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.writeFormat
= Text
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.fileType
= DataStream
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.filePrefix
= events-
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollInterval
= 600
>>>>>>>>>>>>>>>>>>> ##need to run
hive query randomly to check teh long
>>>>>>>>>>>>>>>>>>> running process
, so we  need to commit events in hdfs files regularly
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollCount
= 0
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.batchSize
= 10
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.rollSize
= 0
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.hdfs.useLocalTimeStamp
= true
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # Use a channel
which buffers events in memory
>>>>>>>>>>>>>>>>>>> a1.channels.c1.type
= memory
>>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity
= 10000
>>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity
= 10000
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> # Bind the source
and sink to the channel
>>>>>>>>>>>>>>>>>>> a1.sources.r1.channels
= c1
>>>>>>>>>>>>>>>>>>> a1.sinks.k1.channel
= c1
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 14 July 2014
22:54, Jonathan Natkins <
>>>>>>>>>>>>>>>>>>> natty@streamsets.com>
wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Saravana,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> What does
your sink configuration look like?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Natty
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Jul
11, 2014 at 11:05 PM, SaravanaKumar TR <
>>>>>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Assuming
each line in the logfile is considered as a
>>>>>>>>>>>>>>>>>>>>> event
for flume ,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 1.Do
we have any maximum size of event defined for
>>>>>>>>>>>>>>>>>>>>> memory/file
channel.like any maximum no of characters in a line.
>>>>>>>>>>>>>>>>>>>>> 2.Does
flume supports all formats of data to be
>>>>>>>>>>>>>>>>>>>>> processed
as events or do we have any limitation.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am
just still trying to understanding why the flume
>>>>>>>>>>>>>>>>>>>>> stops
processing events after sometime.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Can someone
please help me out here.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> saravana
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 11
July 2014 17:49, SaravanaKumar TR <
>>>>>>>>>>>>>>>>>>>>> saran0081986@gmail.com>
wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi
,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I
am new to flume and  using Apache Flume 1.5.0.
>>>>>>>>>>>>>>>>>>>>>> Quick
setup explanation here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Source:exec
, tail –F command for a logfile.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Channel:
tried with both Memory & file channel
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Sink:
HDFS
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> When
flume starts , processing events happens
>>>>>>>>>>>>>>>>>>>>>> properly
and its moved to hdfs without any issues.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> But
after sometime flume suddenly stops sending
>>>>>>>>>>>>>>>>>>>>>> events
to HDFS.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I
am not seeing any errors in logfile flume.log as
>>>>>>>>>>>>>>>>>>>>>> well.Please
let me know if I am missing any configuration here.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Below
is the channel configuration defined and I left
>>>>>>>>>>>>>>>>>>>>>> the
remaining to be default values.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.type
= FILE
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.transactionCapacity
= 100000
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a1.channels.c1.capacity
= 10000000
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>> Saravana
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> thanks
>>>>>>>> ashish
>>>>>>>>
>>>>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> thanks
>>>>> ashish
>>>>>
>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Mime
View raw message