flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周梦想 <abloz...@gmail.com>
Subject Re: how to know which line the agent have sent ?
Date Thu, 07 Feb 2013 03:03:28 GMT
I see,there is no easy way or configure way to know the detail of what has
sent and what haven't.
I have to write my own source or sink code to do this.
Thank you,Alex and all friends.

Andy

2013/2/6 Alexander Alten-Lorenz <wget.null@gmail.com>

> You haven't a control in such situations, since tailDir uses tail and
> holds the marker in memory.
>
> We had few days ago a thread about:
>
> http://search-hadoop.com/m/JV0lh2RDXLX/flume+tail+source+problem+and+performance&subj=flume+tail+source+problem+and+performance
>
> - Alex
>
> On Feb 6, 2013, at 3:45 AM, 周梦想 <ablozhou@gmail.com> wrote:
>
> > Hello,
> >
> > I'm using tailDirs('mydir') source of the agent to gather logs to hadoop
> > hdfs. I notice some documents advise that if the agent collapsed,  I have
> > to remove files in 'mydir' and clear flume.agent.logdir. Thus I will lose
> > some data or have duplicate data. And I don't know which line the agent
> > have sent to.
> >
> > I'm worrying about the agent failure and resend or miss-send the content
> to
> > collector. I want to know how to check which line of log file the agent
> > have sent if the agent exit suddenly. The files in flute log dir, such as
> > sending,sent can't be read.
> >
> > Please give some advise to process such situation.
> > Thanks.
> >
> > Andy Zhou
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>

Mime
View raw message