Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7652A97A2 for ; Wed, 18 Jul 2012 06:43:59 +0000 (UTC) Received: (qmail 55237 invoked by uid 500); 18 Jul 2012 06:43:59 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 54950 invoked by uid 500); 18 Jul 2012 06:43:59 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 54916 invoked by uid 99); 18 Jul 2012 06:43:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 06:43:57 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of shekhar2581@gmail.com designates 209.85.160.51 as permitted sender) Received: from [209.85.160.51] (HELO mail-pb0-f51.google.com) (209.85.160.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Jul 2012 06:43:51 +0000 Received: by pbbrp16 with SMTP id rp16so2105332pbb.38 for ; Tue, 17 Jul 2012 23:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=kP8+wY4RoX0UbCC+LnwpCqYKdakLoIXF6HuG2Uxs+Oo=; b=TQTHGHw9Ifx5/vvjS7+/yfb6wOvLXy8GI+KJxs6SLfwb16WVLpJR6LpsX5t8T09W8m yjwJ4f62A4BYcswYKoECtsCHhXBgavHs77T1rzZ15Fn3vBNxeFtpeiXUCV67lsAiFjlv IuGi69s/g1WgP9ttgxdSyuhXUfQeVmAOiQeGm8O1foBeCVys6oV4kCCdC6ug7aDuqZ4o uongEiQoFoZrmo6NO605z4acabTWSzxlT2ZnGZTQCp1H78J//hZ4Iz2I4e+NPaVwTsbs vQT5orc6Pakm7d1ocSWXJtprkGI8/5lEVXJe6UMkyH7jOhMV16ydZSCoCeb0z7B9kq2k HDRw== MIME-Version: 1.0 Received: by 10.68.224.36 with SMTP id qz4mr4453566pbc.161.1342593809527; Tue, 17 Jul 2012 23:43:29 -0700 (PDT) Received: by 10.66.13.1 with HTTP; Tue, 17 Jul 2012 23:43:29 -0700 (PDT) In-Reply-To: <1C4773FD-B111-4C9B-815B-CC04D9F5D383@gmail.com> References: <50064F62.9030504@pubmatic.com> <1C4773FD-B111-4C9B-815B-CC04D9F5D383@gmail.com> Date: Wed, 18 Jul 2012 12:13:29 +0530 Message-ID: Subject: Re: flume-ng failure recovery From: shekhar sharma To: user@flume.apache.org Content-Type: multipart/alternative; boundary=047d7b1605c918326504c514fb28 --047d7b1605c918326504c514fb28 Content-Type: text/plain; charset=ISO-8859-1 Dont use tail source since it does not maintain the state where it left..so i would suggest something like this:: (1)Implement a Reader part which reads the events from the file and also maintains the state if something goes wrong) (2) convert the events to the Flume Event type and using a RPC client send the events to the FLume Avro Source Regards, SOm On Wed, Jul 18, 2012 at 11:39 AM, Justin Workman wrote: > We use a tail -F -n0. This will result in the tail command starting at the > beginning of the file and replaying all events. > > This will however result in duplicate events that you will need to deal > with. > > Sent from my iPhone > > On Jul 17, 2012, at 11:53 PM, Jagadish Bihani < > jagadish.bihani@pubmatic.com> wrote: > > > Hi > > > > We want to deploy flume-ng in the production environment in our > organization. > > Here is the following scenario for which I am not able to find the > answer: > > > > 1. We receive logs using 'tail -f' source. > > 2. Now the agent process gets killed. > > 3. We restart it. > > 4. How would the restarted agent will know the correct state of the file. > > Because in the meantime log file would have been modified and agent has > > no way of knowing from where to resume? > > > > Could you please help me in identifying how to tackle this scenario? > > > > P.S. Instead of tail -f any other command can be used which doesnt > modify the log file. > > > > Regards, > > Jagadish > --047d7b1605c918326504c514fb28 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Dont use tail source since it does not maintain the state where it lef= t..so i would suggest something like this::
=A0
(1)Implement a Reader part which reads the events from the file and al= so maintains the state if something=A0 goes=A0 wrong)
(2)=A0convert the events to the Flume Event type and using a RPC clien= t send the events to the FLume Avro Source
=A0
Regards,
SOm

On Wed, Jul 18, 2012 at 11:39 AM, Justin Workman= <justinjworkman@gmail.com> wrote:
We use a tail -F -n0. This will resul= t in the tail command starting at the beginning of the file and replaying a= ll events.

This will however result in duplicate events that you will need to deal= with.

Sent from my iPhone

On Jul 17, 2012, at 11:53 PM, Jagadish Bihani <jagadish.bihani@pubmatic.com> wrote:

> Hi
>
> We want to deploy flume-ng in = the production environment in our organization.
> Here is the following scenario for which I am not able to find the ans= wer:
>
> 1. We receive logs using 'tail -f' source.
= > 2. Now =A0the agent process gets killed.
> 3. We restart it.
> 4. How would the restarted agent will know the correct state of the fi= le.
> Because in the meantime log file would have been modified and a= gent has
> no way of knowing from where to resume?
>
> Co= uld you please help me in identifying how to tackle this scenario?
>
> P.S. Instead of tail -f any other command can be used which do= esnt modify the log file.
>
> Regards,
> Jagadish

--047d7b1605c918326504c514fb28--