Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of david.cochran@bsee.gov
 designates 137.227.28.13 as permitted sender)
MIME-Version: 1.0
Date: Fri, 21 Dec 2012 14:26:50 -0600
Message-ID: 
 <CAOFxjiCAMyNKcSW-GnkgF=d9Ghsd6Fo=QpaKNsCfbv20ZsF2cA@mail.gmail.com>
Subject: post-processing
From: "Cochran, David" <david.cochran@bsee.gov>
To: <user@flume.apache.org>
Content-Type: multipart/alternative; boundary="e89a8fb1ee7ae5754b04d162aa42"
Received-SPF: Neutral (gSMTP6.doi.gov: 209.85.214.198 is neither permitted
 nor denied by domain of david.cochran@bsee.gov)

--e89a8fb1ee7ae5754b04d162aa42
Content-Type: text/plain; charset="ISO-8859-1"

just had a thought... before I turn this script up and make a mess of
things I figured I'd ask the group...

I'm running FLUME 1.3 running using FILE_ROLL at the sink.... the 'live in
use' files are being periodically scanned for key events while still "live'
and being appending to by Flume... no problems there as they are just being
read....

now the interesting part, I also need to do a little processing of the
stored logs (using sed) to insert a couple pieces of data into each line
(if it doesn't already exist) before my log scanner process does it's thing.

I'm not sure what the odds are of this NOT totally hosing the flume
process/data will be...maybe recognizes the file is in use and waits? The
files are processed by sed pretty quickly ( ~15 secs) as they are rotated
daily.

Has anyone else tried this yet or have any insight as to how Flume might
react before I attempt to make bit soup?

Thanks,
-Dave

--e89a8fb1ee7ae5754b04d162aa42
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

just had a thought... before I turn this script up and make a mess of thing=
s I figured I&#39;d ask the group...<div><br></div><div>I&#39;m running FLU=
ME 1.3 running using FILE_ROLL at the sink.... the &#39;live in use&#39; fi=
les are being periodically scanned for key events while still &quot;live=
9; and being appending to by Flume... no problems there as they are just be=
ing read....=A0</div>
<div><br></div><div>now the interesting part, I also need to do a little pr=
ocessing of the stored logs (using sed) to insert a couple pieces of data i=
nto each line (if it doesn&#39;t already exist) before my log scanner proce=
ss does it&#39;s thing.</div>
<div><br></div><div>I&#39;m not sure what the odds are of this NOT totally =
hosing the flume process/data will be...maybe recognizes the file is in use=
 and waits? The files are processed by sed pretty quickly ( ~15 secs) as th=
ey are rotated daily.</div>
<div><br></div><div>Has anyone else tried this yet or have any insight as t=
o how Flume might react before I attempt to make bit soup?</div><div><br></=
div><div>Thanks,</div><div>-Dave</div>

--e89a8fb1ee7ae5754b04d162aa42--