Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 05F1EE864 for ; Fri, 22 Feb 2013 22:32:32 +0000 (UTC) Received: (qmail 25454 invoked by uid 500); 22 Feb 2013 22:32:31 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 25388 invoked by uid 500); 22 Feb 2013 22:32:31 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 25377 invoked by uid 99); 22 Feb 2013 22:32:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Feb 2013 22:32:31 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ynerella999@gmail.com designates 209.85.220.44 as permitted sender) Received: from [209.85.220.44] (HELO mail-pa0-f44.google.com) (209.85.220.44) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Feb 2013 22:32:23 +0000 Received: by mail-pa0-f44.google.com with SMTP id kp1so710269pab.17 for ; Fri, 22 Feb 2013 14:32:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=MUWLwRvGy8VgZenVmyRFCDiv+CBjZO/CMK4YlCBm3xI=; b=Qm5p4h7T9q+rNbdVGzNm9eACmvCQ4LxSyLZhjJMS8yZF1CYLs8hq0O2XWOayXqsA/6 BcjEA0RBZbnbWvWD2YMc8M4VZk19sMjo1cetZFCEIbIBG9uw6rSHB1QY9HLO2mQ8Hwnh wQNCvRX7+UFo6AbiFdTVNuZH+PjB+Ng9D6cOaSGLKLuvE/E+D3bCpozPtx1QdMm5PDJO s1wo7GzeKvyCGzBiS0/1Bt2fUon2nTva322dhIWa3xeEMkt1KEKR8XNJ4f5UUb8uSNHR XTv4Ae/rJn7hBs/ylDgMs1wkx2G9kO3PI1G4xQ8rr5hbhtENAqv2NG+DAdZFx7imE9Ej Whxg== MIME-Version: 1.0 X-Received: by 10.66.226.71 with SMTP id rq7mr6415138pac.80.1361572322166; Fri, 22 Feb 2013 14:32:02 -0800 (PST) Received: by 10.68.229.134 with HTTP; Fri, 22 Feb 2013 14:32:02 -0800 (PST) In-Reply-To: References: <5126EF4B.6050309@cyberagent.co.jp> <512727FE.7060001@cyberagent.co.jp> Date: Fri, 22 Feb 2013 14:32:02 -0800 Message-ID: Subject: Re: why flume agent do not support --fileheader From: Yogi Nerella To: user@flume.apache.org Content-Type: multipart/alternative; boundary=047d7b15a71b994d5f04d657c271 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b15a71b994d5f04d657c271 Content-Type: text/plain; charset=GB2312 Content-Transfer-Encoding: quoted-printable Hi, I want to ship various types of files from one agent to another agent, and a destination agent should rewrite them to the same filename as the source filename. Like a simple FTP source to FTP destination, but I need it via flume because I want to do some intercepting and processing? Is there any simple configuration for it? Or Flume cannot support such functionality? Thanks, Yogi On Fri, Feb 22, 2013 at 12:36 AM, =D6=DC=C3=CE=CF=EB w= rote: > ok,I see. > > > 2013/2/22 Juhani Connolly > >> Sounds like a job for an editor macro or a simple script. >> >> Perhaps someone else can think of something else, but adding it to the >> command line isn't feasible... If we started throwing random convenient >> stuff there it would quickly become a mess >> >> >> On 02/22/2013 01:30 PM, =D6=DC=C3=CE=CF=EB wrote: >> >> Yes=A3=ACI mean all event headers send from a specific agent. some heade= rs >> need for all events, not only hostname. now I have to configure it on ev= ery >> source time and time again. >> for example: >> if i configure 10 sources, every source need 5 header, I have to >> configure 50 times. >> other configure of the same type source have the same problem. >> >> eg: >> agent1.sources =3Dgamelog src1 src2 ... src10 >> >> agent1.sources.gamelog.fileSuffix=3D.fin >> agent1.sources.gamelog.fileHeader=3Dtrue >> agent1.sources.gamelog.fileHeaderKey=3Dfullfilename >> agent1.sources.gamelog.batchSize=3D100 >> agent1.sources.gamelog.bufferMaxLines=3D1000 >> agent1.sources.gamelog.bufferMaxLineLength=3D5000 >> agent1.sources.gamelog.interceptors =3D i1 i2 i3 >> #for %{host} org.apache.flume.interceptor.HostInterceptor$Builder >> agent1.sources.gamelog.interceptors.i1.type =3D host >> agent1.sources.gamelog.interceptors.i2.type =3D timestamp >> ... >> agent1.sources.gamelog.interceptors.i3.type =3D static >> agent1.sources.gamelog.interceptors.i3.key =3D filename >> agent1.sources.gamelog.interceptors.i3.value =3D gamelog >> >> *repleat configure for src1,...src10 for 10 or even more times.* >> >> it's very boring. >> maybe we could group the sources,sinks and configure them one or several >> times. >> >> Best Regards, >> Andy >> >> >> 2013/2/22 Juhani Connolly >> >>> If I understand your question, using a static interceptor is the >>> expected way to do this. >>> >>> Not sure what you mean by agent header? Do you mean all event headers >>> sent from a specific agent? I don't imagine we will be adding a >>> command-line parameter to do this, it wouldn't be consistent and would = be >>> superfluous. >>> >>> If you want to reuse a configuration file, and add a header to inform >>> what agent it came from, perhaps you could use the hostname interceptor= ? >>> >>> >>> On 02/22/2013 12:06 PM, =D6=DC=C3=CE=CF=EB wrote: >>> >>>> I want add some key/value to the agent header, but it's not convenient >>>> to do so. why not flume agent support --headerFile just like flume >>>> avro-client? >>>> >>>> my requirement is : >>>> I want use spooling source to send files to another flume node to writ= e >>>> to hdfs. >>>> and i want to name hdfs file to have the original file name prefix, no= t >>>> the full path of the orgin file. >>>> >>>> now I have to add interceptors of conf to do that. >>>> >>>> so I have the question, why not flume agent support --headerFile just >>>> like flume avro-client? >>>> >>>> Best Regards, >>>> Andy >>>> >>> >>> >> >> > --047d7b15a71b994d5f04d657c271 Content-Type: text/html; charset=GB2312 Content-Transfer-Encoding: quoted-printable
Hi,

I want to ship various types = of files from one agent to another agent, and a destination agent should re= write them to the same filename as the source filename. 

Like a simple FTP source to FTP destination, but I need it= via flume because I want to do some intercepting and processing?   Is= there any simple configuration for it?   Or Flume cannot support such= functionality?

Thanks,
Yogi


On Fri, Feb 22, = 2013 at 12:36 AM, =D6=DC=C3=CE=CF=EB <ablozhou@gmail.com> w= rote:
ok,I see. 


2013/2/22 Juhani Connolly <juhani_connolly@cyberagent.co.jp>
=20 =20 =20
Sounds like a job for an editor macro or a simple script.

Perhaps someone else can think of something else, but adding it to the command line isn't feasible... If we started throwing random convenient stuff there it would quickly become a mess


On 02/22/2013 01:30 PM, =D6=DC=C3=CE=CF=EB wrote:
Yes=A3=ACI mean all event headers send from a= specific agent. some headers need for all events, not only hostname. now I have to configure it on every source time and time again.
for example:
if i configure 10 sources, every source need 5 header,  I have to configure 50 times.
other configure of the same type source have the same problem.

eg: 
agent1.sources =3Dgamelog src1 src2 ... src10

agent1.sources.gamelog.fileSuffix=3D.fin
agent1.sources.gamelog.fileHeader=3Dtrue
agent1.sources.gamelog.fileHeaderKey=3Dfullfilename
agent1.sources.gamelog.batchSize=3D100
agent1.sources.gamelog.bufferMaxLines=3D1000
agent1.sources.gamelog.bufferMaxLineLength=3D5000 
agent1.sources.gamelog.interceptors =3D i1 i2 i3
#for %{host} org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.gamelog.interceptors.i1.type =3D host
agent1.sources.gamelog.interceptors.i2.type =3D timestamp
...
agent1.sources.gamelog.interceptors.i3.type =3D static
agent1.sources.gamelog.interceptors.i3.key =3D filename
agent1.sources.gamelog.interceptors.i3.value =3D gamelog

repleat configure for src1,...src10 for 10 or even more times.

it's very boring.
maybe we could group the sources,sinks and configure them one or several times.

Best Regards,
Andy


2013/2/22 Juhani Connolly <juhani_connolly@cyberagent.co.jp>
If I understand your question, using a static interceptor is the expected way to do this.

Not sure what you mean by agent header? Do you mean all event headers sent from a specific agent? I don't imagine w= e will be adding a command-line parameter to do this, it wouldn't be consistent and would be superfluous.

If you want to reuse a configuration file, and add a header to inform what agent it came from, perhaps you could use the hostname interceptor?


On 02/22/2013 12:06 PM, =D6=DC=C3=CE=CF=EB wrote:
I want add some key/value to the agent header, but it's not convenient to do so. why not flume agent support --headerFile just like flume avro-client?

my requirement is :
I want use spooling source to send files to another flume node to write to hdfs.
and i want to name hdfs file to have the original file name prefix, not the full path of the orgin file.

now I have to add interceptors of conf to do that.

so I have the question, why not flume agent support --headerFile just like flume avro-client?

Best Regards,
Andy





--047d7b15a71b994d5f04d657c271--