flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bessenyei Balázs Donát <bes...@apache.org>
Subject Re: Define and leverage custom constants in agent configuration
Date Fri, 02 Dec 2016 11:26:55 GMT
(+CC: dev@)

I have requested reviews and comments on the ticket.

If nobody disagrees, I'll commit that change soon.


Donát


2016-12-02 9:33 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it>:

> That would be a cool option, Donat!
>
> On 01/12/2016 19:41, Bessenyei Balázs Donát wrote:
>
> Hi Roberto,
>
> Do you mean something like the proposed change at
> https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your
> requirements?
>
> Thank you,
>
> Donat
>
> On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Shang_Wu@symantec.com> wrote:
>
>> Hi Robert,
>>
>> Looks like what you want is more like a global configuration, and based
>> on my limited knowledge in Apache Flume, it is missing now.
>>
>> BR,
>>
>> Shang
>>
>> *From: *Roberto Coluccio <roberto.coluccio@eng.it>
>> *Reply-To: *"user@flume.apache.org" <user@flume.apache.org>
>> *Date: *Thursday, December 1, 2016 at 2:59 AM
>> *To: *"user@flume.apache.org" <user@flume.apache.org>
>> *Subject: *Re: Define and leverage custom constants in agent
>> configuration
>>
>>
>>
>> Hi Silvio,
>>
>> thanks for your suggestion. It gives me a workaround, but it's not
>> exactly what I was hoping/asking about.
>>
>> I'm going to assume what I'd want to do being just not doable.
>>
>> Thank you,
>>
>> Roberto
>>
>>
>>
>> On 29/11/2016 09:26, Silvio Di gregorio wrote:
>>
>> with the "static interceptor" you can add key/value pairs in your flume
>> event header:
>>
>>
>>
>> agentOne.sources.s1.interceptors = interOne
>>
>> agentOne.sources.s1.interceptors.interOne.type = static
>>
>> agentOne.sources.s1.interceptors.interOne.key = costante
>>
>> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>>
>>
>>
>> I hope it is useful
>>
>> have nice day
>>
>> Silvio
>>
>>
>>
>> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it>:
>>
>> Hello folks,
>>
>> I was wondering if it's possible to define custom constants in a Flume
>> agent configuration file, so to avoid errors and replicated stuff when not
>> needed. An example to make my point clear:
>>
>> assume I want to launch an agent of type *myAgent* configured with file
>> channel and hdfs sink.
>>
>> I want the file channel to persist data and checkpoint into
>> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>>
>> Also, I want the hdfs sink to write files with prefix "myAgent1"
>>
>> In this scenario, I should harcorde the "myAgent1" string 3 times, one
>> for each configuration parameter. This is error prone and kinda frustrating.
>>
>> Is there a way to define a constant like "myConstant" in the
>> configuration file, and leverage it as parameter in the other configuration
>> lines?
>>
>> e.g.
>>
>> myConstant = myAgent1
>> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
>> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>>
>> It seems like the use of flume events header keys, but I don't want to
>> add interceptors or other components just to have parameters in my config
>> file.
>>
>> Thanks for your help.
>>
>> Best regards,
>>
>> Roberto
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Roberto Coluccio*
>>
>> Solution Designer
>> Direzione Ricerca e Innovazione
>> roberto.coluccio@eng.it
>>
>> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>>
>>
>>
>> *Engineering Ingegneria Informatica spa*
>> Via Marconi, 10, 40122 Bologna
>>
>> www.eng.it
>>
>>
>>
>
> --
>
>
>
>
>
>
>
>
>
> *Roberto Coluccio*
>
> Solution Designer
> Direzione Ricerca e Innovazione
> roberto.coluccio@eng.it
>
> Interno 604135 | Esterno +39 0510435135 <+39%20051%20043%205135>
>
>
> *Engineering Ingegneria Informatica spa*
> Via Marconi, 10, 40122 Bologna
>
> www.eng.it
>
>
>

Mime
View raw message