flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Coluccio <roberto.coluc...@eng.it>
Subject Re: Define and leverage custom constants in agent configuration
Date Fri, 02 Dec 2016 08:33:25 GMT
That would be a cool option, Donat!


On 01/12/2016 19:41, Bessenyei Balázs Donát wrote:
>
> Hi Roberto,
>
> Do you mean something like the proposed change at 
> https://issues.apache.org/jira/browse/FLUME-2993 ? Would that fit your 
> requirements?
>
> Thank you,
>
> Donat
>
>
> On Dec 1, 2016 7:33 PM, "Shang Wu (CS)" <Shang_Wu@symantec.com 
> <mailto:Shang_Wu@symantec.com>> wrote:
>
>     Hi Robert,
>
>     Looks like what you want is more like a global configuration, and
>     based on my limited knowledge in Apache Flume, it is missing now.
>
>     BR,
>
>     Shang
>
>     *From: *Roberto Coluccio <roberto.coluccio@eng.it
>     <mailto:roberto.coluccio@eng.it>>
>     *Reply-To: *"user@flume.apache.org <mailto:user@flume.apache.org>"
>     <user@flume.apache.org <mailto:user@flume.apache.org>>
>     *Date: *Thursday, December 1, 2016 at 2:59 AM
>     *To: *"user@flume.apache.org <mailto:user@flume.apache.org>"
>     <user@flume.apache.org <mailto:user@flume.apache.org>>
>     *Subject: *Re: Define and leverage custom constants in agent
>     configuration
>
>     Hi Silvio,
>
>     thanks for your suggestion. It gives me a workaround, but it's not
>     exactly what I was hoping/asking about.
>
>     I'm going to assume what I'd want to do being just not doable.
>
>     Thank you,
>
>     Roberto
>
>     On 29/11/2016 09:26, Silvio Di gregorio wrote:
>
>         with the "static interceptor" you can add key/value pairs in
>         your flume event header:
>
>         agentOne.sources.s1.interceptors = interOne
>
>         agentOne.sources.s1.interceptors.interOne.type = static
>
>         agentOne.sources.s1.interceptors.interOne.key = costante
>
>         agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>
>         I hope it is useful
>
>         have nice day
>
>         Silvio
>
>         2016-11-28 18:26 GMT+01:00 Roberto Coluccio
>         <roberto.coluccio@eng.it <mailto:roberto.coluccio@eng.it>>:
>
>             Hello folks,
>
>             I was wondering if it's possible to define custom
>             constants in a Flume agent configuration file, so to avoid
>             errors and replicated stuff when not needed. An example to
>             make my point clear:
>
>             assume I want to launch an agent of type /myAgent/
>             configured with file channel and hdfs sink.
>
>             I want the file channel to persist data and checkpoint
>             into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
>             Also, I want the hdfs sink to write files with prefix
>             "myAgent1"
>
>             In this scenario, I should harcorde the "myAgent1" string
>             3 times, one for each configuration parameter. This is
>             error prone and kinda frustrating.
>
>             Is there a way to define a constant like "myConstant" in
>             the configuration file, and leverage it as parameter in
>             the other configuration lines?
>
>             e.g.
>
>             myConstant = myAgent1
>             myAgent.channels.myChannel.dataDirs =
>             ~/flume/${myConstant}/data
>             myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
>             It seems like the use of flume events header keys, but I
>             don't want to add interceptors or other components just to
>             have parameters in my config file.
>
>             Thanks for your help.
>
>             Best regards,
>
>             Roberto
>
>     -- 
>
>     **
>
>     **
>
>     **
>
>     **
>
>     *Roberto Coluccio*
>
>     Solution Designer
>     Direzione Ricerca e Innovazione
>     roberto.coluccio@eng.it <mailto:roberto.coluccio@eng.it>
>
>     Interno 604135 | Esterno +39 0510435135 <tel:+39%20051%20043%205135>
>
>
>
>     *Engineering Ingegneria Informatica spa*
>     Via Marconi, 10, 40122 Bologna
>
>     www.eng.it <http://www.eng.it/>
>

-- 

**

**

**

**

**

*Roberto Coluccio*

Solution Designer
Direzione Ricerca e Innovazione
roberto.coluccio@eng.it <mailto:roberto.coluccio@eng.it>

Interno 604135 | Esterno +39 0510435135



*Engineering Ingegneria Informatica spa*
Via Marconi, 10, 40122 Bologna

www.eng.it <http://www.eng.it/>


Mime
View raw message