flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Coluccio <roberto.coluc...@eng.it>
Subject Re: Define and leverage custom constants in agent configuration
Date Thu, 01 Dec 2016 10:59:39 GMT
Hi Silvio,

thanks for your suggestion. It gives me a workaround, but it's not 
exactly what I was hoping/asking about.

I'm going to assume what I'd want to do being just not doable.

Thank you,

Roberto


On 29/11/2016 09:26, Silvio Di gregorio wrote:
> with the "static interceptor" you can add key/value pairs in your 
> flume event header:
>
> agentOne.sources.s1.interceptors = interOne
> agentOne.sources.s1.interceptors.interOne.type = static
> agentOne.sources.s1.interceptors.interOne.key = costante
> agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi
>
> I hope it is useful
> have nice day
> Silvio
>
> 2016-11-28 18:26 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it 
> <mailto:roberto.coluccio@eng.it>>:
>
>     Hello folks,
>
>     I was wondering if it's possible to define custom constants in a
>     Flume agent configuration file, so to avoid errors and replicated
>     stuff when not needed. An example to make my point clear:
>
>     assume I want to launch an agent of type /myAgent/ configured with
>     file channel and hdfs sink.
>
>     I want the file channel to persist data and checkpoint into
>     "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
>     Also, I want the hdfs sink to write files with prefix "myAgent1"
>
>     In this scenario, I should harcorde the "myAgent1" string 3 times,
>     one for each configuration parameter. This is error prone and
>     kinda frustrating.
>
>     Is there a way to define a constant like "myConstant" in the
>     configuration file, and leverage it as parameter in the other
>     configuration lines?
>
>     e.g.
>
>     myConstant = myAgent1
>     myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
>     myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
>     It seems like the use of flume events header keys, but I don't
>     want to add interceptors or other components just to have
>     parameters in my config file.
>
>     Thanks for your help.
>
>     Best regards,
>
>     Roberto
>
>
>

-- 

**

**

**

**

**

*Roberto Coluccio*

Solution Designer
Direzione Ricerca e Innovazione
roberto.coluccio@eng.it <mailto:roberto.coluccio@eng.it>

Interno 604135 | Esterno +39 0510435135



*Engineering Ingegneria Informatica spa*
Via Marconi, 10, 40122 Bologna

www.eng.it <http://www.eng.it/>


Mime
View raw message