Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: neutral (nike.apache.org: 209.85.220.175 is neither permitted
 nor denied by domain of matt.tenenbaum@rockyou.com)
MIME-Version: 1.0
In-Reply-To: 
 <CAE6kwsPJVGpJ9ycCy9EufKv0mVSLbQH22h0ak5j_47Onb1nVPg@mail.gmail.com>
References: 
 <CAE6kwsPJVGpJ9ycCy9EufKv0mVSLbQH22h0ak5j_47Onb1nVPg@mail.gmail.com>
From: Matt Tenenbaum <matt.tenenbaum@rockyou.com>
Date: Wed, 30 Jul 2014 08:46:55 -0700
Message-ID: 
 <CA+ZuXfPcFe1Vr8xcGe58Zd9f6VZJj9iBGTcCtZLAnuNkyA2nWg@mail.gmail.com>
Subject: Re: Interceptor,
 multiplexing Events to two differents path in HDFS dynamically,
To: user <user@flume.apache.org>
Content-Type: multipart/alternative; boundary=20cf302efbb202e49504ff6b1205

--20cf302efbb202e49504ff6b1205
Content-Type: text/plain; charset=UTF-8

Hi Guillermo --

If I understand correctly, you want to create two paths. Events that pass
the validating interceptor get directed to one path in HDFS and events that
fail get directed to another. If so, you could define your interceptor to
set a 'valid' header to 'yes' or 'no' depending on its internal logic. Then
you would be able to use the ordinary 'selector.mapping' configuration
behavior to route to separate channels and attach your separate
destinations as sinks to each of those channels. I'm omitting all but the
relevant configurations:


multi.sources = s1
multi.sinks = valid invalid
multi.channels = cpos cneg

multi.sources.s1.selector.type = multiplexing
# Route based on the value of the 'valid' header set by the interceptor
multi.sources.s1.selector.header = valid
# Send events with valid == yes to 'cpos' channel
multi.sources.s1.selector.mapping.yes = cpos
# Send events with valid == no to 'cneg' channel
multi.sources.s1.selector.mapping.no = cneg

multi.sinks.valid.type = hdfs
multi.sinks.valid.channel = cpos
multi.sinks.valid.hdfs.path = hdfs://nn/path/for/valid/data
# Other configurations for valid data...

multi.sinks.invalid.type = hdfs
multi.sinks.invalid.channel = cneg
multi.sinks.invalid.hdfs.path = hdfs://nn/path/for/invalid/data
# Other configuration for data that fails validation...

Would that do what you want?

Cheers
-mt


On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <konstt2000@gmail.com>
wrote:

> Hello,
>
>
> I want to create an interceptor to validate and multiplexing the Events,
> if they don't get validated to send them a different path in HDFS.
>
> Is it possible to do it?
> I thought.. to have others interceptor and the HDFS path is built on base
> of these static interceptors, so I could modify them on my validator
> interceptor. Is it fine?
>

--20cf302efbb202e49504ff6b1205
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Guillermo --<div><br></div><div>If I understand correct=
ly, you want to create two paths. Events that pass the validating intercept=
or get directed to one path in HDFS and events that fail get directed to an=
other. If so, you could define your interceptor to set a &#39;valid&#39; he=
ader to &#39;yes&#39; or &#39;no&#39; depending on its internal logic. Then=
 you would be able to use the ordinary &#39;selector.mapping&#39; configura=
tion behavior to route to separate channels and attach your separate destin=
ations as sinks to each of those channels. I&#39;m omitting all but the rel=
evant configurations:</div>

<div><br></div><div><br></div><div>multi.sources =3D s1</div><div>multi.sin=
ks =3D valid invalid</div><div>multi.channels =3D cpos cneg</div><div><br><=
/div><div>multi.sources.s1.selector.type =3D multiplexing</div><div># Route=
 based on the value of the &#39;valid&#39; header set by the interceptor</d=
iv>

<div>multi.sources.s1.selector.header =3D valid</div><div># Send events wit=
h valid =3D=3D yes to &#39;cpos&#39; channel</div><div>multi.sources.s1.sel=
ector.mapping.yes =3D cpos</div><div># Send events with valid =3D=3D no to =
&#39;cneg&#39; channel</div>

<div><a href=3D"http://multi.sources.s1.selector.mapping.no">multi.sources.=
s1.selector.mapping.no</a> =3D cneg</div><div><br></div><div>multi.sinks.va=
lid.type =3D hdfs</div><div>multi.sinks.valid.channel =3D cpos</div><div>mu=
lti.sinks.valid.hdfs.path =3D hdfs://nn/path/for/valid/data</div>

<div># Other configurations for valid data...</div><div><br></div><div>mult=
i.sinks.invalid.type =3D hdfs</div><div>multi.sinks.invalid.channel =3D cne=
g</div><div>multi.sinks.invalid.hdfs.path =3D hdfs://nn/path/for/invalid/da=
ta</div>

<div># Other configuration for data that fails validation...</div><div><br>=
</div><div>Would that do what you want?</div><div><br></div><div>Cheers</di=
v><div>-mt</div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmai=
l_quote">

On Wed, Jul 30, 2014 at 6:02 AM, Guillermo Ortiz <span dir=3D"ltr">&lt;<a h=
ref=3D"mailto:konstt2000@gmail.com" target=3D"_blank">konstt2000@gmail.com<=
/a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div dir=3D"ltr"><div><div><div>Hello, <br><br><br></div>I want to create a=
n interceptor to validate and multiplexing the Events, if they don&#39;t ge=
t validated to send them a different path in HDFS. <br><br></div>Is it poss=
ible to do it? <br>


</div><div>I thought.. to have others interceptor and the HDFS path is buil=
t on base of these static interceptors, so I could modify them on my valida=
tor interceptor. Is it fine?<br></div></div>
</blockquote></div><br></div>

--20cf302efbb202e49504ff6b1205--