flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IT CTO <goi....@gmail.com>
Subject Re: Interceptor vrs Serialeztion
Date Mon, 21 Sep 2015 15:18:50 GMT
Thanks, I will try it.

בתאריך יום ב׳, 21 בספט׳ 2015, 18:15 מאת Ahmed Vila <avila@devlogic.eu>:

> I think that interceptor is a way to go and you can use regex_extractor
> interceptor instead of building your own - it simplifies deployment.
> https://flume.apache.org/FlumeUserGuide.html#regex-extractor-interceptor
>
> Further, you can use RegexExtractorInterceptorMillisSerializer for this
> interceptor's type in order to do datetime parsing and do the formatting on
> fly that will be suitable for your hdfs folder (f.e. extract year-month).
> https://flume.apache.org/FlumeUserGuide.html#example-2
>
> Later on, you can use header replacement string in the hdfs path to inject
> a header defined with serializer's name ("timestamp" in the example above).
> Hdfs Sink supports timestamp header out of the box and applies escape
> sequences against it by default, unless hdfs.useLocalTimeStamp = true.
> So, a1.sinks.k1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S  will have
> replacement values from the timestamp header produced by regex extractor.
>
>
>
>
> On Mon, Sep 21, 2015 at 5:03 PM, IT CTO <goi.cto@gmail.com> wrote:
>
>> I want to read files and write them to hdfs but I want to write the to a
>> date partitioned folder based on a date value IN THE ROW.
>> Should I write a custom interceptor or custom serializer?
>> Eran
>> --
>> Eran | "You don't need eyes to see, you need vision" (Faithless)
>>
>
>
>
> --
>
> Best regards,
> Ahmed Vila | Senior software developer
> DevLogic | Sarajevo | Bosnia and Herzegovina
>
> Office : +387 33 942 123
> Mobile: +387 62 139 348
>
> Website: www.devlogic.eu
> E-mail   : avila@devlogic.eu
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.
>
> ---------------------------------------------------------------------
> This e-mail and any attachment is for authorised use by the intended
> recipient(s) only. This email contains confidential information. It should
> not be copied, disclosed to, retained or used by, any party other than the
> intended recipient. Any unauthorised distribution, dissemination or copying
> of this E-mail or its attachments, and/or any use of any information
> contained in them, is strictly prohibited and may be illegal. If you are
> not an intended recipient then please promptly delete this e-mail and any
> attachment and all copies and inform the sender directly via email. Any
> emails that you send to us may be monitored by systems or persons other
> than the named communicant for the purposes of ascertaining whether the
> communication complies with the law and company policies.

-- 
Eran | "You don't need eyes to see, you need vision" (Faithless)

Mime
View raw message