flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From IT CTO <goi....@gmail.com>
Subject Re: HDFS hdfs.path custom config ? possible ?
Date Tue, 08 Sep 2015 17:19:25 GMT
If the path stays with %h what would be the folder name with round 15 unit
minute?

בתאריך יום ג׳, 8 בספט׳ 2015, 19:52 מאת Hari Shreedharan <
hshreedharan@cloudera.com>:

> You can use roundValue as 15 and roundUnit as Minute - so a new dir would
> get created every 15 mins.
>
>
> Thanks,
> Hari
>
> On Tue, Sep 8, 2015 at 9:42 AM, Sutanu Das <sd2302@att.com> wrote:
>
>> Hi Community,
>>
>>
>>
>> Is it possible to create /write HDFS dirs./folders every 15 mins?
>>
>>
>>
>> Right now, we use /hdfs_sink_path/%Y/%m/%d/%H --- ie using every Hour,
>> But, Is it possible to write every 15 mins? – not every minute…
>>
>>
>>
>> If this is not possible out of the box config, can DEV team please advise
>> which code/java-file do we need to customize so Flume creates dirs. Every
>> 15 mins? Like /%Y/%m/%d/%H*/%15M ?*
>>
>>
>>
>> Please help us, thanks.
>>
>>
>>
>> Here is our present 1 hour HDFS dir/folder creation config:
>>
>>
>>
>>
>>
>> airwave.sinks.sink1.type = hdfs
>>
>> airwave.sinks.sink1.channel = channel1
>>
>> airwave.sinks.sink1.hdfs.path =
>> /prod/hadoop/smallsite/airwave_flume_ingest_hak_3_amp_1*/%Y/%m/%d/%H*
>>
>> airwave.sinks.sink1.hdfs.fileType = DataStream
>>
>> airwave.sinks.sink1.hdfs.writeFormat = Text
>>
>> airwave.sinks.sink1.hdfs.filePrefix = Airwave_amp_1
>>
>> airwave.sinks.sink1.hdfs.useLocalTimeStamp = true
>>
>> airwave.sinks.sink1.hdfs.round = true
>>
>> airwave.sinks.sink1.hdfs.roundValue = 1
>>
>> airwave.sinks.sink1.hdfs.roundUnit = hour
>>
>> airwave.sinks.sink1.hdfs.callTimeout = 10000000000
>>
>>
>>
>
> --
Eran | "You don't need eyes to see, you need vision" (Faithless)

Mime
View raw message