flume-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mpe...@apache.org
Subject svn commit: r1358893 - /flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
Date Mon, 09 Jul 2012 02:02:05 GMT
Author: mpercy
Date: Mon Jul  9 02:02:05 2012
New Revision: 1358893

URL: http://svn.apache.org/viewvc?rev=1358893&view=rev
Log:
FLUME-1352. Add documentation for HDFS path rounddown.

(Hari Shreedharan via Mike Percy)

Modified:
    flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst

Modified: flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
URL: http://svn.apache.org/viewvc/flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst?rev=1358893&r1=1358892&r2=1358893&view=diff
==============================================================================
--- flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst (original)
+++ flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst Mon Jul  9 02:02:05 2012
@@ -954,6 +954,9 @@ hdfs.threadsPoolSize    10            Nu
 hdfs.rollTimerPoolSize  1             Number of threads per HDFS sink for scheduling timed
file rolling
 hdfs.kerberosPrincipal  --            Kerberos user principal for accessing secure HDFS
 hdfs.kerberosKeytab     --            Kerberos keytab for accessing secure HDFS
+hdfs.round              false         Should the timestamp be rounded down (if true, affects
all time based escape sequences except %t)
+hdfs.roundValue         1             Rounded down to the highest multiple of this (in the
unit configured using ``hdfs.roundUnit``), less than current time.
+hdfs.roundUnit          second        The unit of the round down value - ``second``, ``minute``
or ``hour``.
 serializer              ``TEXT``      Other possible options include ``AVRO_EVENT`` or the
                                       fully-qualified class name of an implementation of
the
                                       ``EventSerializer.Builder`` interface.
@@ -968,8 +971,13 @@ Example for agent named **agent_foo**:
   agent_foo.sinks = hdfsSink-1
   agent_foo.sinks.hdfsSink-1.type = hdfs
   agent_foo.sinks.hdfsSink-1.channels = memoryChannel-1
-  agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d
-  agent_foo.sinks.hdfsSink-1.hdfs.filePrevix = events-
+  agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+  agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = events-
+  agent_foo.sinks.hdfsSink-1.hdfs.round = true
+  agent_foo.sinks.hdfsSink-1.hdfs.roundValue = 10
+  agent_foo.sinks.hdfsSink-1.hdfs.roundUnit = minute
+
+The above configuration will round down the timestamp to the last 10th minute. For example,
an event with timestamp 11:54:34 AM, June 12, 2012 will cause the hdfs path to become ``/flume/events/2012-06-12/1150/00``.
 
 
 Logger Sink



Mime
View raw message