flume-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mpe...@apache.org
Subject svn commit: r1358917 - in /flume/branches/branch-1.2.0: ./ flume-ng-doc/sphinx/FlumeUserGuide.rst
Date Mon, 09 Jul 2012 05:13:35 GMT
Author: mpercy
Date: Mon Jul  9 05:13:35 2012
New Revision: 1358917

URL: http://svn.apache.org/viewvc?rev=1358917&view=rev
Log:
FLUME-1352. Add documentation for HDFS path rounddown.

(Hari Shreedharan via Mike Percy)

Modified:
    flume/branches/branch-1.2.0/   (props changed)
    flume/branches/branch-1.2.0/flume-ng-doc/sphinx/FlumeUserGuide.rst

Propchange: flume/branches/branch-1.2.0/
------------------------------------------------------------------------------
  Merged /flume/trunk:r1358893

Modified: flume/branches/branch-1.2.0/flume-ng-doc/sphinx/FlumeUserGuide.rst
URL: http://svn.apache.org/viewvc/flume/branches/branch-1.2.0/flume-ng-doc/sphinx/FlumeUserGuide.rst?rev=1358917&r1=1358916&r2=1358917&view=diff
==============================================================================
--- flume/branches/branch-1.2.0/flume-ng-doc/sphinx/FlumeUserGuide.rst (original)
+++ flume/branches/branch-1.2.0/flume-ng-doc/sphinx/FlumeUserGuide.rst Mon Jul  9 05:13:35
2012
@@ -954,6 +954,9 @@ hdfs.threadsPoolSize    10            Nu
 hdfs.rollTimerPoolSize  1             Number of threads per HDFS sink for scheduling timed
file rolling
 hdfs.kerberosPrincipal  --            Kerberos user principal for accessing secure HDFS
 hdfs.kerberosKeytab     --            Kerberos keytab for accessing secure HDFS
+hdfs.round              false         Should the timestamp be rounded down (if true, affects
all time based escape sequences except %t)
+hdfs.roundValue         1             Rounded down to the highest multiple of this (in the
unit configured using ``hdfs.roundUnit``), less than current time.
+hdfs.roundUnit          second        The unit of the round down value - ``second``, ``minute``
or ``hour``.
 serializer              ``TEXT``      Other possible options include ``AVRO_EVENT`` or the
                                       fully-qualified class name of an implementation of
the
                                       ``EventSerializer.Builder`` interface.
@@ -968,8 +971,13 @@ Example for agent named **agent_foo**:
   agent_foo.sinks = hdfsSink-1
   agent_foo.sinks.hdfsSink-1.type = hdfs
   agent_foo.sinks.hdfsSink-1.channels = memoryChannel-1
-  agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d
-  agent_foo.sinks.hdfsSink-1.hdfs.filePrevix = events-
+  agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+  agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = events-
+  agent_foo.sinks.hdfsSink-1.hdfs.round = true
+  agent_foo.sinks.hdfsSink-1.hdfs.roundValue = 10
+  agent_foo.sinks.hdfsSink-1.hdfs.roundUnit = minute
+
+The above configuration will round down the timestamp to the last 10th minute. For example,
an event with timestamp 11:54:34 AM, June 12, 2012 will cause the hdfs path to become ``/flume/events/2012-06-12/1150/00``.
 
 
 Logger Sink



Mime
View raw message