flume-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r1038703 [2/6] - in /websites/staging/flume/trunk/content: ./ .doctrees/ .doctrees/releases/ _sources/ _sources/releases/ css/ fonts/ images/ images/profiles/ img/ js/ releases/
Date Tue, 08 Jan 2019 13:13:56 GMT
Modified: websites/staging/flume/trunk/content/FlumeUserGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeUserGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeUserGuide.html Tue Jan  8 13:13:55 2019
@@ -7,7 +7,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     
-    <title>Flume 1.8.0 User Guide &mdash; Apache Flume</title>
+    <title>Flume 1.9.0 User Guide &mdash; Apache Flume</title>
     
     <link rel="stylesheet" href="_static/flume.css" type="text/css" />
     <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -26,7 +26,7 @@
     <script type="text/javascript" src="_static/doctools.js"></script>
     <link rel="top" title="Apache Flume" href="index.html" />
     <link rel="up" title="Documentation" href="documentation.html" />
-    <link rel="next" title="Flume 1.8.0 Developer Guide" href="FlumeDeveloperGuide.html" />
+    <link rel="next" title="Flume 1.9.0 Developer Guide" href="FlumeDeveloperGuide.html" />
     <link rel="prev" title="Documentation" href="documentation.html" /> 
   </head>
   <body>
@@ -37,6 +37,7 @@
         <div class="logo">
           <a href="index.html">
             <img class="logo" src="_static/flume-logo.png" alt="Logo"/>
+          </a>
         </div>
       </td>
       <td width="2%">
@@ -59,8 +60,8 @@
         <div class="bodywrapper">
           <div class="body">
             
-  <div class="section" id="flume-1-8-0-user-guide">
-<h1>Flume 1.8.0 User Guide<a class="headerlink" href="#flume-1-8-0-user-guide" title="Permalink to this headline">¶</a></h1>
+  <div class="section" id="flume-1-9-0-user-guide">
+<h1>Flume 1.9.0 User Guide<a class="headerlink" href="#flume-1-9-0-user-guide" title="Permalink to this headline">¶</a></h1>
 <div class="section" id="introduction">
 <h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
 <div class="section" id="overview">
@@ -73,13 +74,6 @@ Since data sources are customizable, Flu
 of event data including but not limited to network traffic data, social-media-generated data,
 email messages and pretty much any data source possible.</p>
 <p>Apache Flume is a top level project at the Apache Software Foundation.</p>
-<p>There are currently two release code lines available, versions 0.9.x and 1.x.</p>
-<p>Documentation for the 0.9.x track is available at
-<a class="reference external" href="http://archive.cloudera.com/cdh/3/flume/UserGuide/">the Flume 0.9.x User Guide</a>.</p>
-<p>This documentation applies to the 1.4.x track.</p>
-<p>New and existing users are encouraged to use the 1.x releases so as to
-leverage the performance improvements and configuration flexibilities available
-in the latest architecture.</p>
 </div>
 <div class="section" id="system-requirements">
 <h3>System Requirements<a class="headerlink" href="#system-requirements" title="Permalink to this headline">¶</a></h3>
@@ -715,6 +709,176 @@ specified. If no channels are designated
 the selector will attempt to write the events to the optional channels. Any
 failures are simply ignored in that case.</p>
 </div>
+<div class="section" id="ssl-tls-support">
+<h3>SSL/TLS support<a class="headerlink" href="#ssl-tls-support" title="Permalink to this headline">¶</a></h3>
+<p>Several Flume components support the SSL/TLS protocols in order to communicate with other systems
+securely.</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="55%" />
+<col width="45%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">Component</th>
+<th class="head">SSL server or client</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td>Avro Source</td>
+<td>server</td>
+</tr>
+<tr class="row-odd"><td>Avro Sink</td>
+<td>client</td>
+</tr>
+<tr class="row-even"><td>Thrift Source</td>
+<td>server</td>
+</tr>
+<tr class="row-odd"><td>Thrift Sink</td>
+<td>client</td>
+</tr>
+<tr class="row-even"><td>Kafka Source</td>
+<td>client</td>
+</tr>
+<tr class="row-odd"><td>Kafka Channel</td>
+<td>client</td>
+</tr>
+<tr class="row-even"><td>Kafka Sink</td>
+<td>client</td>
+</tr>
+<tr class="row-odd"><td>HTTP Source</td>
+<td>server</td>
+</tr>
+<tr class="row-even"><td>JMS Source</td>
+<td>client</td>
+</tr>
+<tr class="row-odd"><td>Syslog TCP Source</td>
+<td>server</td>
+</tr>
+<tr class="row-even"><td>Multiport Syslog TCP Source</td>
+<td>server</td>
+</tr>
+</tbody>
+</table>
+<p>The SSL compatible components have several configuration parameters to set up SSL, like
+enable SSL flag, keystore / truststore parameters (location, password, type) and additional
+SSL parameters (eg. disabled protocols).</p>
+<p>Enabling SSL for a component is always specified at component level in the agent configuration file.
+So some components may be configured to use SSL while others not (even with the same component type).</p>
+<p>The keystore / truststore setup can be specified at component level or globally.</p>
+<p>In case of the component level setup, the keystore / truststore is configured in the agent
+configuration file through component specific parameters. The advantage of this method is that the
+components can use different keystores (if this would be needed). The disadvantage is that the
+keystore parameters must be copied for each component in the agent configuration file.
+The component level setup is optional, but if it is defined, it has higher precedence than
+the global parameters.</p>
+<p>With the global setup, it is enough to define the keystore / truststore parameters once
+and use the same settings for all components, which means less and more centralized configuration.</p>
+<p>The global setup can be configured either through system properties or through environment variables.</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="22%" />
+<col width="20%" />
+<col width="59%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">System property</th>
+<th class="head">Environment variable</th>
+<th class="head">Description</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td>javax.net.ssl.keyStore</td>
+<td>FLUME_SSL_KEYSTORE_PATH</td>
+<td>Keystore location</td>
+</tr>
+<tr class="row-odd"><td>javax.net.ssl.keyStorePassword</td>
+<td>FLUME_SSL_KEYSTORE_PASSWORD</td>
+<td>Keystore password</td>
+</tr>
+<tr class="row-even"><td>javax.net.ssl.keyStoreType</td>
+<td>FLUME_SSL_KEYSTORE_TYPE</td>
+<td>Keystore type (by default JKS)</td>
+</tr>
+<tr class="row-odd"><td>javax.net.ssl.trustStore</td>
+<td>FLUME_SSL_TRUSTSTORE_PATH</td>
+<td>Truststore location</td>
+</tr>
+<tr class="row-even"><td>javax.net.ssl.trustStorePassword</td>
+<td>FLUME_SSL_TRUSTSTORE_PASSWORD</td>
+<td>Truststore password</td>
+</tr>
+<tr class="row-odd"><td>javax.net.ssl.trustStoreType</td>
+<td>FLUME_SSL_TRUSTSTORE_TYPE</td>
+<td>Truststore type (by default JKS)</td>
+</tr>
+<tr class="row-even"><td>flume.ssl.include.protocols</td>
+<td>FLUME_SSL_INCLUDE_PROTOCOLS</td>
+<td>Protocols to include when calculating enabled protocols. A comma (,) separated list.
+Excluded protocols will be excluded from this list if provided.</td>
+</tr>
+<tr class="row-odd"><td>flume.ssl.exclude.protocols</td>
+<td>FLUME_SSL_EXCLUDE_PROTOCOLS</td>
+<td>Protocols to exclude when calculating enabled protocols. A comma (,) separated list.</td>
+</tr>
+<tr class="row-even"><td>flume.ssl.include.cipherSuites</td>
+<td>FLUME_SSL_INCLUDE_CIPHERSUITES</td>
+<td>Cipher suites to include when calculating enabled cipher suites. A comma (,) separated list.
+Excluded cipher suites will be excluded from this list if provided.</td>
+</tr>
+<tr class="row-odd"><td>flume.ssl.exclude.cipherSuites</td>
+<td>FLUME_SSL_EXCLUDE_CIPHERSUITES</td>
+<td>Cipher suites to exclude when calculating enabled cipher suites. A comma (,) separated list.</td>
+</tr>
+</tbody>
+</table>
+<p>The SSL system properties can either be passed on the command line or by setting the <tt class="docutils literal"><span class="pre">JAVA_OPTS</span></tt>
+environment variable in <em>conf/flume-env.sh</em>. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)</p>
+<div class="highlight-properties"><div class="highlight"><pre><span class="na">export JAVA_OPTS</span><span class="o">=</span><span class="s">&quot;$JAVA_OPTS -Djavax.net.ssl.keyStore=/path/to/keystore.jks&quot;</span>
+<span class="na">export JAVA_OPTS</span><span class="o">=</span><span class="s">&quot;$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=password&quot;</span>
+</pre></div>
+</div>
+<p>Flume uses the system properties defined in JSSE (Java Secure Socket Extension), so this is
+a standard way for setting up SSL. On the other hand, specifying passwords in system properties
+means that the passwords can be seen in the process list. For cases where it is not acceptable,
+it is also be possible to define the parameters in environment variables. Flume initializes
+the JSSE system properties from the corresponding environment variables internally in this case.</p>
+<p>The SSL environment variables can either be set in the shell environment before
+starting Flume or in <em>conf/flume-env.sh</em>. (Although, using the command line is inadvisable because
+the commands including the passwords will be saved to the command history.)</p>
+<div class="highlight-properties"><div class="highlight"><pre><span class="na">export FLUME_SSL_KEYSTORE_PATH</span><span class="o">=</span><span class="s">/path/to/keystore.jks</span>
+<span class="na">export FLUME_SSL_KEYSTORE_PASSWORD</span><span class="o">=</span><span class="s">password</span>
+</pre></div>
+</div>
+<p><strong>Please note:</strong></p>
+<ul class="simple">
+<li>SSL must be enabled at component level. Specifying the global SSL parameters alone will not
+have any effect.</li>
+<li>If the global SSL parameters are specified at multiple levels, the priority is the
+following (from higher to lower):<ul>
+<li>component parameters in agent config</li>
+<li>system properties</li>
+<li>environment variables</li>
+</ul>
+</li>
+<li>If SSL is enabled for a component, but the SSL parameters are not specified in any of the ways
+described above, then<ul>
+<li>in case of keystores: configuration error</li>
+<li>in case of truststores: the default truststore will be used (<tt class="docutils literal"><span class="pre">jssecacerts</span></tt> / <tt class="docutils literal"><span class="pre">cacerts</span></tt> in Oracle JDK)</li>
+</ul>
+</li>
+<li>The trustore password is optional in all cases. If not specified, then no integrity check will be
+performed on the truststore when it is opened by the JDK.</li>
+</ul>
+</div>
+<div class="section" id="source-and-sink-batch-sizes-and-channel-transaction-capacities">
+<h3>Source and sink batch sizes and channel transaction capacities<a class="headerlink" href="#source-and-sink-batch-sizes-and-channel-transaction-capacities" title="Permalink to this headline">¶</a></h3>
+<p>Sources and sinks can have a batch size parameter that determines the maximum number of events they
+process in one batch. This happens within a channel transaction that has an upper limit called
+transaction capacity. Batch size must be smaller than the channel&#8217;s transaction capacity.
+There is an explicit check to prevent incompatible settings. This check happens
+whenever the configuration is read.</p>
+</div>
 <div class="section" id="flume-sources">
 <h3>Flume Sources<a class="headerlink" href="#flume-sources" title="Permalink to this headline">¶</a></h3>
 <div class="section" id="avro-source">
@@ -725,9 +889,9 @@ it can create tiered collection topologi
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
+<col width="14%" />
 <col width="11%" />
-<col width="10%" />
-<col width="78%" />
+<col width="75%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -778,29 +942,55 @@ Required properties are in <strong>bold<
 </tr>
 <tr class="row-even"><td>ssl</td>
 <td>false</td>
-<td>Set this to true to enable SSL encryption. You must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;.</td>
+<td>Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;,
+either through component level parameters (see below)
+or as global SSL parameters (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).</td>
 </tr>
 <tr class="row-odd"><td>keystore</td>
 <td>&#8211;</td>
-<td>This is the path to a Java keystore file. Required for SSL.</td>
+<td>This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</td>
 </tr>
 <tr class="row-even"><td>keystore-password</td>
 <td>&#8211;</td>
-<td>The password for the Java keystore. Required for SSL.</td>
+<td>The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</td>
 </tr>
 <tr class="row-odd"><td>keystore-type</td>
 <td>JKS</td>
-<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.</td>
+<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</td>
 </tr>
 <tr class="row-even"><td>exclude-protocols</td>
 <td>SSLv3</td>
-<td>Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols specified.</td>
+<td>Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</td>
+</tr>
+<tr class="row-odd"><td>include-protocols</td>
+<td>&#8211;</td>
+<td>Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</td>
 </tr>
-<tr class="row-odd"><td>ipFilter</td>
+<tr class="row-even"><td>exclude-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to exclude.</td>
+</tr>
+<tr class="row-odd"><td>include-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</td>
+</tr>
+<tr class="row-even"><td>ipFilter</td>
 <td>false</td>
 <td>Set this to true to enable ipFiltering for netty</td>
 </tr>
-<tr class="row-even"><td>ipFilterRules</td>
+<tr class="row-odd"><td>ipFilterRules</td>
 <td>&#8211;</td>
 <td>Define N netty ipFilter pattern rules with this config.</td>
 </tr>
@@ -836,7 +1026,7 @@ Thrift source to authenticate to the ker
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="5%" />
+<col width="6%" />
 <col width="3%" />
 <col width="91%" />
 </colgroup>
@@ -885,33 +1075,58 @@ Required properties are in <strong>bold<
 </tr>
 <tr class="row-odd"><td>ssl</td>
 <td>false</td>
-<td>Set this to true to enable SSL encryption. You must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;.</td>
+<td>Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;,
+either through component level parameters (see below)
+or as global SSL parameters (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section)</td>
 </tr>
 <tr class="row-even"><td>keystore</td>
 <td>&#8211;</td>
-<td>This is the path to a Java keystore file. Required for SSL.</td>
+<td>This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</td>
 </tr>
 <tr class="row-odd"><td>keystore-password</td>
 <td>&#8211;</td>
-<td>The password for the Java keystore. Required for SSL.</td>
+<td>The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</td>
 </tr>
 <tr class="row-even"><td>keystore-type</td>
 <td>JKS</td>
-<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.</td>
+<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</td>
 </tr>
 <tr class="row-odd"><td>exclude-protocols</td>
 <td>SSLv3</td>
-<td>Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols specified.</td>
+<td>Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</td>
+</tr>
+<tr class="row-even"><td>include-protocols</td>
+<td>&#8211;</td>
+<td>Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</td>
+</tr>
+<tr class="row-odd"><td>exclude-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to exclude.</td>
+</tr>
+<tr class="row-even"><td>include-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.</td>
 </tr>
-<tr class="row-even"><td>kerberos</td>
+<tr class="row-odd"><td>kerberos</td>
 <td>false</td>
 <td>Set to true to enable kerberos authentication. In kerberos mode, agent-principal and agent-keytab  are required for successful authentication. The Thrift source in secure mode, will accept connections only from Thrift clients that have kerberos enabled and are successfully authenticated to the kerberos KDC.</td>
 </tr>
-<tr class="row-odd"><td>agent-principal</td>
+<tr class="row-even"><td>agent-principal</td>
 <td>&#8211;</td>
 <td>The kerberos principal used by the Thrift Source to authenticate to the kerberos KDC.</td>
 </tr>
-<tr class="row-even"><td>agent-keytab</td>
+<tr class="row-odd"><td>agent-keytab</td>
 <td>—-</td>
 <td>The keytab location used by the Thrift Source in combination with the agent-principal to authenticate to the kerberos KDC.</td>
 </tr>
@@ -1139,8 +1354,8 @@ Required for durable subscriptions.</td>
 </tr>
 </tbody>
 </table>
-<div class="section" id="converter">
-<h5>Converter<a class="headerlink" href="#converter" title="Permalink to this headline">¶</a></h5>
+<div class="section" id="jms-message-converter">
+<h5>JMS message converter<a class="headerlink" href="#jms-message-converter" title="Permalink to this headline">¶</a></h5>
 <p>The JMS source allows pluggable converters, though it&#8217;s likely the default converter will work
 for most purposes. The default converter is able to convert Bytes, Text, and Object messages
 to FlumeEvents. In all cases, the properties in the message are added as headers to the
@@ -1169,6 +1384,39 @@ the resulting array is copied to the bod
 </pre></div>
 </div>
 </div>
+<div class="section" id="ssl-and-jms-source">
+<h5>SSL and JMS Source<a class="headerlink" href="#ssl-and-jms-source" title="Permalink to this headline">¶</a></h5>
+<p>JMS client implementations typically support to configure SSL/TLS via some Java system properties defined by JSSE
+(Java Secure Socket Extension). Specifying these system properties for Flume&#8217;s JVM, JMS Source (or more precisely the
+JMS client implementation used by the JMS Source) can connect to the JMS server through SSL (of course only when the JMS
+server has also been set up to use SSL).
+It should work with any JMS provider and has been tested with ActiveMQ, IBM MQ and Oracle WebLogic.</p>
+<p>The following sections describe the SSL configuration steps needed on the Flume side only. You can find more detailed
+descriptions about the server side setup of the different JMS providers and also full working configuration examples on
+Flume Wiki.</p>
+<p><strong>SSL transport / server authentication:</strong></p>
+<p>If the JMS server uses self-signed certificate or its certificate is signed by a non-trusted CA (eg. the company&#8217;s own
+CA), then a truststore (containing the right certificate) needs to be set up and passed to Flume. It can be done via
+the global SSL parameters. For more details about the global SSL setup, see the <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section.</p>
+<p>Some JMS providers require SSL specific JNDI Initial Context Factory and/or Provider URL settings when using SSL (eg.
+ActiveMQ uses ssl:// URL prefix instead of tcp://).
+In this case the source properties (<tt class="docutils literal"><span class="pre">initialContextFactory</span></tt> and/or <tt class="docutils literal"><span class="pre">providerURL</span></tt>) have to be adjusted in the agent
+config file.</p>
+<p><strong>Client certificate authentication (two-way SSL):</strong></p>
+<p>JMS Source can authenticate to the JMS server through client certificate authentication instead of the usual
+user/password login (when SSL is used and the JMS server is configured to accept this kind of authentication).</p>
+<p>The keystore containing Flume&#8217;s key used for the authentication needs to be configured via the global SSL parameters
+again. For more details about the global SSL setup, see the <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section.</p>
+<p>The keystore should contain only one key (if multiple keys are present, then the first one will be used).
+The key password must be the same as the keystore password.</p>
+<p>In case of client certificate authentication, it is not needed to specify the <tt class="docutils literal"><span class="pre">userName</span></tt> / <tt class="docutils literal"><span class="pre">passwordFile</span></tt> properties
+for the JMS Source in the Flume agent config file.</p>
+<p><strong>Please note:</strong></p>
+<p>There are no component level configuration parameters for JMS Source unlike in case of other components.
+No enable SSL flag either.
+SSL setup is controlled by JNDI/Provider URL settings (ultimately the JMS server settings) and by the presence / absence
+of the truststore / keystore.</p>
+</div>
 </div>
 <div class="section" id="spooling-directory-source">
 <h4>Spooling Directory Source<a class="headerlink" href="#spooling-directory-source" title="Permalink to this headline">¶</a></h4>
@@ -1178,7 +1426,8 @@ This source will watch the specified dir
 events out of new files as they appear.
 The event parsing logic is pluggable.
 After a given file has been fully read
-into the channel, it is renamed to indicate completion (or optionally deleted).</p>
+into the channel, completion by default is indicated by renaming the file or it can be deleted or the trackerDir is used
+to keep track of processed files.</p>
 <p>Unlike the Exec source, this source is reliable and will not miss data, even if
 Flume is restarted or killed. In exchange for this reliability, only immutable,
 uniquely-named files must be dropped into the spooling directory. Flume tries
@@ -1263,7 +1512,15 @@ the file is ignored.</td>
 <td>Directory to store metadata related to processing of files.
 If this path is not an absolute path, then it is interpreted as relative to the spoolDir.</td>
 </tr>
-<tr class="row-even"><td>consumeOrder</td>
+<tr class="row-even"><td>trackingPolicy</td>
+<td>rename</td>
+<td>The tracking policy defines how file processing is tracked. It can be &#8220;rename&#8221; or
+&#8220;tracker_dir&#8221;. This parameter is only effective if the deletePolicy is &#8220;never&#8221;.
+&#8220;rename&#8221; - After processing files they get renamed according to the fileSuffix parameter.
+&#8220;tracker_dir&#8221; - Files are not renamed but a new empty file is created in the trackerDir.
+The new tracker file name is derived from the ingested one plus the fileSuffix.</td>
+</tr>
+<tr class="row-odd"><td>consumeOrder</td>
 <td>oldest</td>
 <td>In which order files in the spooling directory will be consumed <tt class="docutils literal"><span class="pre">oldest</span></tt>,
 <tt class="docutils literal"><span class="pre">youngest</span></tt> and <tt class="docutils literal"><span class="pre">random</span></tt>. In case of <tt class="docutils literal"><span class="pre">oldest</span></tt> and <tt class="docutils literal"><span class="pre">youngest</span></tt>, the last modified
@@ -1274,30 +1531,30 @@ directory will be scanned to pick the ol
 are a large number of files, while using <tt class="docutils literal"><span class="pre">random</span></tt> may cause old files to be consumed
 very late if new files keep coming in the spooling directory.</td>
 </tr>
-<tr class="row-odd"><td>pollDelay</td>
+<tr class="row-even"><td>pollDelay</td>
 <td>500</td>
 <td>Delay (in milliseconds) used when polling for new files.</td>
 </tr>
-<tr class="row-even"><td>recursiveDirectorySearch</td>
+<tr class="row-odd"><td>recursiveDirectorySearch</td>
 <td>false</td>
 <td>Whether to monitor sub directories for new files to read.</td>
 </tr>
-<tr class="row-odd"><td>maxBackoff</td>
+<tr class="row-even"><td>maxBackoff</td>
 <td>4000</td>
 <td>The maximum time (in millis) to wait between consecutive attempts to
 write to the channel(s) if the channel is full. The source will start at
 a low backoff and increase it exponentially each time the channel throws a
 ChannelException, upto the value specified by this parameter.</td>
 </tr>
-<tr class="row-even"><td>batchSize</td>
+<tr class="row-odd"><td>batchSize</td>
 <td>100</td>
 <td>Granularity at which to batch transfer to the channel</td>
 </tr>
-<tr class="row-odd"><td>inputCharset</td>
+<tr class="row-even"><td>inputCharset</td>
 <td>UTF-8</td>
 <td>Character set used by deserializers that treat the input file as text.</td>
 </tr>
-<tr class="row-even"><td>decodeErrorPolicy</td>
+<tr class="row-odd"><td>decodeErrorPolicy</td>
 <td><tt class="docutils literal"><span class="pre">FAIL</span></tt></td>
 <td>What to do when we see a non-decodable character in the input file.
 <tt class="docutils literal"><span class="pre">FAIL</span></tt>: Throw an exception and fail to parse the file.
@@ -1305,37 +1562,37 @@ ChannelException, upto the value specifi
 typically Unicode U+FFFD.
 <tt class="docutils literal"><span class="pre">IGNORE</span></tt>: Drop the unparseable character sequence.</td>
 </tr>
-<tr class="row-odd"><td>deserializer</td>
+<tr class="row-even"><td>deserializer</td>
 <td><tt class="docutils literal"><span class="pre">LINE</span></tt></td>
 <td>Specify the deserializer used to parse the file into events.
 Defaults to parsing each line as an event. The class specified must implement
 <tt class="docutils literal"><span class="pre">EventDeserializer.Builder</span></tt>.</td>
 </tr>
-<tr class="row-even"><td>deserializer.*</td>
+<tr class="row-odd"><td>deserializer.*</td>
 <td>&nbsp;</td>
 <td>Varies per event deserializer.</td>
 </tr>
-<tr class="row-odd"><td>bufferMaxLines</td>
+<tr class="row-even"><td>bufferMaxLines</td>
 <td>&#8211;</td>
 <td>(Obselete) This option is now ignored.</td>
 </tr>
-<tr class="row-even"><td>bufferMaxLineLength</td>
+<tr class="row-odd"><td>bufferMaxLineLength</td>
 <td>5000</td>
 <td>(Deprecated) Maximum length of a line in the commit buffer. Use deserializer.maxLineLength instead.</td>
 </tr>
-<tr class="row-odd"><td>selector.type</td>
+<tr class="row-even"><td>selector.type</td>
 <td>replicating</td>
 <td>replicating or multiplexing</td>
 </tr>
-<tr class="row-even"><td>selector.*</td>
+<tr class="row-odd"><td>selector.*</td>
 <td>&nbsp;</td>
 <td>Depends on the selector.type value</td>
 </tr>
-<tr class="row-odd"><td>interceptors</td>
+<tr class="row-even"><td>interceptors</td>
 <td>&#8211;</td>
 <td>Space-separated list of interceptors</td>
 </tr>
-<tr class="row-even"><td>interceptors.*</td>
+<tr class="row-odd"><td>interceptors.*</td>
 <td>&nbsp;</td>
 <td>&nbsp;</td>
 </tr>
@@ -1524,26 +1781,33 @@ Currently this source does not support t
 <td>100</td>
 <td>Max number of lines to read and send to the channel at a time. Using the default is usually fine.</td>
 </tr>
-<tr class="row-odd"><td>backoffSleepIncrement</td>
+<tr class="row-odd"><td>maxBatchCount</td>
+<td>Long.MAX_VALUE</td>
+<td>Controls the number of batches being read consecutively from the same file.
+If the source is tailing multiple files and one of them is written at a fast rate,
+it can prevent other files to be processed, because the busy file would be read in an endless loop.
+In this case lower this value.</td>
+</tr>
+<tr class="row-even"><td>backoffSleepIncrement</td>
 <td>1000</td>
 <td>The increment for time delay before reattempting to poll for new data, when the last attempt did not find any new data.</td>
 </tr>
-<tr class="row-even"><td>maxBackoffSleep</td>
+<tr class="row-odd"><td>maxBackoffSleep</td>
 <td>5000</td>
 <td>The max time delay between each reattempt to poll for new data, when the last attempt did not find any new data.</td>
 </tr>
-<tr class="row-odd"><td>cachePatternMatching</td>
+<tr class="row-even"><td>cachePatternMatching</td>
 <td>true</td>
 <td>Listing directories and applying the filename regex pattern may be time consuming for directories
 containing thousands of files. Caching the list of matching files can improve performance.
 The order in which files are consumed will also be cached.
 Requires that the file system keeps track of modification times with at least a 1-second granularity.</td>
 </tr>
-<tr class="row-even"><td>fileHeader</td>
+<tr class="row-odd"><td>fileHeader</td>
 <td>false</td>
 <td>Whether to add a header storing the absolute path filename.</td>
 </tr>
-<tr class="row-odd"><td>fileHeaderKey</td>
+<tr class="row-even"><td>fileHeaderKey</td>
 <td>file</td>
 <td>Header key to use when appending absolute path filename to event header.</td>
 </tr>
@@ -1562,6 +1826,7 @@ Requires that the file system keeps trac
 <span class="na">a1.sources.r1.headers.f2.headerKey1</span> <span class="o">=</span> <span class="s">value2</span>
 <span class="na">a1.sources.r1.headers.f2.headerKey2</span> <span class="o">=</span> <span class="s">value2-2</span>
 <span class="na">a1.sources.r1.fileHeader</span> <span class="o">=</span> <span class="s">true</span>
+<span class="na">a1.sources.ri.maxBatchCount</span> <span class="o">=</span> <span class="s">1000</span>
 </pre></div>
 </div>
 </div>
@@ -1642,7 +1907,7 @@ Required properties are in <strong>bold<
 <h4>Kafka Source<a class="headerlink" href="#kafka-source" title="Permalink to this headline">¶</a></h4>
 <p>Kafka Source is an Apache Kafka consumer that reads messages from Kafka topics.
 If you have multiple Kafka sources running, you can configure them with the same Consumer Group
-so each will read a unique set of partitions for the topics.</p>
+so each will read a unique set of partitions for the topics. This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
 <table border="1" class="docutils">
 <colgroup>
 <col width="19%" />
@@ -1723,25 +1988,16 @@ from, if the <tt class="docutils literal
 with the Kafka Sink <tt class="docutils literal"><span class="pre">topicHeader</span></tt> property so as to avoid sending the message back to the same
 topic in a loop.</td>
 </tr>
-<tr class="row-odd"><td>migrateZookeeperOffsets</td>
-<td>true</td>
-<td>When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
-This should be true to support seamless Kafka client migration from older versions of Flume.
-Once migrated this can be set to false, though that should generally not be required.
-If no Zookeeper offset is found, the Kafka configuration kafka.consumer.auto.offset.reset
-defines how offsets are handled.
-Check <a class="reference external" href="http://kafka.apache.org/documentation.html#newconsumerconfigs">Kafka documentation</a> for details</td>
-</tr>
-<tr class="row-even"><td>kafka.consumer.security.protocol</td>
+<tr class="row-odd"><td>kafka.consumer.security.protocol</td>
 <td>PLAINTEXT</td>
 <td>Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.</td>
 </tr>
-<tr class="row-odd"><td><em>more consumer security props</em></td>
+<tr class="row-even"><td><em>more consumer security props</em></td>
 <td>&nbsp;</td>
 <td>If using SASL_PLAINTEXT, SASL_SSL or SSL refer to <a class="reference external" href="http://kafka.apache.org/documentation.html#security">Kafka security</a> for additional
 properties that need to be set on consumer.</td>
 </tr>
-<tr class="row-even"><td>Other Kafka Consumer Properties</td>
+<tr class="row-odd"><td>Other Kafka Consumer Properties</td>
 <td>&#8211;</td>
 <td>These properties are used to configure the Kafka Consumer. Any consumer property supported
 by Kafka can be used. The only requirement is to prepend the property name with the prefix
@@ -1761,9 +2017,9 @@ and value.deserializer(org.apache.kafka.
 <p>Deprecated Properties</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="22%" />
+<col width="21%" />
 <col width="13%" />
-<col width="65%" />
+<col width="66%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -1785,6 +2041,16 @@ and value.deserializer(org.apache.kafka.
 <td>Is no longer supported by kafka consumer client since 0.9.x. Use kafka.bootstrap.servers
 to establish connection with kafka cluster</td>
 </tr>
+<tr class="row-odd"><td>migrateZookeeperOffsets</td>
+<td>true</td>
+<td>When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+This should be true to support seamless Kafka client migration from older versions of Flume.
+Once migrated this can be set to false, though that should generally not be required.
+If no Zookeeper offset is found, the Kafka configuration kafka.consumer.auto.offset.reset
+defines how offsets are handled.
+Check <a class="reference external" href="http://kafka.apache.org/documentation.html#newconsumerconfigs">Kafka documentation</a>
+for details</td>
+</tr>
 </tbody>
 </table>
 <p>Example for topic subscription by comma-separated topic list.</p>
@@ -1833,10 +2099,13 @@ security provider, cipher suites, enable
 <span class="na">a1.sources.source1.kafka.topics</span> <span class="o">=</span> <span class="s">mytopic</span>
 <span class="na">a1.sources.source1.kafka.consumer.group.id</span> <span class="o">=</span> <span class="s">flume-consumer</span>
 <span class="na">a1.sources.source1.kafka.consumer.security.protocol</span> <span class="o">=</span> <span class="s">SSL</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.sources.source1.kafka.consumer.ssl.truststore.location</span><span class="o">=</span><span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.sources.source1.kafka.consumer.ssl.truststore.password</span><span class="o">=</span><span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
 </div>
+<p>Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section.</p>
 <p>Note: By default the property <tt class="docutils literal"><span class="pre">ssl.endpoint.identification.algorithm</span></tt>
 is not defined, so hostname verification is not performed.
 In order to enable hostname verification, set the following properties</p>
@@ -1849,11 +2118,13 @@ against one of the following two fields:
 <li>Common Name (CN) <a class="reference external" href="https://tools.ietf.org/html/rfc6125#section-2.3">https://tools.ietf.org/html/rfc6125#section-2.3</a></li>
 <li>Subject Alternative Name (SAN) <a class="reference external" href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></li>
 </ol>
-<p>If client side authentication is also required then additionally the following should be added to Flume agent configuration.
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).
 Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
 individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
 which in turn is trusted by Kafka brokers.</p>
-<div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.sources.source1.kafka.consumer.ssl.keystore.location</span><span class="o">=</span><span class="s">/path/to/client.keystore.jks</span>
+<div class="highlight-properties"><div class="highlight"><pre><span class="c"># optional, the global keystore can be used alternatively</span>
+<span class="na">a1.sources.source1.kafka.consumer.ssl.keystore.location</span><span class="o">=</span><span class="s">/path/to/client.keystore.jks</span>
 <span class="na">a1.sources.source1.kafka.consumer.ssl.keystore.password</span><span class="o">=</span><span class="s">&lt;password to access the keystore&gt;</span>
 </pre></div>
 </div>
@@ -1889,6 +2160,7 @@ for information on the JAAS file content
 <span class="na">a1.sources.source1.kafka.consumer.security.protocol</span> <span class="o">=</span> <span class="s">SASL_SSL</span>
 <span class="na">a1.sources.source1.kafka.consumer.sasl.mechanism</span> <span class="o">=</span> <span class="s">GSSAPI</span>
 <span class="na">a1.sources.source1.kafka.consumer.sasl.kerberos.service.name</span> <span class="o">=</span> <span class="s">kafka</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.sources.source1.kafka.consumer.ssl.truststore.location</span><span class="o">=</span><span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.sources.source1.kafka.consumer.ssl.truststore.password</span><span class="o">=</span><span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
@@ -2129,9 +2401,9 @@ of characters separated by a newline (&#
 <p>The original, tried-and-true syslog TCP source.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="19%" />
-<col width="15%" />
-<col width="67%" />
+<col width="16%" />
+<col width="9%" />
+<col width="75%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -2170,6 +2442,26 @@ fields can be included: priority, versio
 timestamp, hostname. The values &#8216;true&#8217; and &#8216;false&#8217;
 have been deprecated in favor of &#8216;all&#8217; and &#8216;none&#8217;.</td>
 </tr>
+<tr class="row-even"><td>clientIPHeader</td>
+<td>&#8211;</td>
+<td>If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
+<tr class="row-odd"><td>clientHostnameHeader</td>
+<td>&#8211;</td>
+<td>If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
 <tr class="row-even"><td>selector.type</td>
 <td>&nbsp;</td>
 <td>replicating or multiplexing</td>
@@ -2186,6 +2478,52 @@ have been deprecated in favor of &#8216;
 <td>&nbsp;</td>
 <td>&nbsp;</td>
 </tr>
+<tr class="row-even"><td>ssl</td>
+<td>false</td>
+<td>Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;,
+either through component level parameters (see below)
+or as global SSL parameters (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).</td>
+</tr>
+<tr class="row-odd"><td>keystore</td>
+<td>&#8211;</td>
+<td>This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-even"><td>keystore-password</td>
+<td>&#8211;</td>
+<td>The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-odd"><td>keystore-type</td>
+<td>JKS</td>
+<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</td>
+</tr>
+<tr class="row-even"><td>exclude-protocols</td>
+<td>SSLv3</td>
+<td>Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</td>
+</tr>
+<tr class="row-odd"><td>include-protocols</td>
+<td>&#8211;</td>
+<td>Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</td>
+</tr>
+<tr class="row-even"><td>exclude-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to exclude.</td>
+</tr>
+<tr class="row-odd"><td>include-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</td>
+</tr>
 </tbody>
 </table>
 <p>For example, a syslog TCP source for agent named a1:</p>
@@ -2209,9 +2547,9 @@ Also provides the capability to configur
 basis.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="7%" />
+<col width="8%" />
 <col width="6%" />
-<col width="87%" />
+<col width="86%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -2254,6 +2592,26 @@ have been deprecated in favor of &#8216;
 <td>&#8211;</td>
 <td>If specified, the port number will be stored in the header of each event using the header name specified here. This allows for interceptors and channel selectors to customize routing logic based on the incoming port.</td>
 </tr>
+<tr class="row-odd"><td>clientIPHeader</td>
+<td>&#8211;</td>
+<td>If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
+<tr class="row-even"><td>clientHostnameHeader</td>
+<td>&#8211;</td>
+<td>If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
 <tr class="row-odd"><td>charset.default</td>
 <td>UTF-8</td>
 <td>Default character set used while parsing syslog events into strings.</td>
@@ -2290,6 +2648,52 @@ have been deprecated in favor of &#8216;
 <td>&nbsp;</td>
 <td>&nbsp;</td>
 </tr>
+<tr class="row-even"><td>ssl</td>
+<td>false</td>
+<td>Set this to true to enable SSL encryption. If SSL is enabled,
+you must also specify a &#8220;keystore&#8221; and a &#8220;keystore-password&#8221;,
+either through component level parameters (see below)
+or as global SSL parameters (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).</td>
+</tr>
+<tr class="row-odd"><td>keystore</td>
+<td>&#8211;</td>
+<td>This is the path to a Java keystore file.
+If not specified here, then the global keystore will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-even"><td>keystore-password</td>
+<td>&#8211;</td>
+<td>The password for the Java keystore.
+If not specified here, then the global keystore password will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-odd"><td>keystore-type</td>
+<td>JKS</td>
+<td>The type of the Java keystore. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.
+If not specified here, then the global keystore type will be used
+(if defined, otherwise the default is JKS).</td>
+</tr>
+<tr class="row-even"><td>exclude-protocols</td>
+<td>SSLv3</td>
+<td>Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</td>
+</tr>
+<tr class="row-odd"><td>include-protocols</td>
+<td>&#8211;</td>
+<td>Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</td>
+</tr>
+<tr class="row-even"><td>exclude-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to exclude.</td>
+</tr>
+<tr class="row-odd"><td>include-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.
+If included-cipher-suites is empty, it includes every supported cipher suites.</td>
+</tr>
 </tbody>
 </table>
 <p>For example, a multiport syslog TCP source for agent named a1:</p>
@@ -2307,8 +2711,8 @@ have been deprecated in favor of &#8216;
 <h5>Syslog UDP Source<a class="headerlink" href="#syslog-udp-source" title="Permalink to this headline">¶</a></h5>
 <table border="1" class="docutils">
 <colgroup>
-<col width="19%" />
-<col width="15%" />
+<col width="21%" />
+<col width="12%" />
 <col width="67%" />
 </colgroup>
 <thead valign="bottom">
@@ -2339,6 +2743,26 @@ have been deprecated in favor of &#8216;
 <td>Setting this to true will preserve the Priority,
 Timestamp and Hostname in the body of the event.</td>
 </tr>
+<tr class="row-odd"><td>clientIPHeader</td>
+<td>&#8211;</td>
+<td>If specified, the IP address of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the IP address of the client.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
+<tr class="row-even"><td>clientHostnameHeader</td>
+<td>&#8211;</td>
+<td>If specified, the host name of the client will be stored in
+the header of each event using the header name specified here.
+This allows for interceptors and channel selectors to customize
+routing logic based on the host name of the client.
+Retrieving the host name may involve a name service reverse
+lookup which may affect the performance.
+Do not use the standard Syslog header names here (like _host_)
+because the event header will be overridden in that case.</td>
+</tr>
 <tr class="row-odd"><td>selector.type</td>
 <td>&nbsp;</td>
 <td>replicating or multiplexing</td>
@@ -2382,11 +2806,13 @@ append events to the channel, the source
 unavailable status.</p>
 <p>All events sent in one post request are considered to be one batch and
 inserted into the channel in one transaction.</p>
+<p>This source is based on Jetty 9.4 and offers the ability to set additional
+Jetty-specific parameters which will be passed directly to the Jetty components.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="12%" />
-<col width="30%" />
-<col width="58%" />
+<col width="13%" />
+<col width="27%" />
+<col width="60%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -2431,23 +2857,104 @@ inserted into the channel in one transac
 <td>&nbsp;</td>
 <td>&nbsp;</td>
 </tr>
-<tr class="row-odd"><td>enableSSL</td>
+<tr class="row-odd"><td>ssl</td>
 <td>false</td>
 <td>Set the property true, to enable SSL. <em>HTTP Source does not support SSLv3.</em></td>
 </tr>
-<tr class="row-even"><td>excludeProtocols</td>
+<tr class="row-even"><td>exclude-protocols</td>
 <td>SSLv3</td>
-<td>Space-separated list of SSL/TLS protocols to exclude. SSLv3 is always excluded.</td>
+<td>Space-separated list of SSL/TLS protocols to exclude.
+SSLv3 will always be excluded in addition to the protocols specified.</td>
 </tr>
-<tr class="row-odd"><td>keystore</td>
+<tr class="row-odd"><td>include-protocols</td>
+<td>&#8211;</td>
+<td>Space-separated list of SSL/TLS protocols to include.
+The enabled protocols will be the included protocols without the excluded protocols.
+If included-protocols is empty, it includes every supported protocols.</td>
+</tr>
+<tr class="row-even"><td>exclude-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to exclude.</td>
+</tr>
+<tr class="row-odd"><td>include-cipher-suites</td>
+<td>&#8211;</td>
+<td>Space-separated list of cipher suites to include.
+The enabled cipher suites will be the included cipher suites without the excluded cipher suites.</td>
+</tr>
+<tr class="row-even"><td>keystore</td>
+<td>&nbsp;</td>
+<td>Location of the keystore including keystore file name.
+If SSL is enabled but the keystore is not specified here,
+then the global keystore will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-odd"><td>keystore-password</td>
+<td>&nbsp;</td>
+<td>Keystore password.
+If SSL is enabled but the keystore password is not specified here,
+then the global keystore password will be used
+(if defined, otherwise configuration error).</td>
+</tr>
+<tr class="row-even"><td>keystore-type</td>
+<td>JKS</td>
+<td>Keystore type. This can be &#8220;JKS&#8221; or &#8220;PKCS12&#8221;.</td>
+</tr>
+<tr class="row-odd"><td>QueuedThreadPool.*</td>
+<td>&nbsp;</td>
+<td>Jetty specific settings to be set on org.eclipse.jetty.util.thread.QueuedThreadPool.
+N.B. QueuedThreadPool will only be used if at least one property of this class is set.</td>
+</tr>
+<tr class="row-even"><td>HttpConfiguration.*</td>
+<td>&nbsp;</td>
+<td>Jetty specific settings to be set on org.eclipse.jetty.server.HttpConfiguration</td>
+</tr>
+<tr class="row-odd"><td>SslContextFactory.*</td>
+<td>&nbsp;</td>
+<td>Jetty specific settings to be set on org.eclipse.jetty.util.ssl.SslContextFactory (only
+applicable when <em>ssl</em> is set to true).</td>
+</tr>
+<tr class="row-even"><td>ServerConnector.*</td>
 <td>&nbsp;</td>
-<td>Location of the keystore includng keystore file name</td>
+<td>Jetty specific settings to be set on org.eclipse.jetty.server.ServerConnector</td>
+</tr>
+</tbody>
+</table>
+<p>Deprecated Properties</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="22%" />
+<col width="13%" />
+<col width="65%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">Property Name</th>
+<th class="head">Default</th>
+<th class="head">Description</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td>keystorePassword</td>
+<td>&#8211;</td>
+<td>Use <em>keystore-password</em>. Deprecated value will be overwritten with the new one.</td>
+</tr>
+<tr class="row-odd"><td>excludeProtocols</td>
+<td>SSLv3</td>
+<td>Use <em>exclude-protocols</em>. Deprecated value will be overwritten with the new one.</td>
 </tr>
-<tr class="row-even"><td colspan="3">keystorePassword                                                 Keystore password</td>
+<tr class="row-even"><td>enableSSL</td>
+<td>false</td>
+<td>Use <em>ssl</em>. Deprecated value will be overwritten with the new one.</td>
 </tr>
 </tbody>
 </table>
-<p>For example, a http source for agent named a1:</p>
+<p>N.B. Jetty-specific settings are set using the setter-methods on the objects listed above. For full details see the Javadoc for these classes
+(<a class="reference external" href="http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/util/thread/QueuedThreadPool.html">QueuedThreadPool</a>,
+<a class="reference external" href="http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/server/HttpConfiguration.html">HttpConfiguration</a>,
+<a class="reference external" href="http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/util/ssl/SslContextFactory.html">SslContextFactory</a> and
+<a class="reference external" href="http://www.eclipse.org/jetty/javadoc/9.4.6.v20170531/org/eclipse/jetty/server/ServerConnector.html">ServerConnector</a>).</p>
+<p>When using Jetty-specific setings, named properites above will take precedence (for example excludeProtocols will take
+precedence over SslContextFactory.ExcludeProtocols). All properties will be inital lower case.</p>
+<p>An example http source for agent named a1:</p>
 <div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.sources</span> <span class="o">=</span> <span class="s">r1</span>
 <span class="na">a1.channels</span> <span class="o">=</span> <span class="s">c1</span>
 <span class="na">a1.sources.r1.type</span> <span class="o">=</span> <span class="s">http</span>
@@ -2455,6 +2962,8 @@ inserted into the channel in one transac
 <span class="na">a1.sources.r1.channels</span> <span class="o">=</span> <span class="s">c1</span>
 <span class="na">a1.sources.r1.handler</span> <span class="o">=</span> <span class="s">org.example.rest.RestHandler</span>
 <span class="na">a1.sources.r1.handler.nickname</span> <span class="o">=</span> <span class="s">random props</span>
+<span class="na">a1.sources.r1.HttpConfiguration.sendServerVersion</span> <span class="o">=</span> <span class="s">false</span>
+<span class="na">a1.sources.r1.ServerConnector.idleTimeout</span> <span class="o">=</span> <span class="s">300</span>
 </pre></div>
 </div>
 <div class="section" id="jsonhandler">
@@ -2531,9 +3040,9 @@ Event to be delivered.</p>
 <p>Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="18%" />
+<col width="17%" />
 <col width="10%" />
-<col width="72%" />
+<col width="73%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -2562,6 +3071,10 @@ Event to be delivered.</p>
 <td>1</td>
 <td>Number of Events to be sent in one batch</td>
 </tr>
+<tr class="row-odd"><td>maxEventsPerSecond</td>
+<td>0</td>
+<td>When set to an integer greater than zero, enforces a rate limiter onto the source.</td>
+</tr>
 </tbody>
 </table>
 <p>Example for agent named <strong>a1</strong>:</p>
@@ -2942,9 +3455,9 @@ this automatically is to use the Timesta
 </div>
 <table border="1" class="docutils">
 <colgroup>
-<col width="9%" />
-<col width="5%" />
-<col width="86%" />
+<col width="8%" />
+<col width="4%" />
+<col width="88%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Name</th>
@@ -2981,56 +3494,55 @@ this automatically is to use the Timesta
 <td><tt class="docutils literal"><span class="pre">.tmp</span></tt></td>
 <td>Suffix that is used for temporal files that flume actively writes into</td>
 </tr>
-<tr class="row-odd"><td>hdfs.rollInterval</td>
+<tr class="row-odd"><td>hdfs.emptyInUseSuffix</td>
+<td>false</td>
+<td>If <tt class="docutils literal"><span class="pre">false</span></tt> an <tt class="docutils literal"><span class="pre">hdfs.inUseSuffix</span></tt> is used while writing the output. After closing the output <tt class="docutils literal"><span class="pre">hdfs.inUseSuffix</span></tt> is removed from the output file name. If <tt class="docutils literal"><span class="pre">true</span></tt> the <tt class="docutils literal"><span class="pre">hdfs.inUseSuffix</span></tt> parameter is ignored an empty string is used instead.</td>
+</tr>
+<tr class="row-even"><td>hdfs.rollInterval</td>
 <td>30</td>
 <td>Number of seconds to wait before rolling current file
 (0 = never roll based on time interval)</td>
 </tr>
-<tr class="row-even"><td>hdfs.rollSize</td>
+<tr class="row-odd"><td>hdfs.rollSize</td>
 <td>1024</td>
 <td>File size to trigger roll, in bytes (0: never roll based on file size)</td>
 </tr>
-<tr class="row-odd"><td>hdfs.rollCount</td>
+<tr class="row-even"><td>hdfs.rollCount</td>
 <td>10</td>
 <td>Number of events written to file before it rolled
 (0 = never roll based on number of events)</td>
 </tr>
-<tr class="row-even"><td>hdfs.idleTimeout</td>
+<tr class="row-odd"><td>hdfs.idleTimeout</td>
 <td>0</td>
 <td>Timeout after which inactive files get closed
 (0 = disable automatic closing of idle files)</td>
 </tr>
-<tr class="row-odd"><td>hdfs.batchSize</td>
+<tr class="row-even"><td>hdfs.batchSize</td>
 <td>100</td>
 <td>number of events written to file before it is flushed to HDFS</td>
 </tr>
-<tr class="row-even"><td>hdfs.codeC</td>
+<tr class="row-odd"><td>hdfs.codeC</td>
 <td>&#8211;</td>
 <td>Compression codec. one of following : gzip, bzip2, lzo, lzop, snappy</td>
 </tr>
-<tr class="row-odd"><td>hdfs.fileType</td>
+<tr class="row-even"><td>hdfs.fileType</td>
 <td>SequenceFile</td>
 <td>File format: currently <tt class="docutils literal"><span class="pre">SequenceFile</span></tt>, <tt class="docutils literal"><span class="pre">DataStream</span></tt> or <tt class="docutils literal"><span class="pre">CompressedStream</span></tt>
 (1)DataStream will not compress output file and please don&#8217;t set codeC
 (2)CompressedStream requires set hdfs.codeC with an available codeC</td>
 </tr>
-<tr class="row-even"><td>hdfs.maxOpenFiles</td>
+<tr class="row-odd"><td>hdfs.maxOpenFiles</td>
 <td>5000</td>
 <td>Allow only this number of open files. If this number is exceeded, the oldest file is closed.</td>
 </tr>
-<tr class="row-odd"><td>hdfs.minBlockReplicas</td>
+<tr class="row-even"><td>hdfs.minBlockReplicas</td>
 <td>&#8211;</td>
 <td>Specify minimum number of replicas per HDFS block. If not specified, it comes from the default Hadoop config in the classpath.</td>
 </tr>
-<tr class="row-even"><td>hdfs.writeFormat</td>
+<tr class="row-odd"><td>hdfs.writeFormat</td>
 <td>Writable</td>
 <td>Format for sequence file records. One of <tt class="docutils literal"><span class="pre">Text</span></tt> or <tt class="docutils literal"><span class="pre">Writable</span></tt>. Set to <tt class="docutils literal"><span class="pre">Text</span></tt> before creating data files with Flume, otherwise those files cannot be read by either Apache Impala (incubating) or Apache Hive.</td>
 </tr>
-<tr class="row-odd"><td>hdfs.callTimeout</td>
-<td>10000</td>
-<td>Number of milliseconds allowed for HDFS operations, such as open, write, flush, close.
-This number should be increased if many HDFS timeout operations are occurring.</td>
-</tr>
 <tr class="row-even"><td>hdfs.threadsPoolSize</td>
 <td>10</td>
 <td>Number of threads per HDFS sink for HDFS IO ops (open, write, etc.)</td>
@@ -3096,6 +3608,11 @@ fully-qualified class name of an impleme
 </tr>
 </tbody>
 </table>
+<p>Deprecated Properties</p>
+<p>Name                    Default       Description
+======================  ============  ======================================================================
+hdfs.callTimeout        30000         Number of milliseconds allowed for HDFS operations, such as open, write, flush, close. This number should be increased if many HDFS timeout operations are occurring.
+======================  ============  ======================================================================</p>
 <p>Example for agent named a1:</p>
 <div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.channels</span> <span class="o">=</span> <span class="s">c1</span>
 <span class="na">a1.sinks</span> <span class="o">=</span> <span class="s">k1</span>
@@ -3419,9 +3936,9 @@ batches of the configured batch size.
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="6%" />
-<col width="13%" />
-<col width="81%" />
+<col width="5%" />
+<col width="10%" />
+<col width="84%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -3480,15 +3997,15 @@ Required properties are in <strong>bold<
 </tr>
 <tr class="row-even"><td>truststore</td>
 <td>&#8211;</td>
-<td>The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Avro Source&#8217;s SSL authentication credentials should be trusted. If not specified, the default Java JSSE certificate authority files (typically &#8220;jssecacerts&#8221; or &#8220;cacerts&#8221; in the Oracle JRE) will be used.</td>
+<td>The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Avro Source&#8217;s SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically &#8220;jssecacerts&#8221; or &#8220;cacerts&#8221; in the Oracle JRE) will be used.</td>
 </tr>
 <tr class="row-odd"><td>truststore-password</td>
 <td>&#8211;</td>
-<td>The password for the specified truststore.</td>
+<td>The password for the truststore. If not specified, then the global keystore password will be used (if defined).</td>
 </tr>
 <tr class="row-even"><td>truststore-type</td>
 <td>JKS</td>
-<td>The type of the Java truststore. This can be &#8220;JKS&#8221; or other supported Java truststore type.</td>
+<td>The type of the Java truststore. This can be &#8220;JKS&#8221; or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).</td>
 </tr>
 <tr class="row-odd"><td>exclude-protocols</td>
 <td>SSLv3</td>
@@ -3524,9 +4041,9 @@ principal of the Thrift source this sink
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="7%" />
+<col width="6%" />
 <col width="2%" />
-<col width="91%" />
+<col width="93%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -3573,15 +4090,15 @@ Required properties are in <strong>bold<
 </tr>
 <tr class="row-odd"><td>truststore</td>
 <td>&#8211;</td>
-<td>The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Thrift Source&#8217;s SSL authentication credentials should be trusted. If not specified, the default Java JSSE certificate authority files (typically &#8220;jssecacerts&#8221; or &#8220;cacerts&#8221; in the Oracle JRE) will be used.</td>
+<td>The path to a custom Java truststore file. Flume uses the certificate authority information in this file to determine whether the remote Thrift Source&#8217;s SSL authentication credentials should be trusted. If not specified, then the global keystore will be used. If the global keystore not specified either, then the default Java JSSE certificate authority files (typically &#8220;jssecacerts&#8221; or &#8220;cacerts&#8221; in the Oracle JRE) will be used.</td>
 </tr>
 <tr class="row-even"><td>truststore-password</td>
 <td>&#8211;</td>
-<td>The password for the specified truststore.</td>
+<td>The password for the truststore. If not specified, then the global keystore password will be used (if defined).</td>
 </tr>
 <tr class="row-odd"><td>truststore-type</td>
 <td>JKS</td>
-<td>The type of the Java truststore. This can be &#8220;JKS&#8221; or other supported Java truststore type.</td>
+<td>The type of the Java truststore. This can be &#8220;JKS&#8221; or other supported Java truststore type. If not specified, then the global keystore type will be used (if defined, otherwise the defautl is JKS).</td>
 </tr>
 <tr class="row-even"><td>exclude-protocols</td>
 <td>SSLv3</td>
@@ -3741,7 +4258,7 @@ Required properties are in <strong>bold<
 <td>TEXT</td>
 <td>Other possible options include <tt class="docutils literal"><span class="pre">avro_event</span></tt> or the FQCN of an implementation of EventSerializer.Builder interface.</td>
 </tr>
-<tr class="row-even"><td>batchSize</td>
+<tr class="row-even"><td>sink.batchSize</td>
 <td>100</td>
 <td>&nbsp;</td>
 </tr>
@@ -3896,8 +4413,89 @@ better performance if there are multiple
 </pre></div>
 </div>
 </div>
-<div class="section" id="asynchbasesink">
-<h5>AsyncHBaseSink<a class="headerlink" href="#asynchbasesink" title="Permalink to this headline">¶</a></h5>
+<div class="section" id="hbase2sink">
+<h5>HBase2Sink<a class="headerlink" href="#hbase2sink" title="Permalink to this headline">¶</a></h5>
+<p>HBase2Sink is the equivalent of HBaseSink for HBase version 2.
+The provided functionality and the configuration parameters are the same as in case of HBaseSink (except the hbase2 tag in the sink type and the package/class names).</p>
+<p>The type is the FQCN: org.apache.flume.sink.hbase2.HBase2Sink.</p>
+<p>Required properties are in <strong>bold</strong>.</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="10%" />
+<col width="31%" />
+<col width="58%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">Property Name</th>
+<th class="head">Default</th>
+<th class="head">Description</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td><strong>channel</strong></td>
+<td>&#8211;</td>
+<td>&nbsp;</td>
+</tr>
+<tr class="row-odd"><td><strong>type</strong></td>
+<td>&#8211;</td>
+<td>The component type name, needs to be <tt class="docutils literal"><span class="pre">hbase2</span></tt></td>
+</tr>
+<tr class="row-even"><td><strong>table</strong></td>
+<td>&#8211;</td>
+<td>The name of the table in HBase to write to.</td>
+</tr>
+<tr class="row-odd"><td><strong>columnFamily</strong></td>
+<td>&#8211;</td>
+<td>The column family in HBase to write to.</td>
+</tr>
+<tr class="row-even"><td>zookeeperQuorum</td>
+<td>&#8211;</td>
+<td>The quorum spec. This is the value for the property <tt class="docutils literal"><span class="pre">hbase.zookeeper.quorum</span></tt> in hbase-site.xml</td>
+</tr>
+<tr class="row-odd"><td>znodeParent</td>
+<td>/hbase</td>
+<td>The base path for the znode for the -ROOT- region. Value of <tt class="docutils literal"><span class="pre">zookeeper.znode.parent</span></tt> in hbase-site.xml</td>
+</tr>
+<tr class="row-even"><td>batchSize</td>
+<td>100</td>
+<td>Number of events to be written per txn.</td>
+</tr>
+<tr class="row-odd"><td>coalesceIncrements</td>
+<td>false</td>
+<td>Should the sink coalesce multiple increments to a cell per batch. This might give
+better performance if there are multiple increments to a limited number of cells.</td>
+</tr>
+<tr class="row-even"><td>serializer</td>
+<td>org.apache.flume.sink.hbase2.SimpleHBase2EventSerializer</td>
+<td>Default increment column = &#8220;iCol&#8221;, payload column = &#8220;pCol&#8221;.</td>
+</tr>
+<tr class="row-odd"><td>serializer.*</td>
+<td>&#8211;</td>
+<td>Properties to be passed to the serializer.</td>
+</tr>
+<tr class="row-even"><td>kerberosPrincipal</td>
+<td>&#8211;</td>
+<td>Kerberos user principal for accessing secure HBase</td>
+</tr>
+<tr class="row-odd"><td>kerberosKeytab</td>
+<td>&#8211;</td>
+<td>Kerberos keytab for accessing secure HBase</td>
+</tr>
+</tbody>
+</table>
+<p>Example for agent named a1:</p>
+<div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.channels</span> <span class="o">=</span> <span class="s">c1</span>
+<span class="na">a1.sinks</span> <span class="o">=</span> <span class="s">k1</span>
+<span class="na">a1.sinks.k1.type</span> <span class="o">=</span> <span class="s">hbase2</span>
+<span class="na">a1.sinks.k1.table</span> <span class="o">=</span> <span class="s">foo_table</span>
+<span class="na">a1.sinks.k1.columnFamily</span> <span class="o">=</span> <span class="s">bar_cf</span>
+<span class="na">a1.sinks.k1.serializer</span> <span class="o">=</span> <span class="s">org.apache.flume.sink.hbase2.RegexHBase2EventSerializer</span>
+<span class="na">a1.sinks.k1.channel</span> <span class="o">=</span> <span class="s">c1</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="asynchbasesink">
+<h5>AsyncHBaseSink<a class="headerlink" href="#asynchbasesink" title="Permalink to this headline">¶</a></h5>
 <p>This sink writes data to HBase using an asynchronous model. A class implementing
 AsyncHbaseEventSerializer which is specified by the configuration is used to convert the events into
 HBase puts and/or increments. These puts and increments are then written
@@ -3905,13 +4503,14 @@ to HBase. This sink uses the <a class="r
 HBase. This sink provides the same consistency guarantees as HBase,
 which is currently row-wise atomicity. In the event of Hbase failing to
 write certain events, the sink will replay all events in that transaction.
+AsyncHBaseSink can only be used with HBase 1.x. The async client library used by AsyncHBaseSink is not available for HBase 2.
 The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="10%" />
-<col width="33%" />
-<col width="57%" />
+<col width="9%" />
+<col width="29%" />
+<col width="61%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -3966,6 +4565,13 @@ all events in a transaction.</td>
 <td>&#8211;</td>
 <td>Properties to be passed to the serializer.</td>
 </tr>
+<tr class="row-odd"><td>async.*</td>
+<td>&#8211;</td>
+<td>Properties to be passed to asyncHbase library.
+These properties have precedence over the old <tt class="docutils literal"><span class="pre">zookeeperQuorum</span></tt> and <tt class="docutils literal"><span class="pre">znodeParent</span></tt> values.
+You can find the list of the available properties at
+<a class="reference external" href="http://opentsdb.github.io/asynchbase/docs/build/html/configuration.html#properties">the documentation page of AsyncHBase</a>.</td>
+</tr>
 </tbody>
 </table>
 <p>Note that this sink takes the Zookeeper Quorum and parent znode information in
@@ -4294,8 +4900,8 @@ the kerberos principal</td>
 <p>This is a Flume Sink implementation that can publish data to a
 <a class="reference external" href="http://kafka.apache.org/">Kafka</a> topic. One of the objective is to integrate Flume
 with Kafka so that pull based processing systems can process the data coming
-through various Flume sources. This currently supports Kafka 0.9.x series of releases.</p>
-<p>This version of Flume no longer supports Older Versions (0.8.x) of Kafka.</p>
+through various Flume sources.</p>
+<p>This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
 <p>Required properties are marked in bold font.</p>
 <table border="1" class="docutils">
 <colgroup>
@@ -4471,10 +5077,13 @@ security provider, cipher suites, enable
 <span class="na">a1.sinks.sink1.kafka.bootstrap.servers</span> <span class="o">=</span> <span class="s">kafka-1:9093,kafka-2:9093,kafka-3:9093</span>
 <span class="na">a1.sinks.sink1.kafka.topic</span> <span class="o">=</span> <span class="s">mytopic</span>
 <span class="na">a1.sinks.sink1.kafka.producer.security.protocol</span> <span class="o">=</span> <span class="s">SSL</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.sinks.sink1.kafka.producer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.sinks.sink1.kafka.producer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
 </div>
+<p>Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section.</p>
 <p>Note: By default the property <tt class="docutils literal"><span class="pre">ssl.endpoint.identification.algorithm</span></tt>
 is not defined, so hostname verification is not performed.
 In order to enable hostname verification, set the following properties</p>
@@ -4487,11 +5096,13 @@ against one of the following two fields:
 <li>Common Name (CN) <a class="reference external" href="https://tools.ietf.org/html/rfc6125#section-2.3">https://tools.ietf.org/html/rfc6125#section-2.3</a></li>
 <li>Subject Alternative Name (SAN) <a class="reference external" href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></li>
 </ol>
-<p>If client side authentication is also required then additionally the following should be added to Flume agent configuration.
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).
 Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
 individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
 which in turn is trusted by Kafka brokers.</p>
-<div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.sinks.sink1.kafka.producer.ssl.keystore.location</span> <span class="o">=</span> <span class="s">/path/to/client.keystore.jks</span>
+<div class="highlight-properties"><div class="highlight"><pre><span class="c"># optional, the global keystore can be used alternatively</span>
+<span class="na">a1.sinks.sink1.kafka.producer.ssl.keystore.location</span> <span class="o">=</span> <span class="s">/path/to/client.keystore.jks</span>
 <span class="na">a1.sinks.sink1.kafka.producer.ssl.keystore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the keystore&gt;</span>
 </pre></div>
 </div>
@@ -4525,6 +5136,7 @@ for information on the JAAS file content
 <span class="na">a1.sinks.sink1.kafka.producer.security.protocol</span> <span class="o">=</span> <span class="s">SASL_SSL</span>
 <span class="na">a1.sinks.sink1.kafka.producer.sasl.mechanism</span> <span class="o">=</span> <span class="s">GSSAPI</span>
 <span class="na">a1.sinks.sink1.kafka.producer.sasl.kerberos.service.name</span> <span class="o">=</span> <span class="s">kafka</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.sinks.sink1.kafka.producer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.sinks.sink1.kafka.producer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
@@ -4858,8 +5470,7 @@ replication, so in case an agent or a ka
 <li>With Flume source and interceptor but no sink - it allows writing Flume events into a Kafka topic, for use by other apps</li>
 <li>With Flume sink, but no source - it is a low-latency, fault tolerant way to send events from Kafka to Flume sinks such as HDFS, HBase or Solr</li>
 </ol>
-<p>This version of Flume requires Kafka version 0.9 or greater due to the reliance on the Kafka clients shipped with that version. The configuration of
-the channel has changed compared to previous flume versions.</p>
+<p>This currently supports Kafka server releases 0.10.1.0 or higher. Testing was done up to 2.0.1 that was the highest avilable version at the time of the release.</p>
 <p>The configuration parameters are organized as such:</p>
 <ol class="arabic simple">
 <li>Configuration values related to the channel generically are applied at the channel config level, eg: a1.channel.k1.type =</li>
@@ -4910,33 +5521,26 @@ This should be true if Flume source is w
 writing into the topic that the channel is using. Flume source messages to Kafka can be parsed outside of Flume by using
 org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk artifact</td>
 </tr>
-<tr class="row-odd"><td>migrateZookeeperOffsets</td>
-<td>true</td>
-<td>When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
-This should be true to support seamless Kafka client migration from older versions of Flume. Once migrated this can be set
-to false, though that should generally not be required. If no Zookeeper offset is found the kafka.consumer.auto.offset.reset
-configuration defines how offsets are handled.</td>
-</tr>
-<tr class="row-even"><td>pollTimeout</td>
+<tr class="row-odd"><td>pollTimeout</td>
 <td>500</td>
 <td>The amount of time(in milliseconds) to wait in the &#8220;poll()&#8221; call of the consumer.
 <a class="reference external" href="https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long">https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long</a>)</td>
 </tr>
-<tr class="row-odd"><td>defaultPartitionId</td>
+<tr class="row-even"><td>defaultPartitionId</td>
 <td>&#8211;</td>
 <td>Specifies a Kafka partition ID (integer) for all events in this channel to be sent to, unless
 overriden by <tt class="docutils literal"><span class="pre">partitionIdHeader</span></tt>. By default, if this property is not set, events will be
 distributed by the Kafka Producer&#8217;s partitioner - including by <tt class="docutils literal"><span class="pre">key</span></tt> if specified (or by a
 partitioner specified by <tt class="docutils literal"><span class="pre">kafka.partitioner.class</span></tt>).</td>
 </tr>
-<tr class="row-even"><td>partitionIdHeader</td>
+<tr class="row-odd"><td>partitionIdHeader</td>
 <td>&#8211;</td>
 <td>When set, the producer will take the value of the field named using the value of this property
 from the event header and send the message to the specified partition of the topic. If the
 value represents an invalid partition the event will not be accepted into the channel. If the header value
 is present then this setting overrides <tt class="docutils literal"><span class="pre">defaultPartitionId</span></tt>.</td>
 </tr>
-<tr class="row-odd"><td>kafka.consumer.auto.offset.reset</td>
+<tr class="row-even"><td>kafka.consumer.auto.offset.reset</td>
 <td>latest</td>
 <td>What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server
 (e.g. because that data has been deleted):
@@ -4945,15 +5549,15 @@ latest: automatically reset the offset t
 none: throw exception to the consumer if no previous offset is found for the consumer&#8217;s group
 anything else: throw exception to the consumer.</td>
 </tr>
-<tr class="row-even"><td>kafka.producer.security.protocol</td>
+<tr class="row-odd"><td>kafka.producer.security.protocol</td>
 <td>PLAINTEXT</td>
 <td>Set to SASL_PLAINTEXT, SASL_SSL or SSL if writing to Kafka using some level of security. See below for additional info on secure setup.</td>
 </tr>
-<tr class="row-odd"><td>kafka.consumer.security.protocol</td>
+<tr class="row-even"><td>kafka.consumer.security.protocol</td>
 <td>PLAINTEXT</td>
 <td>Same as kafka.producer.security.protocol but for reading/consuming from Kafka.</td>
 </tr>
-<tr class="row-even"><td><em>more producer/consumer security props</em></td>
+<tr class="row-odd"><td><em>more producer/consumer security props</em></td>
 <td>&nbsp;</td>
 <td>If using SASL_PLAINTEXT, SASL_SSL or SSL refer to <a class="reference external" href="http://kafka.apache.org/documentation.html#security">Kafka security</a> for additional
 properties that need to be set on producer/consumer.</td>
@@ -4963,9 +5567,9 @@ properties that need to be set on produc
 <p>Deprecated Properties</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="19%" />
-<col width="15%" />
-<col width="66%" />
+<col width="18%" />
+<col width="14%" />
+<col width="68%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -4992,6 +5596,13 @@ The format is comma separated list of ho
 <td>false</td>
 <td>Use kafka.consumer.auto.offset.reset</td>
 </tr>
+<tr class="row-even"><td>migrateZookeeperOffsets</td>
+<td>true</td>
+<td>When no Kafka stored offset is found, look up the offsets in Zookeeper and commit them to Kafka.
+This should be true to support seamless Kafka client migration from older versions of Flume. Once migrated this can be set
+to false, though that should generally not be required. If no Zookeeper offset is found the kafka.consumer.auto.offset.reset
+configuration defines how offsets are handled.</td>
+</tr>
 </tbody>
 </table>
 <div class="admonition note">
@@ -5033,13 +5644,17 @@ security provider, cipher suites, enable
 <span class="na">a1.channels.channel1.kafka.topic</span> <span class="o">=</span> <span class="s">channel1</span>
 <span class="na">a1.channels.channel1.kafka.consumer.group.id</span> <span class="o">=</span> <span class="s">flume-consumer</span>
 <span class="na">a1.channels.channel1.kafka.producer.security.protocol</span> <span class="o">=</span> <span class="s">SSL</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.channels.channel1.kafka.producer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.channels.channel1.kafka.producer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 <span class="na">a1.channels.channel1.kafka.consumer.security.protocol</span> <span class="o">=</span> <span class="s">SSL</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
 </div>
+<p>Specyfing the truststore is optional here, the global truststore can be used instead.
+For more details about the global SSL setup, see the <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section.</p>
 <p>Note: By default the property <tt class="docutils literal"><span class="pre">ssl.endpoint.identification.algorithm</span></tt>
 is not defined, so hostname verification is not performed.
 In order to enable hostname verification, set the following properties</p>
@@ -5053,12 +5668,15 @@ against one of the following two fields:
 <li>Common Name (CN) <a class="reference external" href="https://tools.ietf.org/html/rfc6125#section-2.3">https://tools.ietf.org/html/rfc6125#section-2.3</a></li>
 <li>Subject Alternative Name (SAN) <a class="reference external" href="https://tools.ietf.org/html/rfc5280#section-4.2.1.6">https://tools.ietf.org/html/rfc5280#section-4.2.1.6</a></li>
 </ol>
-<p>If client side authentication is also required then additionally the following should be added to Flume agent configuration.
+<p>If client side authentication is also required then additionally the following needs to be added to Flume agent
+configuration or the global SSL setup can be used (see <a class="reference internal" href="#ssl-tls-support">SSL/TLS support</a> section).
 Each Flume agent has to have its client certificate which has to be trusted by Kafka brokers either
 individually or by their signature chain. Common example is to sign each client certificate by a single Root CA
 which in turn is trusted by Kafka brokers.</p>
-<div class="highlight-properties"><div class="highlight"><pre><span class="na">a1.channels.channel1.kafka.producer.ssl.keystore.location</span> <span class="o">=</span> <span class="s">/path/to/client.keystore.jks</span>
+<div class="highlight-properties"><div class="highlight"><pre><span class="c"># optional, the global keystore can be used alternatively</span>
+<span class="na">a1.channels.channel1.kafka.producer.ssl.keystore.location</span> <span class="o">=</span> <span class="s">/path/to/client.keystore.jks</span>
 <span class="na">a1.channels.channel1.kafka.producer.ssl.keystore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the keystore&gt;</span>
+<span class="c"># optional, the global keystore can be used alternatively</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.keystore.location</span> <span class="o">=</span> <span class="s">/path/to/client.keystore.jks</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.keystore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the keystore&gt;</span>
 </pre></div>
@@ -5099,11 +5717,13 @@ for information on the JAAS file content
 <span class="na">a1.channels.channel1.kafka.producer.security.protocol</span> <span class="o">=</span> <span class="s">SASL_SSL</span>
 <span class="na">a1.channels.channel1.kafka.producer.sasl.mechanism</span> <span class="o">=</span> <span class="s">GSSAPI</span>
 <span class="na">a1.channels.channel1.kafka.producer.sasl.kerberos.service.name</span> <span class="o">=</span> <span class="s">kafka</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.channels.channel1.kafka.producer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.channels.channel1.kafka.producer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 <span class="na">a1.channels.channel1.kafka.consumer.security.protocol</span> <span class="o">=</span> <span class="s">SASL_SSL</span>
 <span class="na">a1.channels.channel1.kafka.consumer.sasl.mechanism</span> <span class="o">=</span> <span class="s">GSSAPI</span>
 <span class="na">a1.channels.channel1.kafka.consumer.sasl.kerberos.service.name</span> <span class="o">=</span> <span class="s">kafka</span>
+<span class="c"># optional, the global truststore can be used alternatively</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.truststore.location</span> <span class="o">=</span> <span class="s">/path/to/truststore.jks</span>
 <span class="na">a1.channels.channel1.kafka.consumer.ssl.truststore.password</span> <span class="o">=</span> <span class="s">&lt;password to access the truststore&gt;</span>
 </pre></div>
@@ -5939,7 +6559,7 @@ This interceptor can preserve an existin
 <td>&#8211;</td>
 <td>The component type name, has to be <tt class="docutils literal"><span class="pre">timestamp</span></tt> or the FQCN</td>
 </tr>
-<tr class="row-odd"><td>header</td>
+<tr class="row-odd"><td>headerName</td>
 <td>timestamp</td>
 <td>The name of the header in which to place the generated timestamp.</td>
 </tr>
@@ -6381,11 +7001,196 @@ polling rather than terminating.</p>
 </div>
 </div>
 </div>
+<div class="section" id="configuration-filters">
+<h2>Configuration Filters<a class="headerlink" href="#configuration-filters" title="Permalink to this headline">¶</a></h2>
+<p>Flume provides a tool for injecting sensitive or generated data into the configuration
+in the form of configuration filters. A configuration key can be set as the value of configuration properties
+and it will be replaced by the configuration filter with the value it represents.</p>
+<div class="section" id="common-usage-of-config-filters">
+<h3>Common usage of config filters<a class="headerlink" href="#common-usage-of-config-filters" title="Permalink to this headline">¶</a></h3>
+<p>The format is similar to the Java Expression Language, however
+it is currently not a fully working EL expression parser, just a format that looks like it.</p>
+<div class="highlight-properties"><div class="highlight"><pre><span class="na">&lt;agent_name&gt;.configfilters</span> <span class="o">=</span> <span class="s">&lt;filter_name&gt;</span>
+<span class="na">&lt;agent_name&gt;.configfilters.&lt;filter_name&gt;.type</span> <span class="o">=</span> <span class="s">&lt;filter_type&gt;</span>
+

[... 1164 lines stripped ...]


Mime
View raw message