flume-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r960078 [1/2] - in /websites/staging/flume/trunk/content: ./ .doctrees/ .doctrees/releases/ _sources/ releases/
Date Wed, 29 Jul 2015 20:33:51 GMT
Author: buildbot
Date: Wed Jul 29 20:33:50 2015
New Revision: 960078

Log:
Staging update by buildbot for flume

Modified:
    websites/staging/flume/trunk/content/   (props changed)
    websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
    websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
    websites/staging/flume/trunk/content/.doctrees/documentation.doctree
    websites/staging/flume/trunk/content/.doctrees/download.doctree
    websites/staging/flume/trunk/content/.doctrees/environment.pickle
    websites/staging/flume/trunk/content/.doctrees/getinvolved.doctree
    websites/staging/flume/trunk/content/.doctrees/index.doctree
    websites/staging/flume/trunk/content/.doctrees/license.doctree
    websites/staging/flume/trunk/content/.doctrees/mailinglists.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.0.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.1.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.2.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.3.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.3.1.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.4.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.1.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.5.2.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
    websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
    websites/staging/flume/trunk/content/.doctrees/source.doctree
    websites/staging/flume/trunk/content/.doctrees/team.doctree
    websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
    websites/staging/flume/trunk/content/FlumeUserGuide.html
    websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
    websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
    websites/staging/flume/trunk/content/documentation.html
    websites/staging/flume/trunk/content/project-reports.html
    websites/staging/flume/trunk/content/releases/index.html
    websites/staging/flume/trunk/content/searchindex.js

Propchange: websites/staging/flume/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Jul 29 20:33:50 2015
@@ -1 +1 @@
-1683005
+1693341

Modified: websites/staging/flume/trunk/content/.doctrees/FlumeDeveloperGuide.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/FlumeUserGuide.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/documentation.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/download.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/environment.pickle
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/getinvolved.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/index.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/license.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/mailinglists.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.0.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.1.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.2.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.3.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.3.1.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.4.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.1.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.5.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.5.2.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/1.6.0.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/releases/index.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/source.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/.doctrees/team.doctree
==============================================================================
Binary files - no diff available.

Modified: websites/staging/flume/trunk/content/FlumeDeveloperGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeDeveloperGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeDeveloperGuide.html Wed Jul 29 20:33:50 2015
@@ -7,7 +7,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     
-    <title>Flume 1.6.0 Developer Guide &mdash; Apache Flume</title>
+    <title>Flume 1.7.0-SNAPSHOT Developer Guide &mdash; Apache Flume</title>
     
     <link rel="stylesheet" href="_static/flume.css" type="text/css" />
     <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -27,7 +27,7 @@
     <link rel="top" title="Apache Flume" href="index.html" />
     <link rel="up" title="Documentation" href="documentation.html" />
     <link rel="next" title="Releases" href="releases/index.html" />
-    <link rel="prev" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html" /> 
+    <link rel="prev" title="Flume 1.7.0-SNAPSHOT User Guide" href="FlumeUserGuide.html"
/> 
   </head>
   <body>
 <div class="header">
@@ -59,8 +59,8 @@
         <div class="bodywrapper">
           <div class="body">
             
-  <div class="section" id="flume-1-6-0-developer-guide">
-<h1>Flume 1.6.0 Developer Guide<a class="headerlink" href="#flume-1-6-0-developer-guide"
title="Permalink to this headline">¶</a></h1>
+  <div class="section" id="flume-1-7-0-snapshot-developer-guide">
+<h1>Flume 1.7.0-SNAPSHOT Developer Guide<a class="headerlink" href="#flume-1-7-0-snapshot-developer-guide"
title="Permalink to this headline">¶</a></h1>
 <div class="section" id="introduction">
 <h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this
headline">¶</a></h2>
 <div class="section" id="overview">
@@ -677,9 +677,10 @@ Flume <tt class="docutils literal"><span
 <img alt="Transaction sequence diagram" src="_images/DevGuide_image01.png" />
 </div>
 <p>A <tt class="docutils literal"><span class="pre">Transaction</span></tt>
is implemented within a <tt class="docutils literal"><span class="pre">Channel</span></tt>
implementation. Each
-<tt class="docutils literal"><span class="pre">Source</span></tt>
and <tt class="docutils literal"><span class="pre">Sink</span></tt>
that is connected to <tt class="docutils literal"><span class="pre">Channel</span></tt>
must obtain a
-<tt class="docutils literal"><span class="pre">Transaction</span></tt>
object. The <tt class="docutils literal"><span class="pre">Source</span></tt>s
actually use a <tt class="docutils literal"><span class="pre">ChannelSelector</span></tt>
-interface to encapsulate the <tt class="docutils literal"><span class="pre">Transaction</span></tt>.
The operation to stage an
+<tt class="docutils literal"><span class="pre">Source</span></tt>
and <tt class="docutils literal"><span class="pre">Sink</span></tt>
that is connected to a <tt class="docutils literal"><span class="pre">Channel</span></tt>
must obtain a
+<tt class="docutils literal"><span class="pre">Transaction</span></tt>
object. The <tt class="docutils literal"><span class="pre">Source</span></tt>s
use a <tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
+to manage the <tt class="docutils literal"><span class="pre">Transaction</span></tt>s,
the <tt class="docutils literal"><span class="pre">Sink</span></tt>s
manage them explicitly via
+their configured <tt class="docutils literal"><span class="pre">Channel</span></tt>.
The operation to stage an
 <tt class="docutils literal"><span class="pre">Event</span></tt>
(put it into a <tt class="docutils literal"><span class="pre">Channel</span></tt>)
or extract an <tt class="docutils literal"><span class="pre">Event</span></tt>
(take it out of a
 <tt class="docutils literal"><span class="pre">Channel</span></tt>)
is done inside an active <tt class="docutils literal"><span class="pre">Transaction</span></tt>.
For example:</p>
 <div class="highlight-java"><div class="highlight"><pre><span class="n">Channel</span>
<span class="n">ch</span> <span class="o">=</span> <span class="k">new</span>
<span class="n">MemoryChannel</span><span class="o">();</span>
@@ -786,8 +787,6 @@ processing its own configuration setting
       <span class="k">if</span> <span class="o">(</span><span
class="n">t</span> <span class="k">instanceof</span> <span class="n">Error</span><span
class="o">)</span> <span class="o">{</span>
         <span class="k">throw</span> <span class="o">(</span><span
class="n">Error</span><span class="o">)</span><span class="n">t</span><span
class="o">;</span>
       <span class="o">}</span>
-    <span class="o">}</span> <span class="k">finally</span> <span
class="o">{</span>
-      <span class="n">txn</span><span class="o">.</span><span
class="na">close</span><span class="o">();</span>
     <span class="o">}</span>
     <span class="k">return</span> <span class="n">status</span><span
class="o">;</span>
   <span class="o">}</span>
@@ -798,13 +797,12 @@ processing its own configuration setting
 <div class="section" id="source">
 <h4>Source<a class="headerlink" href="#source" title="Permalink to this headline">¶</a></h4>
 <p>The purpose of a <tt class="docutils literal"><span class="pre">Source</span></tt>
is to receive data from an external client and store
-it into the <tt class="docutils literal"><span class="pre">Channel</span></tt>.
A <tt class="docutils literal"><span class="pre">Source</span></tt>
can get an instance of its own
-<tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
to process an <tt class="docutils literal"><span class="pre">Event</span></tt>.
The <tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
in turn
-can get an instance of its own <tt class="docutils literal"><span class="pre">ChannelSelector</span></tt>
that&#8217;s used to get the
-<tt class="docutils literal"><span class="pre">Channel</span></tt>s
associated with the <tt class="docutils literal"><span class="pre">Source</span></tt>,
as configured in the Flume
-properties file. A <tt class="docutils literal"><span class="pre">Transaction</span></tt>
can then be retrieved from each associated
-<tt class="docutils literal"><span class="pre">Channel</span></tt>
so that the <tt class="docutils literal"><span class="pre">Source</span></tt>
can place <tt class="docutils literal"><span class="pre">Event</span></tt>s
into the <tt class="docutils literal"><span class="pre">Channel</span></tt>
-reliably, within a <tt class="docutils literal"><span class="pre">Transaction</span></tt>.</p>
+it into the configured <tt class="docutils literal"><span class="pre">Channel</span></tt>s.
A <tt class="docutils literal"><span class="pre">Source</span></tt>
can get an instance of its own
+<tt class="docutils literal"><span class="pre">ChannelProcessor</span></tt>
to process an <tt class="docutils literal"><span class="pre">Event</span></tt>,
commited within a <tt class="docutils literal"><span class="pre">Channel</span></tt>
+local transaction, in serial. In the case of an exception, required
+<tt class="docutils literal"><span class="pre">Channel</span></tt>s
will propagate the exception, all <tt class="docutils literal"><span class="pre">Channel</span></tt>s
will rollback their
+transaction, but events processed previously on other <tt class="docutils literal"><span
class="pre">Channel</span></tt>s will remain
+committed.</p>
 <p>Similar to the <tt class="docutils literal"><span class="pre">SinkRunner.PollingRunner</span></tt>
<tt class="docutils literal"><span class="pre">Runnable</span></tt>,
there’s
 a <tt class="docutils literal"><span class="pre">PollingRunner</span></tt>
<tt class="docutils literal"><span class="pre">Runnable</span></tt>
that executes on a thread created when the
 Flume framework calls <tt class="docutils literal"><span class="pre">PollableSourceRunner.start()</span></tt>.
Each configured
@@ -849,24 +847,17 @@ mechanism that captures the new data and
   <span class="kd">public</span> <span class="n">Status</span> <span
class="nf">process</span><span class="o">()</span> <span class="kd">throws</span>
<span class="n">EventDeliveryException</span> <span class="o">{</span>
     <span class="n">Status</span> <span class="n">status</span> <span
class="o">=</span> <span class="kc">null</span><span class="o">;</span>
 
-    <span class="c1">// Start transaction</span>
-    <span class="n">Channel</span> <span class="n">ch</span> <span
class="o">=</span> <span class="n">getChannel</span><span class="o">();</span>
-    <span class="n">Transaction</span> <span class="n">txn</span>
<span class="o">=</span> <span class="n">ch</span><span class="o">.</span><span
class="na">getTransaction</span><span class="o">();</span>
-    <span class="n">txn</span><span class="o">.</span><span class="na">begin</span><span
class="o">();</span>
     <span class="k">try</span> <span class="o">{</span>
-      <span class="c1">// This try clause includes whatever Channel operations you
want to do</span>
+      <span class="c1">// This try clause includes whatever Channel/Event operations
you want to do</span>
 
       <span class="c1">// Receive new data</span>
       <span class="n">Event</span> <span class="n">e</span> <span
class="o">=</span> <span class="n">getSomeData</span><span class="o">();</span>
 
       <span class="c1">// Store the Event into this Source&#39;s associated Channel(s)</span>
-      <span class="n">getChannelProcessor</span><span class="o">().</span><span
class="na">processEvent</span><span class="o">(</span><span class="n">e</span><span
class="o">)</span>
+      <span class="n">getChannelProcessor</span><span class="o">().</span><span
class="na">processEvent</span><span class="o">(</span><span class="n">e</span><span
class="o">);</span>
 
-      <span class="n">txn</span><span class="o">.</span><span
class="na">commit</span><span class="o">();</span>
       <span class="n">status</span> <span class="o">=</span> <span
class="n">Status</span><span class="o">.</span><span class="na">READY</span><span
class="o">;</span>
     <span class="o">}</span> <span class="k">catch</span> <span
class="o">(</span><span class="n">Throwable</span> <span class="n">t</span><span
class="o">)</span> <span class="o">{</span>
-      <span class="n">txn</span><span class="o">.</span><span
class="na">rollback</span><span class="o">();</span>
-
       <span class="c1">// Log exception, handle individual exceptions as needed</span>
 
       <span class="n">status</span> <span class="o">=</span> <span
class="n">Status</span><span class="o">.</span><span class="na">BACKOFF</span><span
class="o">;</span>
@@ -933,7 +924,7 @@ mechanism that captures the new data and
 
 <h3><a href="index.html">This Page</a></h3>
 <ul>
-<li><a class="reference internal" href="#">Flume 1.6.0 Developer Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.7.0-SNAPSHOT Developer Guide</a><ul>
 <li><a class="reference internal" href="#introduction">Introduction</a><ul>
 <li><a class="reference internal" href="#overview">Overview</a></li>
 <li><a class="reference internal" href="#architecture">Architecture</a><ul>

Modified: websites/staging/flume/trunk/content/FlumeUserGuide.html
==============================================================================
--- websites/staging/flume/trunk/content/FlumeUserGuide.html (original)
+++ websites/staging/flume/trunk/content/FlumeUserGuide.html Wed Jul 29 20:33:50 2015
@@ -7,7 +7,7 @@
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
     
-    <title>Flume 1.6.0 User Guide &mdash; Apache Flume</title>
+    <title>Flume 1.7.0-SNAPSHOT User Guide &mdash; Apache Flume</title>
     
     <link rel="stylesheet" href="_static/flume.css" type="text/css" />
     <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
@@ -26,7 +26,7 @@
     <script type="text/javascript" src="_static/doctools.js"></script>
     <link rel="top" title="Apache Flume" href="index.html" />
     <link rel="up" title="Documentation" href="documentation.html" />
-    <link rel="next" title="Flume 1.6.0 Developer Guide" href="FlumeDeveloperGuide.html"
/>
+    <link rel="next" title="Flume 1.7.0-SNAPSHOT Developer Guide" href="FlumeDeveloperGuide.html"
/>
     <link rel="prev" title="Documentation" href="documentation.html" /> 
   </head>
   <body>
@@ -59,8 +59,8 @@
         <div class="bodywrapper">
           <div class="body">
             
-  <div class="section" id="flume-1-6-0-user-guide">
-<h1>Flume 1.6.0 User Guide<a class="headerlink" href="#flume-1-6-0-user-guide" title="Permalink
to this headline">¶</a></h1>
+  <div class="section" id="flume-1-7-0-snapshot-user-guide">
+<h1>Flume 1.7.0-SNAPSHOT User Guide<a class="headerlink" href="#flume-1-7-0-snapshot-user-guide"
title="Permalink to this headline">¶</a></h1>
 <div class="section" id="introduction">
 <h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this
headline">¶</a></h2>
 <div class="section" id="overview">
@@ -1504,6 +1504,19 @@ indicates that they are part of the same
 <td>Maximum time (in ms) before a batch will be written to Channel
 The batch will be written whenever the first of size and time will be reached.</td>
 </tr>
+<tr class="row-odd"><td>backoffSleepIncrement</td>
+<td>1000</td>
+<td>Initial and incremental wait time that is triggered when a Kafka Topic appears
to be empty.
+Wait period will reduce aggressive pinging of an empty Kafka Topic.  One second is ideal
for
+ingestion use cases but a lower value may be required for low latency operations with
+interceptors.</td>
+</tr>
+<tr class="row-even"><td>maxBackoffSleep</td>
+<td>5000</td>
+<td>Maximum wait time that is triggered when a Kafka Topic appears to be empty.  Five
seconds is
+ideal for ingestion use cases but a lower value may be required for low latency operations
+with interceptors.</td>
+</tr>
 <tr class="row-odd"><td>Other Kafka Consumer Properties</td>
 <td>&#8211;</td>
 <td>These properties are used to configure the Kafka Consumer. Any producer property
supported
@@ -3023,8 +3036,8 @@ Required properties are in <strong>bold<
 <td>The type of the Java truststore. This can be &#8220;JKS&#8221; or other
supported Java truststore type.</td>
 </tr>
 <tr class="row-odd"><td>exclude-protocols</td>
-<td>SSLv2Hello SSLv3</td>
-<td>Space-separated list of SSL/TLS protocols to exclude</td>
+<td>SSLv3</td>
+<td>Space-separated list of SSL/TLS protocols to exclude. SSLv3 will always be excluded
in addition to the protocols specified.</td>
 </tr>
 <tr class="row-even"><td>maxIoWorkers</td>
 <td>2 * the number of available processors in the machine</td>
@@ -3521,9 +3534,9 @@ will read this information from the firs
 <p>Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
-<col width="8%" />
-<col width="30%" />
-<col width="62%" />
+<col width="7%" />
+<col width="16%" />
+<col width="77%" />
 </colgroup>
 <thead valign="bottom">
 <tr class="row-odd"><th class="head">Property Name</th>
@@ -3560,6 +3573,18 @@ will read this information from the firs
 <td>org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl</td>
 <td>The FQCN of a class implementing org.apache.flume.sink.solr.morphline.MorphlineHandler</td>
 </tr>
+<tr class="row-odd"><td>isProductionMode</td>
+<td>false</td>
+<td>This flag should be enabled for mission critical, large-scale online production
systems that need to make progress without downtime when unrecoverable exceptions occur. Corrupt
or malformed parser input data, parser bugs, and errors related to unknown Solr schema fields
produce unrecoverable exceptions.</td>
+</tr>
+<tr class="row-even"><td>recoverableExceptionClasses</td>
+<td>org.apache.solr.client.solrj.SolrServerException</td>
+<td>Comma separated list of recoverable exceptions that tend to be transient, in which
case the corresponding task can be retried. Examples include network connection errors, timeouts,
etc. When the production mode flag is set to true, the recoverable exceptions configured using
this parameter will not be ignored and hence will lead to retries.</td>
+</tr>
+<tr class="row-odd"><td>isIgnoringRecoverableExceptions</td>
+<td>false</td>
+<td>This flag should be enabled, if an unrecoverable exception is accidentally misclassified
as recoverable. This enables the sink to make progress and avoid retrying an event forever.</td>
+</tr>
 </tbody>
 </table>
 <p>Example for agent named a1:</p>
@@ -5320,7 +5345,7 @@ polling rather than terminating.</p>
 <h2>Log4J Appender<a class="headerlink" href="#log4j-appender" title="Permalink
to this headline">¶</a></h2>
 <p>Appends Log4j events to a flume agent&#8217;s avro source. A client using this
 appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar).
+flume-ng-sdk-1.7.0-SNAPSHOT.jar).
 Required properties are in <strong>bold</strong>.</p>
 <table border="1" class="docutils">
 <colgroup>
@@ -5400,7 +5425,7 @@ then the schema will be included as a Fl
 <h2>Load Balancing Log4J Appender<a class="headerlink" href="#load-balancing-log4j-appender"
title="Permalink to this headline">¶</a></h2>
 <p>Appends Log4j events to a list of flume agent&#8217;s avro source. A client
using this
 appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar). This appender supports a round-robin and random
+flume-ng-sdk-1.7.0-SNAPSHOT.jar). This appender supports a round-robin and random
 scheme for performing the load balancing. It also supports a configurable backoff
 timeout so that down agents are removed temporarily from the set of hosts
 Required properties are in <strong>bold</strong>.</p>
@@ -6159,7 +6184,7 @@ can be leveraged to move the Flume agent
 
 <h3><a href="index.html">This Page</a></h3>
 <ul>
-<li><a class="reference internal" href="#">Flume 1.6.0 User Guide</a><ul>
+<li><a class="reference internal" href="#">Flume 1.7.0-SNAPSHOT User Guide</a><ul>
 <li><a class="reference internal" href="#introduction">Introduction</a><ul>
 <li><a class="reference internal" href="#overview">Overview</a></li>
 <li><a class="reference internal" href="#system-requirements">System Requirements</a></li>

Modified: websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt
==============================================================================
--- websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt (original)
+++ websites/staging/flume/trunk/content/_sources/FlumeDeveloperGuide.txt Wed Jul 29 20:33:50
2015
@@ -15,7 +15,7 @@
 
 
 ======================================
-Flume 1.6.0 Developer Guide
+Flume 1.7.0-SNAPSHOT Developer Guide
 ======================================
 
 Introduction
@@ -644,9 +644,10 @@ Flume ``Transaction``.
    :alt: Transaction sequence diagram
 
 A ``Transaction`` is implemented within a ``Channel`` implementation. Each
-``Source`` and ``Sink`` that is connected to ``Channel`` must obtain a
-``Transaction`` object. The ``Source``\ s actually use a ``ChannelSelector``
-interface to encapsulate the ``Transaction``. The operation to stage an
+``Source`` and ``Sink`` that is connected to a ``Channel`` must obtain a
+``Transaction`` object. The ``Source``\ s use a ``ChannelProcessor``
+to manage the ``Transaction``\ s, the ``Sink``\ s manage them explicitly via
+their configured ``Channel``. The operation to stage an
 ``Event`` (put it into a ``Channel``) or extract an ``Event`` (take it out of a
 ``Channel``) is done inside an active ``Transaction``. For example:
 
@@ -759,8 +760,6 @@ processing its own configuration setting
         if (t instanceof Error) {
           throw (Error)t;
         }
-      } finally {
-        txn.close();
       }
       return status;
     }
@@ -770,13 +769,12 @@ Source
 ~~~~~~
 
 The purpose of a ``Source`` is to receive data from an external client and store
-it into the ``Channel``. A ``Source`` can get an instance of its own
-``ChannelProcessor`` to process an ``Event``. The ``ChannelProcessor`` in turn
-can get an instance of its own ``ChannelSelector`` that's used to get the
-``Channel``\ s associated with the ``Source``, as configured in the Flume
-properties file. A ``Transaction`` can then be retrieved from each associated
-``Channel`` so that the ``Source`` can place ``Event``\ s into the ``Channel``
-reliably, within a ``Transaction``.
+it into the configured ``Channel``\ s. A ``Source`` can get an instance of its own
+``ChannelProcessor`` to process an ``Event``, commited within a ``Channel``
+local transaction, in serial. In the case of an exception, required
+``Channel``\ s will propagate the exception, all ``Channel``\ s will rollback their
+transaction, but events processed previously on other ``Channel``\ s will remain
+committed.
 
 Similar to the ``SinkRunner.PollingRunner`` ``Runnable``, there’s
 a ``PollingRunner`` ``Runnable`` that executes on a thread created when the
@@ -826,24 +824,17 @@ mechanism that captures the new data and
     public Status process() throws EventDeliveryException {
       Status status = null;
 
-      // Start transaction
-      Channel ch = getChannel();
-      Transaction txn = ch.getTransaction();
-      txn.begin();
       try {
-        // This try clause includes whatever Channel operations you want to do
+        // This try clause includes whatever Channel/Event operations you want to do
 
         // Receive new data
         Event e = getSomeData();
 
         // Store the Event into this Source's associated Channel(s)
-        getChannelProcessor().processEvent(e)
+        getChannelProcessor().processEvent(e);
 
-        txn.commit();
         status = Status.READY;
       } catch (Throwable t) {
-        txn.rollback();
-
         // Log exception, handle individual exceptions as needed
 
         status = Status.BACKOFF;

Modified: websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt
==============================================================================
--- websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt (original)
+++ websites/staging/flume/trunk/content/_sources/FlumeUserGuide.txt Wed Jul 29 20:33:50 2015
@@ -15,7 +15,7 @@
 
 
 ======================================
-Flume 1.6.0 User Guide
+Flume 1.7.0-SNAPSHOT User Guide
 ======================================
 
 Introduction
@@ -1152,6 +1152,13 @@ Property Name                    Default
 batchSize                        1000         Maximum number of messages written to Channel
in one batch
 batchDurationMillis              1000         Maximum time (in ms) before a batch will be
written to Channel
                                               The batch will be written whenever the first
of size and time will be reached.
+backoffSleepIncrement            1000         Initial and incremental wait time that is triggered
when a Kafka Topic appears to be empty.
+                                              Wait period will reduce aggressive pinging
of an empty Kafka Topic.  One second is ideal for
+                                              ingestion use cases but a lower value may be
required for low latency operations with
+                                              interceptors.
+maxBackoffSleep                  5000         Maximum wait time that is triggered when a
Kafka Topic appears to be empty.  Five seconds is
+                                              ideal for ingestion use cases but a lower value
may be required for low latency operations
+                                              with interceptors.
 Other Kafka Consumer Properties  --           These properties are used to configure the
Kafka Consumer. Any producer property supported
                                               by Kafka can be used. The only requirement
is to prepend the property name with the prefix ``kafka.``.
                                               For example: kafka.consumer.timeout.ms
@@ -1952,7 +1959,7 @@ trust-all-certs              false
 truststore                   --                                                     The path
to a custom Java truststore file. Flume uses the certificate authority information in this
file to determine whether the remote Avro Source's SSL authentication credentials should be
trusted. If not specified, the default Java JSSE certificate authority files (typically "jssecacerts"
or "cacerts" in the Oracle JRE) will be used.
 truststore-password          --                                                     The password
for the specified truststore.
 truststore-type              JKS                                                    The type
of the Java truststore. This can be "JKS" or other supported Java truststore type.
-exclude-protocols            SSLv2Hello SSLv3                                       Space-separated
list of SSL/TLS protocols to exclude
+exclude-protocols            SSLv3                                                  Space-separated
list of SSL/TLS protocols to exclude. SSLv3 will always be excluded in addition to the protocols
specified.
 maxIoWorkers                 2 * the number of available processors in the machine  The maximum
number of I/O worker threads. This is configured on the NettyAvroRpcClient NioClientSocketChannelFactory.
 ==========================   =====================================================  ===========================================================================================
 
@@ -2241,17 +2248,20 @@ The type is the FQCN: org.apache.flume.s
 
 Required properties are in **bold**.
 
-===================  =======================================================================
 ========================
-Property Name        Default                                                            
     Description
-===================  =======================================================================
 ========================
-**channel**          --
-**type**             --                                                                 
     The component type name, needs to be ``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
-**morphlineFile**    --                                                                 
     The relative or absolute path on the local file system to the morphline configuration
file. Example: ``/etc/flume-ng/conf/morphline.conf``
-morphlineId          null                                                               
     Optional name used to identify a morphline if there are multiple morphlines in a morphline
config file
-batchSize            1000                                                               
     The maximum number of events to take per flume transaction.
-batchDurationMillis  1000                                                               
     The maximum duration per flume transaction (ms). The transaction commits after this duration
or when batchSize is exceeded, whichever comes first.
-handlerClass         org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl          
     The FQCN of a class implementing org.apache.flume.sink.solr.morphline.MorphlineHandler
-===================  =======================================================================
 ========================
+===============================   =======================================================================
 ========================
+Property Name                     Default                                               
                  Description
+===============================   =======================================================================
 ========================
+**channel**                       --
+**type**                          --                                                    
                  The component type name, needs to be ``org.apache.flume.sink.solr.morphline.MorphlineSolrSink``
+**morphlineFile**                 --                                                    
                  The relative or absolute path on the local file system to the morphline
configuration file. Example: ``/etc/flume-ng/conf/morphline.conf``
+morphlineId                       null                                                  
                  Optional name used to identify a morphline if there are multiple morphlines
in a morphline config file
+batchSize                         1000                                                  
                  The maximum number of events to take per flume transaction.
+batchDurationMillis               1000                                                  
                  The maximum duration per flume transaction (ms). The transaction commits
after this duration or when batchSize is exceeded, whichever comes first.
+handlerClass                      org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl
               The FQCN of a class implementing org.apache.flume.sink.solr.morphline.MorphlineHandler
+isProductionMode                  false                                                 
                  This flag should be enabled for mission critical, large-scale online production
systems that need to make progress without downtime when unrecoverable exceptions occur. Corrupt
or malformed parser input data, parser bugs, and errors related to unknown Solr schema fields
produce unrecoverable exceptions.
+recoverableExceptionClasses       org.apache.solr.client.solrj.SolrServerException      
                  Comma separated list of recoverable exceptions that tend to be transient,
in which case the corresponding task can be retried. Examples include network connection errors,
timeouts, etc. When the production mode flag is set to true, the recoverable exceptions configured
using this parameter will not be ignored and hence will lead to retries.
+isIgnoringRecoverableExceptions   false                                                 
                  This flag should be enabled, if an unrecoverable exception is accidentally
misclassified as recoverable. This enables the sink to make progress and avoid retrying an
event forever.
+===============================   =======================================================================
 ========================
 
 Example for agent named a1:
 
@@ -3420,7 +3430,7 @@ Log4J Appender
 
 Appends Log4j events to a flume agent's avro source. A client using this
 appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar).
+flume-ng-sdk-1.7.0-SNAPSHOT.jar).
 Required properties are in **bold**.
 
 =====================  =======  ==================================================================================
@@ -3484,7 +3494,7 @@ Load Balancing Log4J Appender
 
 Appends Log4j events to a list of flume agent's avro source. A client using this
 appender must have the flume-ng-sdk in the classpath (eg,
-flume-ng-sdk-1.6.0.jar). This appender supports a round-robin and random
+flume-ng-sdk-1.7.0-SNAPSHOT.jar). This appender supports a round-robin and random
 scheme for performing the load balancing. It also supports a configurable backoff
 timeout so that down agents are removed temporarily from the set of hosts
 Required properties are in **bold**.

Modified: websites/staging/flume/trunk/content/documentation.html
==============================================================================
--- websites/staging/flume/trunk/content/documentation.html (original)
+++ websites/staging/flume/trunk/content/documentation.html Wed Jul 29 20:33:50 2015
@@ -25,7 +25,7 @@
     <script type="text/javascript" src="_static/underscore.js"></script>
     <script type="text/javascript" src="_static/doctools.js"></script>
     <link rel="top" title="Apache Flume" href="index.html" />
-    <link rel="next" title="Flume 1.6.0 User Guide" href="FlumeUserGuide.html" />
+    <link rel="next" title="Flume 1.7.0-SNAPSHOT User Guide" href="FlumeUserGuide.html"
/>
     <link rel="prev" title="Download" href="download.html" /> 
   </head>
   <body>

Modified: websites/staging/flume/trunk/content/project-reports.html
==============================================================================
--- websites/staging/flume/trunk/content/project-reports.html (original)
+++ websites/staging/flume/trunk/content/project-reports.html Wed Jul 29 20:33:50 2015
@@ -1,5 +1,5 @@
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
-<!-- Generated by Apache Maven Doxia at Jun 1, 2015 -->
+<!-- Generated by Apache Maven Doxia at Jul 29, 2015 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
@@ -12,7 +12,7 @@
     <script type="text/javascript" src="./js/apache-maven-fluido.min.js"></script>
 
     
-    <meta name="Date-Revision-yyyymmdd" content="20150601" />
+    <meta name="Date-Revision-yyyymmdd" content="20150729" />
     <meta http-equiv="Content-Language" content="en" />
     
         </head>
@@ -27,7 +27,7 @@
                 
                             
                 
-                  <li id="publishDate" class="pull-right">Last Published: 2015-06-01</li>

+                  <li id="publishDate" class="pull-right">Last Published: 2015-07-29</li>

                     </ul>
       </div>
 

Modified: websites/staging/flume/trunk/content/releases/index.html
==============================================================================
--- websites/staging/flume/trunk/content/releases/index.html (original)
+++ websites/staging/flume/trunk/content/releases/index.html Wed Jul 29 20:33:50 2015
@@ -26,7 +26,7 @@
     <script type="text/javascript" src="../_static/doctools.js"></script>
     <link rel="top" title="Apache Flume" href="../index.html" />
     <link rel="next" title="Version 1.6.0" href="1.6.0.html" />
-    <link rel="prev" title="Flume 1.6.0 Developer Guide" href="../FlumeDeveloperGuide.html"
/> 
+    <link rel="prev" title="Flume 1.7.0-SNAPSHOT Developer Guide" href="../FlumeDeveloperGuide.html"
/> 
   </head>
   <body>
 <div class="header">



Mime
View raw message