flink-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From g...@apache.org
Subject [3/3] flink git commit: [FLINK-6513] [docs] cleaned up some typos and grammatical flaws
Date Wed, 10 May 2017 18:47:20 GMT
[FLINK-6513] [docs] cleaned up some typos and grammatical flaws

This closes #3858


Project: http://git-wip-us.apache.org/repos/asf/flink/repo
Commit: http://git-wip-us.apache.org/repos/asf/flink/commit/81f58bae
Tree: http://git-wip-us.apache.org/repos/asf/flink/tree/81f58bae
Diff: http://git-wip-us.apache.org/repos/asf/flink/diff/81f58bae

Branch: refs/heads/release-1.3
Commit: 81f58bae1a6730e9e6e06185e1dc6cac36e82892
Parents: 81b6c82
Author: David Anderson <david@alpinegizmo.com>
Authored: Tue May 9 16:50:53 2017 +0200
Committer: Greg Hogan <code@greghogan.com>
Committed: Wed May 10 14:37:40 2017 -0400

----------------------------------------------------------------------
 docs/dev/best_practices.md             | 34 +++++++++++++---------------
 docs/dev/stream/checkpointing.md       | 11 ++++-----
 docs/dev/stream/process_function.md    |  4 ++--
 docs/dev/stream/side_output.md         |  4 ++--
 docs/internals/stream_checkpointing.md | 35 ++++++++++++++++-------------
 5 files changed, 46 insertions(+), 42 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/flink/blob/81f58bae/docs/dev/best_practices.md
----------------------------------------------------------------------
diff --git a/docs/dev/best_practices.md b/docs/dev/best_practices.md
index 4dfd7fd..6328d22 100644
--- a/docs/dev/best_practices.md
+++ b/docs/dev/best_practices.md
@@ -30,14 +30,12 @@ This page contains a collection of best practices for Flink programmers
on how t
 
 ## Parsing command line arguments and passing them around in your Flink application
 
+Almost all Flink applications, both batch and streaming, rely on external configuration parameters.
+They are used to specify input and output sources (like paths or addresses), system parameters
(parallelism, runtime configuration), and application specific parameters (typically used
within user functions).
 
-Almost all Flink applications, both batch and streaming rely on external configuration parameters.
-For example for specifying input and output sources (like paths or addresses), also system
parameters (parallelism, runtime configuration) and application specific parameters (often
used within the user functions).
-
-Since version 0.9 we are providing a simple utility called `ParameterTool` to provide at
least some basic tooling for solving these problems.
-
-Please note that you don't have to use the `ParameterTool` explained here. Other frameworks
such as [Commons CLI](https://commons.apache.org/proper/commons-cli/),
-[argparse4j](http://argparse4j.sourceforge.net/) and others work well with Flink as well.
+Flink provides a simple utility called `ParameterTool` to provide some basic tooling for
solving these problems.
+Please note that you don't have to use the `ParameterTool` described here. Other frameworks
such as [Commons CLI](https://commons.apache.org/proper/commons-cli/) and
+[argparse4j](http://argparse4j.sourceforge.net/) also work well with Flink.
 
 
 ### Getting your configuration values into the `ParameterTool`
@@ -89,8 +87,8 @@ parameter.getNumberOfParameters()
 // .. there are more methods available.
 {% endhighlight %}
 
-You can use the return values of these methods directly in the main() method (=the client
submitting the application).
-For example you could set the parallelism of a operator like this:
+You can use the return values of these methods directly in the `main()` method of the client
submitting the application.
+For example, you could set the parallelism of a operator like this:
 
 {% highlight java %}
 ParameterTool parameters = ParameterTool.fromArgs(args);
@@ -105,10 +103,10 @@ ParameterTool parameters = ParameterTool.fromArgs(args);
 DataSet<Tuple2<String, Integer>> counts = text.flatMap(new Tokenizer(parameters));
 {% endhighlight %}
 
-and then use them inside the function for getting values from the command line.
+and then use it inside the function for getting values from the command line.
 
 
-#### Passing it as a `Configuration` object to single functions
+#### Passing parameters as a `Configuration` object to single functions
 
 The example below shows how to pass the parameters as a `Configuration` object to a user
defined function.
 
@@ -131,9 +129,9 @@ public static final class Tokenizer extends RichFlatMapFunction<String,
Tuple2<S
 
 #### Register the parameters globally
 
-Parameters registered as a global job parameter at the `ExecutionConfig` allow you to access
the configuration values from the JobManager web interface and all functions defined by the
user.
+Parameters registered as global job parameters in the `ExecutionConfig` can be accessed as
configuration values from the JobManager web interface and in all functions defined by the
user.
 
-**Register the parameters globally**
+Register the parameters globally:
 
 {% highlight java %}
 ParameterTool parameters = ParameterTool.fromArgs(args);
@@ -286,14 +284,14 @@ Change your projects `pom.xml` file like this:
 
 The following changes were done in the `<dependencies>` section:
 
- * Exclude all `log4j` dependencies from all Flink dependencies: This causes Maven to ignore
Flink's transitive dependencies to log4j.
- * Exclude the `slf4j-log4j12` artifact from Flink's dependencies: Since we are going to
use the slf4j to logback binding, we have to remove the slf4j to log4j binding.
+ * Exclude all `log4j` dependencies from all Flink dependencies: this causes Maven to ignore
Flink's transitive dependencies to log4j.
+ * Exclude the `slf4j-log4j12` artifact from Flink's dependencies: since we are going to
use the slf4j to logback binding, we have to remove the slf4j to log4j binding.
  * Add the Logback dependencies: `logback-core` and `logback-classic`
  * Add dependencies for `log4j-over-slf4j`. `log4j-over-slf4j` is a tool which allows legacy
applications which are directly using the Log4j APIs to use the Slf4j interface. Flink depends
on Hadoop which is directly using Log4j for logging. Therefore, we need to redirect all logger
calls from Log4j to Slf4j which is in turn logging to Logback.
 
 Please note that you need to manually add the exclusions to all new Flink dependencies you
are adding to the pom file.
 
-You may also need to check if other dependencies (non Flink) are pulling in log4j bindings.
You can analyze the dependencies of your project with `mvn dependency:tree`.
+You may also need to check if other (non-Flink) dependencies are pulling in log4j bindings.
You can analyze the dependencies of your project with `mvn dependency:tree`.
 
 
 
@@ -301,7 +299,7 @@ You may also need to check if other dependencies (non Flink) are pulling
in log4
 
 This tutorial is applicable when running Flink on YARN or as a standalone cluster.
 
-In order to use Logback instead of Log4j with Flink, you need to remove the `log4j-1.2.xx.jar`
and `sfl4j-log4j12-xxx.jar` from the `lib/` directory.
+In order to use Logback instead of Log4j with Flink, you need to remove `log4j-1.2.xx.jar`
and `sfl4j-log4j12-xxx.jar` from the `lib/` directory.
 
 Next, you need to put the following jar files into the `lib/` folder:
 
@@ -309,7 +307,7 @@ Next, you need to put the following jar files into the `lib/` folder:
  * `logback-core.jar`
  * `log4j-over-slf4j.jar`: This bridge needs to be present in the classpath for redirecting
logging calls from Hadoop (which is using Log4j) to Slf4j.
 
-Note that you need to explicitly set the `lib/` directory when using a per job YARN cluster.
+Note that you need to explicitly set the `lib/` directory when using a per-job YARN cluster.
 
 The command to submit Flink on YARN with a custom logger is: `./bin/flink run -yt $FLINK_HOME/lib
<... remaining arguments ...>`
 

http://git-wip-us.apache.org/repos/asf/flink/blob/81f58bae/docs/dev/stream/checkpointing.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/checkpointing.md b/docs/dev/stream/checkpointing.md
index 774d9ef..3a0a1ae 100644
--- a/docs/dev/stream/checkpointing.md
+++ b/docs/dev/stream/checkpointing.md
@@ -32,7 +32,7 @@ any type of more elaborate operation.
 In order to make state fault tolerant, Flink needs to **checkpoint** the state. Checkpoints
allow Flink to recover state and positions
 in the streams to give the application the same semantics as a failure-free execution.
 
-The [documentation on streaming fault tolerance](../../internals/stream_checkpointing.html)
describe in detail the technique behind Flink's streaming fault tolerance mechanism.
+The [documentation on streaming fault tolerance](../../internals/stream_checkpointing.html)
describes in detail the technique behind Flink's streaming fault tolerance mechanism.
 
 
 ## Prerequisites
@@ -124,12 +124,13 @@ env.getCheckpointConfig.setMaxConcurrentCheckpoints(1)
 
 ## Selecting a State Backend
 
-The checkpointing mechanism stores the progress in the data sources and data sinks, the state
of windows, as well as the [user-defined state](state.html) consistently to
-provide *exactly once* processing semantics. Where the checkpoints are stored (e.g., JobManager
memory, file system, database) depends on the configured
+Flink's [checkpointing mechanism]({{ site.baseurl }}/internals/stream_checkpointing.html)
stores consistent snapshots
+of all the state in timers and stateful operators, including connectors, windows, and any
[user-defined state](state.html).
+Where the checkpoints are stored (e.g., JobManager memory, file system, database) depends
on the configured
 **State Backend**. 
 
-By default state will be kept in memory, and checkpoints will be stored in-memory at the
master node (the JobManager). For proper persistence of large state,
-Flink supports various forms of storing and checkpointing state in so called **State Backends**,
which can be set via `StreamExecutionEnvironment.setStateBackend(…)`.
+By default, state is kept in memory in the TaskManagers and checkpoints are stored in memory
in the JobManager. For proper persistence of large state,
+Flink supports various approaches for storing and checkpointing state in other state backends.
The choice of state backend can be configured via `StreamExecutionEnvironment.setStateBackend(…)`.
 
 See [state backends](../../ops/state_backends.html) for more details on the available state
backends and options for job-wide and cluster-wide configuration.
 

http://git-wip-us.apache.org/repos/asf/flink/blob/81f58bae/docs/dev/stream/process_function.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/process_function.md b/docs/dev/stream/process_function.md
index d682a89..fb5f39d 100644
--- a/docs/dev/stream/process_function.md
+++ b/docs/dev/stream/process_function.md
@@ -42,11 +42,11 @@ For fault-tolerant state, the `ProcessFunction` gives access to Flink's
[keyed s
 `RuntimeContext`, similar to the way other stateful functions can access keyed state.
 
 The timers allow applications to react to changes in processing time and in [event time](../event_time.html).
-Every call to the function `processElement(...)` gets a `Context` object with gives access
to the element's
+Every call to the function `processElement(...)` gets a `Context` object which gives access
to the element's
 event time timestamp, and to the *TimerService*. The `TimerService` can be used to register
callbacks for future
 event-/processing-time instants. When a timer's particular time is reached, the `onTimer(...)`
method is
 called. During that call, all states are again scoped to the key with which the timer was
created, allowing
-timers to perform keyed state manipulation as well.
+timers to manipulate keyed state.
 
 <span class="label label-info">Note</span> If you want to access keyed state
and timers you have
 to apply the `ProcessFunction` on a keyed stream:

http://git-wip-us.apache.org/repos/asf/flink/blob/81f58bae/docs/dev/stream/side_output.md
----------------------------------------------------------------------
diff --git a/docs/dev/stream/side_output.md b/docs/dev/stream/side_output.md
index e4c4c19..63b7172 100644
--- a/docs/dev/stream/side_output.md
+++ b/docs/dev/stream/side_output.md
@@ -55,8 +55,8 @@ val outputTag = OutputTag[String]("side-output")
 Notice how the `OutputTag` is typed according to the type of elements that the side output
stream
 contains.
 
-Emitting data to a side output it only possible when using a
-[ProcessFunction]({{ site.baseurl }}/dev/stream/process_function.html). In the function,
you can use the `Context` parameter
+Emitting data to a side output is only possible from within a
+[ProcessFunction]({{ site.baseurl }}/dev/stream/process_function.html). You can use the `Context`
parameter
 to emit data to a side output identified by an `OutputTag`:
 
 <div class="codetabs" markdown="1">

http://git-wip-us.apache.org/repos/asf/flink/blob/81f58bae/docs/internals/stream_checkpointing.md
----------------------------------------------------------------------
diff --git a/docs/internals/stream_checkpointing.md b/docs/internals/stream_checkpointing.md
index edc7967..d701c5e 100644
--- a/docs/internals/stream_checkpointing.md
+++ b/docs/internals/stream_checkpointing.md
@@ -37,17 +37,20 @@ record from the data stream **exactly once**. Note that there is a switch
to *do
 (described below).
 
 The fault tolerance mechanism continuously draws snapshots of the distributed streaming data
flow. For streaming applications
-with small state, these snapshots are very light-weight and can be drawn frequently without
impacting the performance much.
+with small state, these snapshots are very light-weight and can be drawn frequently without
much impact on performance.
 The state of the streaming applications is stored at a configurable place (such as the master
node, or HDFS).
 
 In case of a program failure (due to machine-, network-, or software failure), Flink stops
the distributed streaming dataflow.
 The system then restarts the operators and resets them to the latest successful checkpoint.
The input streams are reset to the
 point of the state snapshot. Any records that are processed as part of the restarted parallel
dataflow are guaranteed to not
-have been part of the checkpointed state before.
+have been part of the previously checkpointed state.
+
+*Note:* By default, checkpointing is disabled. See [Checkpointing]({{ site.baseurl }}/dev/stream/checkpointing.html)
for details on how to enable and configure checkpointing.
 
 *Note:* For this mechanism to realize its full guarantees, the data stream source (such as
message queue or broker) needs to be able
 to rewind the stream to a defined recent point. [Apache Kafka](http://kafka.apache.org) has
this ability and Flink's connector to
-Kafka exploits this ability.
+Kafka exploits this ability. See [Fault Tolerance Guarantees of Data Sources and Sinks]({{
site.baseurl }}/dev/connectors/guarantees.html) for
+more information about the guarantees provided by Flink's connectors.
 
 *Note:* Because Flink's checkpoints are realized through distributed snapshots, we use the
words *snapshot* and *checkpoint* interchangeably.
 
@@ -79,12 +82,12 @@ Stream barriers are injected into the parallel data flow at the stream
sources.
 (let's call it <i>S<sub>n</sub></i>) is the position in the source
stream up to which the snapshot covers the data. For example, in Apache Kafka, this
 position would be the last record's offset in the partition. This position <i>S<sub>n</sub></i>
is reported to the *checkpoint coordinator* (Flink's JobManager).
 
-The barriers then flow downstream. When an intermediate operator has received a barrier for
snapshot *n* from all of its input streams, it emits itself a barrier
+The barriers then flow downstream. When an intermediate operator has received a barrier for
snapshot *n* from all of its input streams, it emits a barrier
 for snapshot *n* into all of its outgoing streams. Once a sink operator (the end of a streaming
DAG) has received the barrier *n* from all of its
-input streams, it acknowledges that snapshot *n* to the checkpoint coordinator. After all
sinks acknowledged a snapshot, it is considered completed.
+input streams, it acknowledges that snapshot *n* to the checkpoint coordinator. After all
sinks have acknowledged a snapshot, it is considered completed.
 
-When snapshot *n* is completed, it is certain that no records from before <i>S<sub>n</sub></i>
will be needed any more from the source, because these records (and
-their descendant records) have passed through the entire data flow topology.
+Once snapshot *n* has been completed, the job will never again ask the source for records
from before <i>S<sub>n</sub></i>, since at that point these records
(and
+their descendant records) will have passed through the entire data flow topology.
 
 <div style="text-align: center">
   <img src="{{ site.baseurl }}/fig/stream_aligning.svg" alt="Aligning data streams at
operators with multiple inputs" style="width:100%; padding-top:10px; padding-bottom:10px;"
/>
@@ -92,8 +95,8 @@ their descendant records) have passed through the entire data flow topology.
 
 Operators that receive more than one input stream need to *align* the input streams on the
snapshot barriers. The figure above illustrates this:
 
-  - As soon as the operator received snapshot barrier *n* from an incoming stream, it cannot
process any further records from that stream until it has received
-the barrier *n* from the other inputs as well. Otherwise, it would have mixed records that
belong to snapshot *n* and with records that belong to snapshot *n+1*.
+  - As soon as the operator receives snapshot barrier *n* from an incoming stream, it cannot
process any further records from that stream until it has received
+the barrier *n* from the other inputs as well. Otherwise, it would mix records that belong
to snapshot *n* and with records that belong to snapshot *n+1*.
   - Streams that report barrier *n* are temporarily set aside. Records that are received
from these streams are not processed, but put into an input buffer.
   - Once the last stream has received barrier *n*, the operator emits all pending outgoing
records, and then emits snapshot *n* barriers itself.
   - After that, it resumes processing records from all input streams, processing records
from the input buffers before processing the records from the streams.
@@ -103,10 +106,10 @@ the barrier *n* from the other inputs as well. Otherwise, it would have
mixed re
 
 When operators contain any form of *state*, this state must be part of the snapshots as well.
Operator state comes in different forms:
 
-  - *User-defined state*: This is state that is created and modified directly by the transformation
functions (like `map()` or `filter()`). User-defined state can either be a simple variable
in the function's java object, or the associated key/value state of a function (see [State
in Streaming Applications]({{ site.baseurl }}/dev/stream/state.html) for details).
+  - *User-defined state*: This is state that is created and modified directly by the transformation
functions (like `map()` or `filter()`). See [State in Streaming Applications]({{ site.baseurl
}}/dev/stream/state.html) for details.
   - *System state*: This state refers to data buffers that are part of the operator's computation.
A typical example for this state are the *window buffers*, inside which the system collects
(and aggregates) records for windows until the window is evaluated and evicted.
 
-Operators snapshot their state at the point in time when they received all snapshot barriers
from their input streams, before emitting the barriers to their output streams. At that point,
all updates to the state from records before the barriers will have been made, and no updates
that depend on records from after the barriers have been applied. Because the state of a snapshot
may be potentially large, it is stored in a configurable *state backend*. By default, this
is the JobManager's memory, but for serious setups, a distributed reliable storage should
be configured (such as HDFS). After the state has been stored, the operator acknowledges the
checkpoint, emits the snapshot barrier into the output streams, and proceeds.
+Operators snapshot their state at the point in time when they have received all snapshot
barriers from their input streams, and before emitting the barriers to their output streams.
At that point, all updates to the state from records before the barriers will have been made,
and no updates that depend on records from after the barriers have been applied. Because the
state of a snapshot may be large, it is stored in a configurable *[state backend]({{ site.baseurl
}}/ops/state_backends.html)*. By default, this is the JobManager's memory, but for production
use a distributed reliable storage should be configured (such as HDFS). After the state has
been stored, the operator acknowledges the checkpoint, emits the snapshot barrier into the
output streams, and proceeds.
 
 The resulting snapshot now contains:
 
@@ -120,7 +123,7 @@ The resulting snapshot now contains:
 
 ### Exactly Once vs. At Least Once
 
-The alignment step may add latency to the streaming program. Usually, this extra latency
is in the order of a few milliseconds, but we have seen cases where the latency
+The alignment step may add latency to the streaming program. Usually, this extra latency
is on the order of a few milliseconds, but we have seen cases where the latency
 of some outliers increased noticeably. For applications that require consistently super low
latencies (few milliseconds) for all records, Flink has a switch to skip the
 stream alignment during a checkpoint. Checkpoint snapshots are still drawn as soon as an
operator has seen the checkpoint barrier from each input.
 
@@ -138,9 +141,9 @@ in *at least once* mode.
 
 Note that the above described mechanism implies that operators stop processing input records
while they are storing a snapshot of their state in the *state backend*. This *synchronous*
state snapshot introduces a delay every time a snapshot is taken.
 
-It is possible to let an operator continue processing while it stores its state snapshot,
effectively letting the state snapshots happen *asynchronously* in the background. To do that,
the operator must be able to produce a state object that should be stored in a way such that
further modifications to the operator state do not affect that state object. An example for
that are *copy-on-write* style data structures, such as used for example in RocksDB.
+It is possible to let an operator continue processing while it stores its state snapshot,
effectively letting the state snapshots happen *asynchronously* in the background. To do that,
the operator must be able to produce a state object that should be stored in a way such that
further modifications to the operator state do not affect that state object. For example,
*copy-on-write* data structures, such as are used in RocksDB, have this behavior.
 
-After receiving the checkpoint barriers on its inputs, the operator starts the asynchronous
snapshot copying of its state. It immediately emits the barrier to its outputs and continues
with the regular stream processing. Once the background copy process has completed, it acknowledges
the checkpoint to the checkpoint coordinator (the JobManager). The checkpoint is now only
complete after all sinks received the barriers and all stateful operators acknowledged their
completed backup (which may be later than the barriers reaching the sinks).
+After receiving the checkpoint barriers on its inputs, the operator starts the asynchronous
snapshot copying of its state. It immediately emits the barrier to its outputs and continues
with the regular stream processing. Once the background copy process has completed, it acknowledges
the checkpoint to the checkpoint coordinator (the JobManager). The checkpoint is now only
complete after all sinks have received the barriers and all stateful operators have acknowledged
their completed backup (which may be after the barriers reach the sinks).
 
 See [State Backends]({{ site.baseurl }}/ops/state_backends.html) for details on the state
snapshots.
 
@@ -153,6 +156,8 @@ stream from position <i>S<sub>k</sub></i>. For
example in Apache Kafka, that mea
 
 If state was snapshotted incrementally, the operators start with the state of the latest
full snapshot and then apply a series of incremental snapshot updates to that state.
 
+See [Restart Strategies]({{ site.baseurl }}/dev/restart_strategies.html) for more information.
+
 ## Operator Snapshot Implementation
 
 When operator snapshots are taken, there are two parts: the **synchronous** and the **asynchronous**
parts.
@@ -163,4 +168,4 @@ is completed and the *asynchronous* part is pending. The asynchronous
part is th
 Operators that checkpoint purely synchronously return an already completed `FutureTask`.
 If an asynchronous operation needs to be performed, it is executed in the `run()` method
of that `FutureTask`.
 
-The tasks are cancelable, in order to release streams and other resource consuming handles.
+The tasks are cancelable, so that streams and other resource consuming handles can be released.


Mime
View raw message