ReduceFnRunner (Apache Beam SDK for Java, version 0.4.0)

In-Reply-To: <84111c2e83cc404e92f8e292fe1e5eeb@git.apache.org> References: <84111c2e83cc404e92f8e292fe1e5eeb@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [32/52] [partial] beam-site git commit: Publish Javadoc for version 0.4.0 archived-at: Mon, 09 Jan 2017 22:31:43 -0000 http://git-wip-us.apache.org/repos/asf/beam-site/blob/de6a0417/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/ReduceFnRunner.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/ReduceFnRunner.html b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/ReduceFnRunner.html new file mode 100644 index 0000000..f96c50a --- /dev/null +++ b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/ReduceFnRunner.html @@ -0,0 +1,378 @@ + + + + + + +ReduceFnRunner (Apache Beam SDK for Java, version 0.4.0) + + + + + + + + +

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ + + +

java.lang.Object
+
- org.apache.beam.runners.core.ReduceFnRunner<K,InputT,OutputT,W>
+

+
+
Type Parameters:
+
K - The type of key being processed.
+
InputT - The type of values associated with the key.
+
OutputT - The output type that will be produced for each key.
+
W - The type of windows this operates on.
+
+
+
+
```
public class ReduceFnRunner<K,InputT,OutputT,W extends BoundedWindow>
+extends Object
```
+
Manages the execution of a ReduceFn after a GroupByKeyViaGroupByKeyOnly.GroupByKeyOnly has partitioned the + PCollection by key. + +
The onTrigger(org.apache.beam.runners.core.ReduceFn<K, InputT, OutputT, W>.Context, org.apache.beam.runners.core.ReduceFn<K, InputT, OutputT, W>.Context, boolean, boolean) relies on a TriggerStateMachineRunner to manage the execution + of the triggering logic. The ReduceFnRunners responsibilities are: + +
- Tracking the windows that are active (have buffered data) as elements arrive and triggers are + fired. +
- Holding the watermark based on the timestamps of elements in a pane and releasing it when the + trigger fires. +
- Calling the appropriate callbacks on ReduceFn based on trigger execution, timer + firings, etc, and providing appropriate contexts to the ReduceFn for actions such as + output. +
- Scheduling garbage collection of state associated with a specific window, and making that + happen when the appropriate timer fires. +
+

+ +

+ + +

Constructor Summary

+ + + + + + + + +

Constructors
Constructor and Description
`ReduceFnRunner(K key, + WindowingStrategy<?,W> windowingStrategy, + ExecutableTriggerStateMachine triggerStateMachine, + StateInternals<K> stateInternals, + TimerInternals timerInternals, + OutputWindowedValue<KV<K,OutputT>> outputter, + SideInputReader sideInputReader, + Aggregator<Long,Long> droppedDueToClosedWindow, + ReduceFn<K,InputT,OutputT,W> reduceFn, + PipelineOptions options)`

+ +

+ + +

Method Summary

+ + + + + + + + + + + + + + + + + + +

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`onTimers(Iterable<TimerInternals.TimerData> timers)`
`void`	`persist()`
`void`	`processElements(Iterable<WindowedValue<InputT>> values)` + Incorporate `values` into the underlying reduce function, and manage holds, timers, + triggers, and window merging. +

+ + +
Methods inherited from class java.lang.Object
+clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

+ +
- + + +
  Constructor Detail
  + + + + + +
  - +
    ReduceFnRunner
    +
```
public ReduceFnRunner(K key,
+                      WindowingStrategy<?,W> windowingStrategy,
+                      ExecutableTriggerStateMachine triggerStateMachine,
+                      StateInternals<K> stateInternals,
+                      TimerInternals timerInternals,
+                      OutputWindowedValue<KV<K,OutputT>> outputter,
+                      SideInputReader sideInputReader,
+                      Aggregator<Long,Long> droppedDueToClosedWindow,
+                      ReduceFn<K,InputT,OutputT,W> reduceFn,
+                      PipelineOptions options)
```
    +
  +
+ +
- + + +
  Method Detail
  + + + +
  - +
    processElements
    +
```
public void processElements(Iterable<WindowedValue<InputT>> values)
+                     throws Exception
```
    +
    Incorporate values into the underlying reduce function, and manage holds, timers, + triggers, and window merging. + +
    The general strategy is: +
    1. Use WindowedValue.getWindows() (itself determined using + WindowFn.assignWindows(org.apache.beam.sdk.transforms.windowing.WindowFn<T, W>.AssignContext)) to determine which windows each element belongs to. Some + of those windows will already have state associated with them. The rest are considered + NEW. +
    2. Use WindowFn.mergeWindows(org.apache.beam.sdk.transforms.windowing.WindowFn<T, W>.MergeContext) to attempt to merge currently ACTIVE and NEW windows. + Each NEW window will become either ACTIVE or be discardedL. + (See ActiveWindowSet for definitions of these terms.) +
    3. If at all possible, eagerly substitute NEW windows with their ACTIVE state address + windows before any state is associated with the NEW window. In the common case that + windows for new elements are merged into existing ACTIVE windows then no additional + storage or merging overhead will be incurred. +
    4. Otherwise, keep track of the state address windows for ACTIVE windows so that their + states can be merged on-demand when a pane fires. +
    5. Process the element for each of the windows it's windows have been merged into according + to ActiveWindowSet. Processing may require running triggers, setting timers, + setting holds, and invoking ReduceFn.onTrigger(org.apache.beam.runners.core.ReduceFn<K, InputT, OutputT, W>.OnTriggerContext). +
    +
    +
    Throws:
    +
    Exception
    +
    +
  + + + +
  - +
    persist
    +
```
public void persist()
```
    +
  + + + +
  - +
    onTimers
    +
```
public void onTimers(Iterable<TimerInternals.TimerData> timers)
+              throws Exception
```
    +
    +
    Throws:
    +
    Exception
    +
    +
  +
+

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ +

+ + http://git-wip-us.apache.org/repos/asf/beam-site/blob/de6a0417/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SideInputHandler.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SideInputHandler.html b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SideInputHandler.html new file mode 100644 index 0000000..5d8231a --- /dev/null +++ b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SideInputHandler.html @@ -0,0 +1,437 @@ + + + + + + +SideInputHandler (Apache Beam SDK for Java, version 0.4.0) + + + + + + + + +

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ + + +

java.lang.Object
+
- org.apache.beam.runners.core.SideInputHandler
+

+
+
All Implemented Interfaces:
+
ReadyCheckingSideInputReader, SideInputReader
+
+
+
+
```
public class SideInputHandler
+extends Object
+implements ReadyCheckingSideInputReader
```
+
Generic side input handler that uses StateInternals to store all data. Both the actual + side-input data and data about the windows for which we have side inputs available are stored + using StateInternals. + +
The given StateInternals must not be scoped to an element key. The state + must instead be scoped to one key group for which the side input is being managed. + +
This is useful for runners that transmit the side-input elements in band, as opposed + to how Dataflow has an external service for managing side inputs. + +
Note: storing the available windows in an extra state is redundant for now but in the + future we might want to know which windows we have available so that we can garbage collect + side input data. For now, this will never clean up side-input data because we have no way + of knowing when we reach the GC horizon.
+

+ +

+ + +
Field Summary
+ + + + + + + + + + +
Fields
Modifier and Type Field and Description
protected Collection<PCollectionView<?>> sideInputs +
The list of side inputs that we're handling.
+
+

Fields
Modifier and Type	Field and Description
`protected Collection<PCollectionView<?>>`	`sideInputs` + The list of side inputs that we're handling. +

+ +

+ + +

Constructor Summary

+ + + + + + + + +

Constructors
Constructor and Description
`SideInputHandler(Collection<PCollectionView<?>> sideInputs, + StateInternals<Void> stateInternals)` + Creates a new `SideInputHandler` for the given side inputs that uses + the given `StateInternals` to store side input data and side-input meta data. +

+ +

+ + +

Method Summary

+ + + + + + + + + + + + + + + + + + + + + + + + + + +

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`addSideInputValue(PCollectionView<?> sideInput, + WindowedValue<Iterable<?>> value)` + Add the given value to the internal side-input store of the given side input. +
`<T> boolean`	`contains(PCollectionView<T> view)` + Returns true if the given `PCollectionView` is valid for this reader. +
`<T> T`	`get(PCollectionView<T> sideInput, + BoundedWindow window)` + Returns the value of the given `PCollectionView` for the given `BoundedWindow`. +
`boolean`	`isEmpty()` + Returns true if there are no side inputs in this reader. +
`boolean`	`isReady(PCollectionView<?> sideInput, + BoundedWindow window)` + Returns true if the `PCollectionView` is ready in the provided `BoundedWindow`. +

+ + +
Methods inherited from class java.lang.Object
+clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

+ +
- + + +
  Field Detail
  + + + +
  - +
    sideInputs
    +
```
protected final Collection<PCollectionView<?>> sideInputs
```
    +
    The list of side inputs that we're handling.
    +
  +
+ +
- + + +
  Constructor Detail
  + + + +
  - +
    SideInputHandler
    +
```
public SideInputHandler(Collection<PCollectionView<?>> sideInputs,
+                        StateInternals<Void> stateInternals)
```
    +
    Creates a new SideInputHandler for the given side inputs that uses + the given StateInternals to store side input data and side-input meta data.
    +
  +
+ +
- + + +
  Method Detail
  + + + +
  - +
    addSideInputValue
    +
```
public void addSideInputValue(PCollectionView<?> sideInput,
+                              WindowedValue<Iterable<?>> value)
```
    +
    Add the given value to the internal side-input store of the given side input. This + might change the result of isReady(PCollectionView, BoundedWindow) for that side + input.
    +
  + + + +
  - +
    get
    +
```
@Nullable
+public <T> T get(PCollectionView<T> sideInput,
+                           BoundedWindow window)
```
    +
    Description copied from interface: SideInputReader
    +
    Returns the value of the given PCollectionView for the given BoundedWindow. + +
    It is valid for a side input to be null. It is not valid for this to + return null for any other reason.
    +
    +
    Specified by:
    +
    get in interface SideInputReader
    +
    +
  + + + +
  - +
    isReady
    +
```
public boolean isReady(PCollectionView<?> sideInput,
+                       BoundedWindow window)
```
    +
    Description copied from interface: ReadyCheckingSideInputReader
    +
    Returns true if the PCollectionView is ready in the provided BoundedWindow.
    +
    +
    Specified by:
    +
    isReady in interface ReadyCheckingSideInputReader
    +
    +
  + + + +
  - +
    contains
    +
```
public <T> boolean contains(PCollectionView<T> view)
```
    +
    Description copied from interface: SideInputReader
    +
    Returns true if the given PCollectionView is valid for this reader.
    +
    +
    Specified by:
    +
    contains in interface SideInputReader
    +
    +
  + + + +
  - +
    isEmpty
    +
```
public boolean isEmpty()
```
    +
    Description copied from interface: SideInputReader
    +
    Returns true if there are no side inputs in this reader.
    +
    +
    Specified by:
    +
    isEmpty in interface SideInputReader
    +
    +
  +
+

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ +

+ + http://git-wip-us.apache.org/repos/asf/beam-site/blob/de6a0417/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SimpleDoFnRunner.html ---------------------------------------------------------------------- diff --git a/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SimpleDoFnRunner.html b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SimpleDoFnRunner.html new file mode 100644 index 0000000..9ce31f9 --- /dev/null +++ b/content/documentation/sdks/javadoc/0.4.0/org/apache/beam/runners/core/SimpleDoFnRunner.html @@ -0,0 +1,373 @@ + + + + + + +SimpleDoFnRunner (Apache Beam SDK for Java, version 0.4.0) + + + + + + + + +

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ + + +

java.lang.Object
+
- org.apache.beam.runners.core.SimpleDoFnRunner<InputT,OutputT>
+

+
+
Type Parameters:
+
InputT - the type of the DoFn (main) input elements
+
OutputT - the type of the DoFn (main) output elements
+
+
+
All Implemented Interfaces:
+
DoFnRunner<InputT,OutputT>
+
+
+
+
```
public class SimpleDoFnRunner<InputT,OutputT>
+extends Object
+implements DoFnRunner<InputT,OutputT>
```
+
Runs a DoFn by constructing the appropriate contexts and passing them in. + +
Also, if the DoFn observes the window of the element, then SimpleDoFnRunner + explodes windows of the input WindowedValue and calls DoFn.ProcessElement for + each window individually.
+

+ +

+ + +
Nested Class Summary
+
- + + +
  Nested classes/interfaces inherited from interface org.apache.beam.runners.core.DoFnRunner
  +DoFnRunner.ReduceFnExecutor<K,InputT,OutputT,W>
+

+ +

+ + +

Constructor Summary

+ + + + + + + + +

Constructors
Constructor and Description
`SimpleDoFnRunner(PipelineOptions options, + DoFn<InputT,OutputT> fn, + SideInputReader sideInputReader, + DoFnRunners.OutputManager outputManager, + TupleTag<OutputT> mainOutputTag, + List<TupleTag<?>> sideOutputTags, + ExecutionContext.StepContext stepContext, + Aggregator.AggregatorFactory aggregatorFactory, + WindowingStrategy<?,?> windowingStrategy)`

+ +

+ + +

Method Summary

+ + + + + + + + + + + + + + + + + + +

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`finishBundle()` + Calls a `DoFn's` `@FinishBundle` method and performs + additional tasks, such as flushing in-memory states. +
`void`	`processElement(WindowedValue<InputT> compressedElem)` + Calls a `DoFn's` `@ProcessElement` method with a + `DoFn.ProcessContext` containing the provided element. +
`void`	`startBundle()` + Prepares and calls a `DoFn's` `@StartBundle` method. +

+ + +
Methods inherited from class java.lang.Object
+clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

+ +

+ + +

Constructor Detail

+ + + +

SimpleDoFnRunner

public SimpleDoFnRunner(PipelineOptions options,
+                        DoFn<InputT,OutputT> fn,
+                        SideInputReader sideInputReader,
+                        DoFnRunners.OutputManager outputManager,
+                        TupleTag<OutputT> mainOutputTag,
+                        List<TupleTag<?>> sideOutputTags,
+                        ExecutionContext.StepContext stepContext,
+                        Aggregator.AggregatorFactory aggregatorFactory,
+                        WindowingStrategy<?,?> windowingStrategy)

+ +

+ + +
Method Detail
+ + + +
- +
  startBundle
  +
```
public void startBundle()
```
  +
  Description copied from interface: DoFnRunner
  +
  Prepares and calls a DoFn's @StartBundle method.
  +
  +
  Specified by:
  +
  startBundle in interface DoFnRunner<InputT,OutputT>
  +
  +
+ + + +
- +
  processElement
  +
```
public void processElement(WindowedValue<InputT> compressedElem)
```
  +
  Description copied from interface: DoFnRunner
  +
  Calls a DoFn's @ProcessElement method with a + DoFn.ProcessContext containing the provided element.
  +
  +
  Specified by:
  +
  processElement in interface DoFnRunner<InputT,OutputT>
  +
  +
+ + + +
- +
  finishBundle
  +
```
public void finishBundle()
```
  +
  Description copied from interface: DoFnRunner
  +
  Calls a DoFn's @FinishBundle method and performs + additional tasks, such as flushing in-memory states.
  +
  +
  Specified by:
  +
  finishBundle in interface DoFnRunner<InputT,OutputT>
  +
  +
+

+ + +

Skip navigation links

+ + + +

Prev Class
Next Class

Frames
No Frames

All Classes

+ +

Summary:
Nested |
Field |
Constr |
Method

Detail:
Field |
Constr |
Method

+ + +

+ +

Class ReduceFnRunner<K,InputT,OutputT,W extends BoundedWindow>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

ReduceFnRunner

Method Detail

processElements

persist

onTimers

Class SideInputHandler

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

sideInputs

Constructor Detail

SideInputHandler

Method Detail

addSideInputValue

get

isReady

contains

isEmpty

Class SimpleDoFnRunner<InputT,OutputT>

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.beam.runners.core.DoFnRunner

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

SimpleDoFnRunner

Method Detail

startBundle

processElement

finishBundle