flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Need help on understanding flink runtime and window function
Date Tue, 24 Jan 2017 12:58:47 GMT
Hi,depending on which version of Flink you're using the answer changes. If
you use Flink 1.1 AggregatingProcessingTimeWindowOperator should be
responsible for executing that. In Flink 1.2 it should be WindowOperator.

For a quick overview of how scheduling works in Flink you could look at
this:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/internals/job_scheduling.html.
I'm not aware of documentation on the shipping of Jar files but it roughly
works like this: the job is submitted with the jar file. The JobManager
puts the Jar file into the Blob Manager (which runs on the JobManager).
When parts of the job get scheduled on TaskManagers they retrieve the
required Jar File from the Blob Manager on the JobManager. Internally,
there is a custom ClassLoader that loads code from the user submitted jar
that was retrieved.

Does does help somewhat?

What's the reason for using ContinuousProcessingTimeTrigger? In general I
think almost always it is not right for a use case.

Cheers,
Aljoscha

On Fri, 20 Jan 2017 at 20:10 Fritz Budiyanto <fbudiyan@icloud.com> wrote:

> Hi Flink Dev,
>
> I’m new to Flink and have a few questions below:
>
> 1. I’m trying to understand Flink runtime on the server side, and couldn’t
> figure out where the code which execute the window function sum below. I
> wanted to put a break point but got lost in the code base. Could someone
> shed a light ?
> val counts = text.flatMap { _.toLowerCase.split("\\W+") filter {
> _.nonEmpty } }
>   .map { (_, 1) }
>   .keyBy(0)
>   .window(ProcessingTimeSessionWindows.withGap(Time.seconds(5)))
>   .sum(1)
> 2. How is the Scala jar file get executed on the server side ? Is there
> internal documentation explaining the process ?
>
> 3. I’m planning to use ContinuousProcessingTimeTrigger on a session
> window. Is there possibility in the window function to figure out if the
> window is about to be retired ? For instance, for the recurring trigger I’m
> planning to do some processing. When the window is about to be retired, I’d
> like to do a different processing (ie. Computing final value and flush).
> Any suggestion ?
>
> —
> Fritz

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message