Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
From: Henri Heiskanen <henri.heiskanen@gmail.com>
Date: Mon, 2 Jan 2017 11:25:35 +0200
Message-ID: <CAPxJMyRnHfMc-+U6fm58-Uuxr3PAC7NBMPR72myyBVxC9x=VPw@mail.gmail.com>
Subject: Flink streaming questions
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a11440aec5150b80545192218
archived-at: Mon, 02 Jan 2017 09:25:40 -0000

--001a11440aec5150b80545192218
Content-Type: text/plain; charset=UTF-8

Hi,

I have few questions related to Flink streaming. I am on 1.2-SNAPSHOT and
what I would like to accomplish is to have a stream that reads data from
multiple kafka topics, identifies user sessions, uses an external user user
profile to enrich the data, evaluates an script to produce session
aggregates and then create updated profiles from session aggregates. I am
working with high volume data and user sessions may be long, so using
generic window apply might not work. Below is the simplification of the
stream.

stream = createKafkaStreams(...);
env.setParallelism(4);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
stream
                .keyBy(2)
                .window(EventTimeSessionWindows.withGap(Time.minutes(10)))
                .fold(new SessionData(), new SessionFold(), new
ProfilerApply())
                .print();

The questions:

1. Initially when I used event time windowing I could not get any of my
windows to close. The reason seemed to be that I had 6 partitions in my
test kafka setup and only 4 of them generated traffic. If I used
parallelism above 4, then no windows were closed. Is this by design or a
defect? We use flink-connector-kafka-0.10 because earlier versions did not
commit the offsets correctly.

2. Rich fold functions are not supported. However I would like execute a
piece of custom script in the fold function that requires initialisation
part. I would have used the open and close lifecycle methods of rich
functions but they are not available now in fold. What would be the
preferred way to run some initialisation routines (and closing the
gracefully) when using fold?

3. Kind of related to above. I would also like to fetch a user profile from
external source in the beginning of the session. What would be a best
practice for that kind of operation? If I would be using the generic window
apply I could fetch in in the beginning of the apply method. I was thinking
of introducing a mapper that fetches this profiler periodically and caches
it to flink state. However, with this setup I would not be able to tie this
to user sessions identified for windows.

4. I also may have an additional requirement of writing out each event
enriched with current session and profile data. I basically could do this
again with generic window function and write out each event with collector
when iterating, but would there be a better pattern to use? Maybe sharing
state with functions or something.

Br,
Henri H

--001a11440aec5150b80545192218
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><span style=3D"font-size:12.8px">Hi,</span><div style=3D"f=
ont-size:12.8px"><br></div><div style=3D"font-size:12.8px">I have few quest=
ions related to Flink streaming. I am on 1.2-SNAPSHOT and what I would like=
 to accomplish is to have a stream that reads data from multiple kafka topi=
cs, identifies user sessions, uses an external user user profile to enrich =
the data, evaluates an script to produce session aggregates and then create=
 updated profiles from session aggregates. I am working with high volume da=
ta and user sessions may be long, so using generic window apply might not w=
ork. Below is the simplification of the stream.</div><div style=3D"font-siz=
e:12.8px"><br></div><div style=3D"font-size:12.8px">stream =3D createKafkaS=
treams(...);</div><div style=3D"font-size:12.8px"><div>env.setParallelism(4=
);</div><div>env.<wbr>setStreamTimeCharacteristic(<wbr>TimeCharacteristic.E=
ventTime);</div><div>stream<br></div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 .keyBy(2)</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 .window(<wbr>EventTimeSessionWindows.<wbr>withG=
ap(Time.minutes(10)))</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 .fold(new SessionData(), new SessionFold(), new ProfilerApply=
())</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 .prin=
t();</div></div><div style=3D"font-size:12.8px"><br></div><div style=3D"fon=
t-size:12.8px">The questions:</div><div style=3D"font-size:12.8px"><br></di=
v><div style=3D"font-size:12.8px">1. Initially when I used event time windo=
wing I could not get any of my windows to close. The reason seemed to be th=
at I had 6 partitions in my test kafka setup and only 4 of them generated t=
raffic. If I used parallelism above 4, then no windows were closed. Is this=
 by design or a defect? We use=C2=A0flink-connector-kafka-0.10 because earl=
ier versions did not commit the offsets correctly.</div><div style=3D"font-=
size:12.8px"><br></div><div style=3D"font-size:12.8px">2. Rich fold functio=
ns are not supported. However I would like execute a piece of custom script=
 in the fold function that requires initialisation part. I would have used =
the open and close lifecycle methods of rich functions but they are not ava=
ilable now in fold. What would be the preferred way to run some initialisat=
ion routines (and closing the gracefully) when using fold?</div><div style=
=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">3. Kind of =
related to above. I would also like to fetch a user profile from external s=
ource in the beginning of the session. What would be a best practice for th=
at kind of operation? If I would be using the generic window apply I could =
fetch in in the beginning of the apply method. I was thinking of introducin=
g a mapper that fetches this profiler periodically and caches it to flink s=
tate. However, with this setup I would not be able to tie this to user sess=
ions identified for windows.</div><div style=3D"font-size:12.8px"><br></div=
><div style=3D"font-size:12.8px">4. I also may have an additional requireme=
nt of writing out each event enriched with current session and profile data=
. I basically could do this again with generic window function and write ou=
t each event with collector when iterating, but would there be a better pat=
tern to use? Maybe sharing state with functions or something.</div><div sty=
le=3D"font-size:12.8px"><br></div><div style=3D"font-size:12.8px">Br,</div>=
<div style=3D"font-size:12.8px">Henri H</div></div>

--001a11440aec5150b80545192218--