flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shannon Carey <sca...@expedia.com>
Subject Re: [DISCUSS] Proposal for Asynchronous I/O in FLINK
Date Thu, 22 Sep 2016 17:43:14 GMT
Not to derail this thread onto another topic but the problem with using a static instance is
that there's no way to shut it down when the job stops. So if, for example, it starts threads,
I don't think those threads will stop when the job stops. I'm not very well versed in how
various Java 8 implementations perform unloading of classloaders & class definitions/statics
therein, but it seems problematic unless the job provides a shutdown hook to which user code
can subscribe.

On 9/21/16, 8:05 PM, "David Wang" <dwang2k@gmail.com> wrote:

>Hi Shannon,
>That's right. This FLIP aims to boost TPS of the task workers with async
>i/o operation.
>As what Stephan has mentioned, by placing static attribute to shared
>resources(like event pool, connection), it is possible to share those
>resources among different slots in the same JVM.
>I will make a note in the FLIP about how to share resources ;D
>2016-09-22 1:46 GMT+08:00 Stephan Ewen <sewen@apache.org>:
>> @Shannon: One could have a "static" broker to share the same netty across
>> slots in the same JVM. Implicitly, Flink does the same with broadcast
>> variables.
>> On Wed, Sep 21, 2016 at 5:04 PM, Shannon Carey <scarey@expedia.com> wrote:
>> > David,
>> >
>> > I just wanted to say "thanks" for making this proposal! I'm also
>> > interested in performing nonblocking I/O (multiplexing threads/reactive
>> > programming) within Flink operators so that we can, for example,
>> > communicate with external web services with Netty/RxNetty without
>> blocking
>> > an entire Flink slot (aka a thread) while we wait for the operation to
>> > complete. It looks like your FLIP will enable that use case.
>> >
>> > I'm not sure whether it will be possible to share one Netty
>> EventLoopGroup
>> > (or the equivalent for any other non-blocking framework, connection pool,
>> > etc.) among multiple slots in a single JVM though. Flink supports
>> > open/close operation on a RichFunction, but that's on a per-slot basis. I
>> > don't know of a way to open/close objects on a per-job-JVM basis. But I
>> > suppose that's an issue that should be discussed and resolved separately.
>> >
>> > -Shannon
>> >
>> >
View raw message